Monitor Amazon Kinesis data streams
editMonitor Amazon Kinesis data streams
editAmazon Kinesis Data Streams is a real-time data streaming service that you can use to capture, process, and store large amounts of data from a variety of sources, including websites, mobile applications, IoT devices, and more.
Amazon Kinesis Data Streams and Amazon CloudWatch are integrated so that you can collect and monitor CloudWatch metrics for your Kinesis data streams, such as tracking shard usage and recording related operations for each Kinesis data stream. The Elastic Amazon Kinesis Data Stream integration collects metrics from Amazon CloudWatch using Elastic Agent.
By default, Kinesis Data Streams sends stream-level (basic level) metrics to
CloudWatch every minute automatically. There is also shard-level data (enhanced
level) that is sent to CloudWatch every minute and incurs an additional cost per
stream. To get shard-level data, you must specifically enable it for each
stream by using the AWS Kinesis enable-enhanced-monitoring
API. For example
aws kinesis enable-enhanced-monitoring --stream-name samplestream --shard-level-metrics ALL
For more details, refer to the EnableEnhancedMonitoring documentation.
Get started
editTo collect Kinesis data stream metrics from Amazon CloudWatch, you typically need to install the Elastic Amazon Kinesis Data Stream integration and deploy an Elastic Agent.
Expand the quick guide to learn how, or skip to the next section if your data is already in Elasticsearch.
Quick guide: Add data
Elastic Agent is currently the preferred way to add Kinesis data stream metrics. For other ways, refer to Adding data to Elasticsearch.
Dashboards
editKibana provides a full data analytics platform with out-of-the-box dashboards that you can clone and enhance to satisfy your custom visualization use cases. For example, to see an overview of your Kinesis data streams in Kibana, go to the Dashboard app and navigate to the [Metrics AWS] Kinesis Overview dashboard.
Metrics to watch
editThis section lists the key metrics that you should watch, organized by category. For a full description of fields exported by the integration, refer to the Amazon Kinesis Data Stream integration docs.
-
GetRecords
-
aws.kinesis.metrics.GetRecords.Bytes.avg
-
aws.kinesis.metrics.GetRecords.IteratorAgeMilliseconds.avg
-
aws.kinesis.metrics.GetRecords.Latency.avg
-
aws.kinesis.metrics.GetRecords.Records.avg
-
aws.kinesis.metrics.GetRecords.Records.sum
-
aws.kinesis.metrics.GetRecords.Success.avg
-
aws.kinesis.metrics.GetRecords.Success.sum
-
aws.kinesis.metrics.ReadProvisionedThroughputExceeded.avg
-
aws.kinesis.metrics.IteratorAgeMilliseconds.avg
-
-
PutRecord
-
aws.kinesis.metrics.PutRecord.Bytes.avg
-
aws.kinesis.metrics.PutRecord.Latency.avg
-
aws.kinesis.metrics.PutRecord.Success.avg
-
-
PutRecords
-
aws.kinesis.metrics.PutRecords.Bytes.avg
-
aws.kinesis.metrics.PutRecords.Latency.avg
-
aws.kinesis.metrics.PutRecords.Success.avg
-
aws.kinesis.metrics.PutRecords.TotalRecords.sum
-
aws.kinesis.metrics.PutRecords.SuccessfulRecords.sum
-
aws.kinesis.metrics.PutRecords.FailedRecords.sum
-
aws.kinesis.metrics.PutRecords.ThrottleRecords.sum
-
aws.kinesis.metrics.WriteProvisionedThroughputExceeded.avg
-
-
Incoming and outgoing
-
aws.kinesis.metrics.IncomingBytes.avg
-
aws.kinesis.metrics.IncomingRecords.avg
-
aws.kinesis.metrics.OutgoingBytes.avg
-
aws.kinesis.metrics.OutgoingRecords.avg
-
aws.kinesis.metrics.OutgoingRecords.sum
-
-
SubscribeToShard
-
aws.kinesis.metrics.SubscribeToShard.RateExceeded.avg
-
aws.kinesis.metrics.SubscribeToShard.Success.avg
-
aws.kinesis.metrics.SubscribeToShardEvent.Bytes.avg
-
aws.kinesis.metrics.SubscribeToShardEvent.MillisBehindLatest.avg
-
aws.kinesis.metrics.SubscribeToShardEvent.Success.avg
-
aws.kinesis.metrics.SubscribeToShardEvent.Records.sum
-