On this page
On this page
- What Is Kinesis
- Kinesis Data Streams
- Architecture
- Shards — The Capacity Unit
- Data Retention
- Consumer Types
- Producers
- Consumers
- Scaling — Shard Management
- Kinesis Data Firehose
- Buffering
- Transformation and Format Conversion
- Source Integration
- Kinesis Data Analytics
- Kinesis vs SQS vs SNS
- Common Architecture Patterns
- Pattern 1: Real-Time Analytics Pipeline
- Pattern 2: Log Delivery to S3
- Pattern 3: Multi-Consumer Stream
- Key Exam Scenarios
- CloudWatch Integration
- Automatically Published — Kinesis Data Streams (AWS/Kinesis namespace)
- Requires Setup
Kinesis
Kinesis is AWS's platform for real-time data streaming. It ingests large volumes of continuously arriving data — application logs, clickstream events, IoT sensor readings, financial transactions, video feeds — and makes that data available for processing within milliseconds of arrival.
The Streaming Problem
Batch processing works well when you can wait. You collect data all day and process it at midnight. But many use cases can't wait: detecting fraud as a transaction happens, monitoring a server's health in real time, personalizing a web page based on what a user just clicked, or alerting on a sensor reading the moment it goes out of range. Kinesis is built for these cases.
The Main Services
Kinesis Data Streams is the core building block — a durable, ordered stream of records. Producers write records to the stream; consumers read them and process them. The stream retains data for 24 hours by default (up to 365 days), so consumers can re-read historical data or catch up after downtime. Multiple consumers can read the same stream independently.
Kinesis Data Firehose is the simplest way to load streaming data into a destination — S3, Redshift, OpenSearch, or a third-party service like Splunk. You point producers at Firehose, configure a destination, and data flows there automatically. Firehose can transform, compress, and encrypt data in transit. There's no consumer to write; Firehose handles delivery for you.
Kinesis Data Analytics lets you run SQL queries or Apache Flink applications directly on a stream in real time — computing running averages, detecting patterns, filtering events — without building and managing a separate processing cluster.
Shards and Throughput
A Kinesis Data Stream is divided into shards — each shard provides 1 MB/s of write capacity and 2 MB/s of read capacity. To handle more data, you add more shards. Records in a shard are ordered by arrival time, which is useful when the order of events from the same source matters.
The partition key you assign to each record determines which shard it lands on. Choose partition keys that distribute load evenly — if all records share the same key, they all go to one shard and you waste the rest of your capacity.