Skip to content

SolutionsData engineering

Data you can act on while it still matters.

Streaming and batch pipelines on AWS, built as code and tested like software. Records that fail are kept for replay, never silently dropped.

Plan your pipeline

Definition

What is a streaming data pipeline?

A pipeline that moves each event as it happens, not in a nightly batch. On AWS that usually means Amazon Kinesis Data Streams or Amazon MSK to buffer events, AWS Lambda or Amazon Managed Service for Apache Flink to process them, and Amazon S3 or Amazon Redshift to keep them.

Credentials

AWS Services Partner
Advanced Tier
AWS service deliveries
8
AWS certifications
25+
AWS certified
Every engineer

Batch or streaming? Choose per use case.

Not every report needs streaming. We keep batch where a daily answer is enough, and stream where a late answer costs you.

One orders feed, two designs

Nightly batch

  • Yesterday's orders land this morningDaily
  • A failed job is found at 7amManual
  • A rerun reprocesses the whole dayFull reload

Streaming

  • Orders arrive as they are placedContinuous
  • Failed records go to a dead-letter queueKept
  • Only the failed records are replayedTargeted

What keeps a pipeline trustworthy?

Three controls do most of the work: a buffer, a schema contract and a place where failures wait.

Ingest

A buffer between every producer and consumer.

Kinesis or MSK holds a burst. The consumer works through it at its own pace.

Kinesis consumer · a burstIllustrative
Illustrative chart: records arriving spike well above what the consumer can process. The stream holds the backlog, and the consumer keeps processing at its steady rate until the backlog is gone. Records inRecords processedBacklog heldin the stream02:0002:1502:3002:4503:00
Contract

Schema changes are checked before they ship.

Producers register schemas in the AWS Glue Schema Registry. An incompatible change fails in review, not in production.

schemas/order-created.avscAvro+1−1
{  "type": "record",  "name": "OrderCreated",  "fields": [    { "name": "order_id", "type": "string" },    { "name": "amount", "type": "double" },Removed:     { "name": "channel", "type": "string" }Added:     { "name": "channel", "type": "string", "default": "web" }default  ]}
Policy checkGlue Schema Registry · BACKWARDFAIL · incompatible→PASS · compatible
Recovery

Failed records wait to be replayed.

They land in an Amazon SQS dead-letter queue with their error. Once the cause is fixed, we move them back.

# payments API returned 503 from 02:10 to 02:25aws sqs get-queue-attributes --queue-url $DLQ_URL \    --attribute-names ApproximateNumberOfMessages  "ApproximateNumberOfMessages": "42"aws sqs start-message-move-task --source-arn $DLQ_ARNaws sqs list-message-move-tasks --source-arn $DLQ_ARN  "Status": "COMPLETED", "ApproximateNumberOfMessagesMoved": 42PASS42 records replayed to orders-processingPASSdead-letter queue depth: 0

How does an engagement run?

  1. Step 01

    Map the events.

    Sources, consumers and how fresh each answer must be.

  2. Step 02

    Build it as code.

    Terraform or AWS CDK, tested transforms, deployed through CI/CD.

  3. Step 03

    Run it, then hand over.

    Alarms on iterator age, replay runbooks, a trained team.

FAQ

Common questions.

Often batch is enough. If a decision can wait until tomorrow, a scheduled AWS Glue job is simpler and cheaper to run. We stream where a late answer costs you: fraud checks, stock levels, live operations.

Amazon Kinesis Data Streams is fully managed and suits most AWS-native workloads. Amazon MSK suits teams that already run Apache Kafka or depend on its clients and connectors. We choose per workload, not per fashion.

Records that cannot be processed are retried, then moved to a dead-letter queue with their error. When the system is back, we replay them. The stream holds everything else until consumers catch up, within its retention period.

Next step

Plan your pipeline with an engineer.

Tell us which events matter. We sketch the pipeline with you.

Plan your pipeline