AWS Certified Data Engineer – Associate (DEA-C01) Study Guide

AWS Certified Data Engineer – Associate (DEA-C01) Study Guide

Issuing body: Amazon Web Services (AWS) Jurisdiction: Global (vendor certification — not a legal license to practice) Level: Associate Prerequisites: None formal; AWS recommends 2–3 years of data engineering plus 1–2 years hands-on AWS. (Source: catalogue record data-ai-ml-cert-005.) Renewal: 3 years (retake the current exam or pass a higher-level exam).

1. What this exam is

DEA-C01 tests whether you can build and operate data pipelines on AWS: ingestion, transformation, storage, and serving, with the security, monitoring, and cost discipline production data systems require. It's the data-platform counterpart to the ML exam — where MLA asks "how do we train and serve models," DEA asks "how does the data get there reliably, fresh, and cheap." The exam is relentlessly practical: every question is a pipeline design or debugging scenario.

2. Pathway and exam format

• Typical path: AWS Solutions Architect – Associate or Cloud Practitioner (for AWS literacy) → 1–2 years building pipelines (Glue, Lambda, S3, SQL) → DEA-C01.
• Format: 65 questions, 130 minutes (2 min/question), multiple-choice/multiple-response, scored 100–1000, passing 720. $150 USD via Pearson VUE. (Source: catalogue record data-ai-ml-cert-005, verified against 2026 community study guides.)
• The exam's mental model: every question involves data moving through stages — source → ingest → store → transform → serve — with constraints on latency, volume, cost, and freshness. Identify the bottleneck stage first; the answer almost always targets it.

3. Knowledge domains (with theory)

Public blueprint (approximate): Data Ingestion and Transformation (~34%), Data Store Management (~26%), Data Operations and Support (~22%), Data Security and Governance (~18%).

Ingestion: streaming vs batch, and the theory of "fresh"

Freshness vs cost vs complexity: batch (hourly/daily — cheap, simple, high latency) → micro-batch (minutes) → streaming (seconds — expensive, complex). The exam punishes over-engineering: a daily report doesn't need Kinesis.

AWS ingestion services:

• Kinesis Data Streams: ordered, replayable, shard-based. Shard = 1 MB/s in, 2 MB/s out. Provisioned vs on-demand capacity mode. Partition keys distribute across shards — hot keys = hot shards (the #1 Kinesis failure). Enhanced fan-out for multiple consumers (dedicated 2 MB/s per consumer).
• Kinesis Data Firehose: managed delivery to S3/Redshift/OpenSearch — buffering (size/time hints), format conversion (to Parquet!), compression. No ordering guarantees needed = Firehose over Streams.
• MSK (Managed Streaming for Kafka): when you need Kafka semantics (exactly-once transactions, Kafka ecosystem, compacted topics). Know when Kafka is justified vs Kinesis (existing Kafka expertise, ecosystem connectors, exactly-once needs).
• DMS: database migration and ongoing CDC replication (full load + change data capture) — the "keep the warehouse in sync with the OLTP DB" answer.
• DataSync / Transfer Family / Snowball: bulk file movement (NFS→S3, SFTP ingestion, petabyte offline).
• AppFlow: SaaS ingestion (Salesforce → S3/Redshift) — the no-code SaaS connector answer.

Exactly-once semantics (the theory): true exactly-once requires idempotent sinks + transactional writes (Kafka transactions, Kinesis + DynamoDB conditional writes). Firehose/SQS are at-least-once — deduplicate downstream (event IDs, MERGE/UPSERT keys). The exam tests whether you design idempotency rather than assuming the pipe provides it.

Transformation: ETL/ELT and the Glue ecosystem

ETL vs ELT: ETL (transform before load — legacy, rigid schemas) vs ELT (load raw, transform in the warehouse/lakehouse — the modern default: preserve raw data, transform with SQL/Spark on demand). The exam favors ELT for analytics lakes.

• AWS Glue: serverless Spark ETL; Glue Data Catalog (central metastore — Hive-compatible); crawlers (schema discovery); Glue jobs (Spark/Python shell); bookmarks (process only new data — incremental loads); DataBrew (visual prep).
• EMR: managed Hadoop/Spark when you need cluster control (long-running, custom AMIs, specific Spark versions). EMR Serverless for the middle ground.
• Lambda: event-driven micro-transforms (S3 PUT → transform → write). 15-min limit — not for big batch.
• Step Functions: orchestration (the Glue Workflow alternative with better error handling); EventBridge Scheduler for cron-like triggers; MWAA (managed Airflow) for complex DAGs with existing Airflow expertise.

File formats (know cold): Parquet/ORC (columnar — 10–100× less scanned for analytics; Snappy/GZIP compression), Avro (row-based, schema evolution for streaming), JSON/CSV (human-readable, expensive at scale — never the analytics answer). Partitioning (by date/region — prune at query time) is the #1 Athena/Redshift-Spectrum cost lever.

Storage: the lakehouse decision tree

• S3 as the data lake: storage classes + lifecycle policies (Standard → Intelligent-Tiering/Standard-IA → Glacier); S3 Tables / Iceberg support for lakehouse table formats; object versioning for auditability.
• Lake Formation: centralized permissions for the lake (LF-tags, row/column-level security) — the governance answer replacing bucket-policy spaghetti.
• Redshift: RA3 (managed storage + compute separation, data sharing); distribution styles (KEY for joins on the key, ALL for small dims, EVEN default, AUTO the modern answer); sort keys (compound vs interleaved); WLM queues for workload isolation. Redshift Serverless for variable workloads.
• DynamoDB: operational serving (single-digit ms lookups); streams → Lambda for CDC out.
• Aurora/RDS: transactional sources; zero-ETL integrations (Aurora → Redshift) for the "no pipeline" answer.
• OpenSearch: search/analytics on logs; Timestream/QLDB for their niches.

The exam's storage questions are freshness/cost/latency trade-offs: hot operational → DynamoDB; analytical SQL → Redshift; ad-hoc exploration → Athena on S3; full-text → OpenSearch.

Operations, security, governance

• Monitoring: CloudWatch metrics/alarms on pipeline stages (Glue job failures, Kinesis iterator age — iterator age is the streaming-health metric: growing age = consumer falling behind), data quality (Glue Data Quality / Deequ-style checks: completeness, uniqueness, freshness SLAs).
• Orchestration failure handling: retries with backoff, dead-letter queues, idempotent replays. Backfill design: partitioned data + idempotent transforms = safe replays.
• Security: KMS encryption (SSE-S3/SSE-KMS; KMS for the audit trail), IAM roles per pipeline stage (least privilege), VPC endpoints for S3/DynamoDB (no internet traversal), Lake Formation fine-grained access, Macie for PII discovery, CloudTrail for audit.
• Governance: Data Catalog as the metastore, Glue crawlers vs explicit schemas (crawlers drift — explicit DDL for production), data lineage (who produced this table?).

4. Key formulas and numbers

• Kinesis shard math: shards needed = max(peak MB/s in, peak records/s ÷ 1000). 5 MB/s in → 5 shards minimum. Resharding splits/merges — plan for 2× headroom.
• Iterator age alarm: alert when age > 5–10 min (consumer lagging); age growing unboundedly = under-provisioned consumers or poison-pill records.
• Parquet + partitioning: expect 10–100× scan reduction vs raw JSON — the Athena cost math: $5/TB scanned; 1 TB/day unpartitioned JSON → partitioned Parquet ≈ $0.05–0.50/day.
• Little's Law for streams: consumers needed = arrival rate × processing time per record ÷ parallelism per consumer.
• Batch window math: daily job must finish in <24 h with headroom — target <50% of the window (leaves room for retries/backfills).
• Redshift distribution: KEY on the join key when two large tables join frequently; ALL for dimensions <~3M rows; wrong distribution = network-heavy joins (the #1 Redshift perf bug).

5. Worked examples

Example 1 — Clickstream pipeline. Scenario: 20k events/sec web clickstream; need real-time dashboard (30 s freshness) + daily ML features + 2-year retention, cost-conscious. Solution: Kinesis Data Streams (partition key = user ID; on-demand for variable traffic) → Firehose (buffer 60 s/128 MB, convert to Parquet, compress) → S3 (partitioned by date/hour; lifecycle to Intelligent-Tiering); Kinesis Analytics/Flink or Lambda for the real-time aggregates → DynamoDB serving the dashboard; Glue jobs for daily feature engineering → S3 curated zone → SageMaker Feature Store offline. Athena for ad-hoc. One ingestion path, three serving paths — the lakehouse pattern.

Example 2 — CDC from RDS to Redshift. Scenario: Keep Redshift within 5 min of a 500 GB MySQL RDS; minimal operational overhead. Solution: DMS full load + ongoing CDC (binlog) → S3 staging (Parquet) → Redshift (COPY or Redshift integration); or Aurora zero-ETL to Redshift if on Aurora (no DMS at all — near-real-time, the exam's preferred answer when available). DMS task tuning: parallel load for the full load, CDC latency alarms. Handle schema changes: DMS with permissive mapping + Glue crawler alerts on drift.

Example 3 — Poison-pill records. Scenario: Kinesis consumer iterator age climbing; logs show JSON parse errors on some records. Solution: Wrap parsing in try/except; route malformed records to a dead-letter S3 prefix (with the raw payload + error); continue processing (don't let one bad record stall the shard — Kinesis checkpoints per shard, so a stuck record blocks everything behind it). Add a Glue Data Quality / Lambda validator upstream; alert on DLQ growth rate. The principle: never let data quality halt the pipeline — quarantine and continue.

Example 4 — Cost blowup in Athena. Scenario: Athena bill 10×'d after a new team started querying. Solution: Audit via CloudTrail/CloudWatch (who scanned what); findings usually: SELECT * on unpartitioned JSON. Fix: convert source to partitioned Parquet (Glue ETL or Firehose conversion), enforce workgroup settings (per-query data limits, query result reuse), consider Redshift Serverless or provisioned for sustained heavy use. The math: 500 GB/day scanned = $75/day → partitioned Parquet ≈ $1–5/day.

Example 5 — Multi-tenant SaaS analytics. Scenario: SaaS with 1,000 tenants; each tenant's analysts query their own data; strict tenant isolation. Solution: S3 partitioned by tenant_id + Lake Formation LF-tags with row-level filtering (tenant_id = caller) — one table, policy-enforced isolation; or per-tenant Glue tables for the largest tenants. Athena workgroups per tenant for cost attribution. The trap: separate S3 buckets per tenant (1,000 buckets of operational overhead) or, worse, no isolation (cross-tenant leak = breach).

6. Standards and regulations cross-reference

• GDPR — data-subject rights over pipelines: erasure must propagate (tombstone records, reprocess partitions, expire caches — design the erasure pipeline, not just the feature); data minimization (don't ingest what you don't need); DPIAs for high-risk processing.
• DAMA-DMBOK — the data-management body of knowledge: governance, architecture, modeling, storage, integration, quality, metadata, security. The vocabulary for data-governance conversations.
• ISO/IEC 27001 — A.8.10 (deletion of information), A.8.13 (backup), A.5.33 (protection of procedures) map to pipeline controls: retention policies, backup/restore tests, runbook discipline.
• PCI DSS v4.0.1 — cardholder data in pipelines: tokenize at ingestion (never land raw PAN in the lake), scope the CDE pipeline segments, mask in logs, encrypt everywhere.
• SOC 2 — pipeline change control (all transforms versioned in git, deployed via pipeline), access reviews on data access, audit evidence from CloudTrail/Lake Formation.
• EU AI Act / SR 11-7 — when the pipeline feeds regulated models: lineage (which data trained which model), quality gates, and documented data provenance become legal artifacts.

7. Pitfalls, traps, and exam strategy

Trap patterns:

1. The streaming trap: batch-sufficient workload → Kinesis. Always check the freshness requirement first.
2. The Firehose-vs-Streams trap: need replay/ordering/multiple consumers → Streams; need simple delivery to S3 → Firehose (with format conversion!).
3. The JSON trap: JSON in the lake for analytics — always Parquet/ORC + partitioning for the curated zones.
4. The iterator-age trap: "is the stream healthy?" → check iterator age, not just throughput.
5. The DMS trap: DMS needs the source's binlog/CDC enabled and sufficient retention — the #1 DMS setup failure.
6. The Redshift distribution trap: wrong DISTSTYLE = slow joins. KEY for large-table joins, ALL for small dims, AUTO if unsure.

Strategy: 2 min/question — fast. For each scenario, identify: volume, freshness, consumers, then pick the ingestion → storage → serving chain. Memorize: Kinesis shard math, the four file formats and when each wins, Glue components (catalog/crawler/job/bookmark), Redshift distribution styles, and the Lake Formation governance model. Everything else is pipeline common sense.

8. Practice problems

P1. A stream peaks at 8 MB/s in, 12,000 records/s. How many shards minimum, and what happens if one partition key gets 50% of traffic? Answer: max(8 MB/s ÷ 1, 12000 ÷ 1000) = max(8, 12) = 12 shards. Hot key: one shard gets ~6,000 records/s > 1,000 limit → WriteProvisionedThroughputExceeded, throttling, iterator age climbs on that shard. Fix: better partition key (composite with randomness suffix), or more shards won't help a single hot key — key design is the fix.

P2. A Glue job processes "all new files" daily but sometimes reprocesses or misses files. What's the mechanism to fix it and how does it work? Answer: Glue job bookmarks: track processed state (by file path/modified time); on rerun, only new/unprocessed files are read. For misses: ensure bookmark keys cover the partition scheme; for exactly-once semantics, combine bookmarks with idempotent writes (partition overwrite, not append). Alternative: event-driven (S3 → SQS → job) for near-real-time.

P3. Two 500 GB tables join on customer_id in Redshift; the join is slow with high network traffic. Diagnose and fix. Answer: Data redistribution across nodes during the join — the tables have different DISTSTYLEs (or EVEN). Fix: DISTSTYLE KEY (customer_id) on both (co-located joins, no network), or DISTSTYLE ALL on the smaller if <~3M rows. Verify with EXPLAIN (look for DS_BCAST_INNER / DS_DIST_ALL_NONE — want DS_DIST_NONE).

P4. Design GDPR erasure for a lakehouse: user 12345 requests deletion; their data is in raw S3 (JSON), curated Parquet (partitioned by date), Redshift, and SageMaker training sets. Answer: Erasure pipeline: (1) tombstone/delete in raw (S3 delete markers or rewrite affected objects), (2) rewrite affected Parquet partitions excluding the user (partition-level rewrite — the reason for date partitioning), (3) DELETE in Redshift + VACUUM, (4) flag training datasets as tainted; retrain or document model-retraining policy (the hard part — state the policy explicitly), (5) expire caches/DynamoDB TTL. Automate via Step Functions; log each step for the audit trail. Design the pipeline before the first request arrives.

P5. DMS CDC lag is growing on a busy MySQL source. List three causes in likelihood order. Answer: (1) Source binlog retention too short / binlog reader falling behind under write spikes (check CDC latency metrics; scale the replication instance). (2) Long-running transactions on the source holding binlog position (find and kill/split them). (3) Target apply bottleneck (Redshift WLM queue saturated; tune batch apply). Also: replication instance undersized — the boring, common cause.

P6. Choose: Kinesis Data Streams vs MSK for a team with deep Kafka expertise, needing exactly-once processing and 50+ consumer teams. Answer: MSK: exactly-once via Kafka transactions, consumer groups scale to 50+ teams natively, and the team's expertise is the deciding operational factor. Kinesis would need enhanced fan-out per consumer ($$) and lacks Kafka's ecosystem (Schema Registry, ksqlDB). The exam's lesson: ecosystem + expertise + semantics outweigh "simpler" when the requirements genuinely need Kafka.

Original ETS reference writing, verified against public sources on 2026-09-30. Thin sourcing is flagged inline. Not legal advice — confirm current editions with the issuing body.