Understand how data engineering works in cloud environments.
School of Data & AIData EngineeringIntermediateIncluded in a Professional Diploma
Cloud Data Engineering
Learn how modern data pipelines and warehouses run in the cloud.
Understand cloud storage, managed databases, warehouses, compute, access control, monitoring, logging, deployment, and cost-aware architecture for data engineering workflows.
Duration
8 weeks - 6-8 hours/week
Project
Understand how data engineering works in cloud environments.
Support
Pricing and enrolment are handled through the Professional Diploma
Overview
A practical Short Course built around a visible project.
Learn how data engineering works in the cloud, including storage, compute, managed databases, warehouses, pipeline deployment, access control, monitoring, and cost-aware architecture.
Use cloud storage concepts for raw and processed data.
Understand managed databases and warehouse-style services.
Choose basic compute options for data workloads.
Apply access control and security basics.
Deploy data scripts and pipeline workflows.
Monitor logs, failures, and pipeline health.
Understand cost-aware cloud data architecture.
Prepare analytics-ready cloud data workflows.
Build portfolio-ready cloud data engineering projects.
Course roadmap
What you will work through.
The sequence below is specific to this course. It shows the phases, modules, lessons, and page outlines that move you toward Understand how data engineering works in cloud environments..
1Phase 1 - Cloud Data Engineering FoundationsBuild the cloud data engineering mindset: why data engineering moved to the cloud, platform architecture, team roles, core cloud concepts, and service mapping.2 modules9 lessons1–2 weeks
Module 1: Cloud Data Engineering MindsetUnderstand why cloud platforms matter, how cloud data architectures work, who owns what, and what core cloud concepts data engineers must know.4 lessons
Lesson 1: Why Data Engineering Moved to the CloudUnderstand why modern data platforms moved to cloud infrastructure: storage scale, elastic compute, managed services, serverless tools, reliability, global access, and cost tradeoffs.85 minarticle6 pages
Welcome and Learning Objectives
Introduce the reason cloud became central to data engineering.
8 min
From Fixed Infrastructure to Elastic Platforms
Explain elastic cloud infrastructure.
18 min
Managed Services and Serverless Data Tools
Explain managed services and serverless concepts.
18 min
Reliability and Cost Tradeoffs
Explain cloud tradeoffs.
18 min
Cloud Does Not Replace Data Engineering Fundamentals
Connect cloud to previous path courses.
18 min
Exercise - On-Premise vs Cloud Pipeline Comparison
Students compare on-premise and cloud data pipelines.
23 min
Lesson 2: Cloud Data Platform ArchitectureDesign cloud data platform architecture using source systems, ingestion, object storage, data lake, metadata catalog, transformation, warehouse, BI, monitoring, and governance.85 minarticle5 pages
Welcome and Learning Objectives
Introduce cloud data platform architecture.
8 min
Core Architecture Components
Explain architecture components.
20 min
AWS-First Reference Architecture
Show AWS architecture.
20 min
Architecture Translation Across Clouds
Introduce multi-cloud translation.
18 min
Exercise - Cloud Data Platform Diagram
Students draw a cloud data platform.
39 min
Lesson 3: Cloud Data Engineering RolesMap responsibilities across cloud data engineers, analytics engineers, platform engineers, BI engineers, data architects, DevOps, and security collaborators.80 minarticle4 pages
Welcome and Learning Objectives
Introduce cloud data platform roles.
8 min
Core Roles in Cloud Data Platforms
Explain key roles.
24 min
Ownership Across Platform Layers
Map responsibilities.
18 min
Exercise - Cloud Data Platform Responsibility Map
Students map team responsibilities.
30 min
Lesson 4: Core Cloud Concepts for Data EngineersExplain regions, availability zones, managed services, serverless, compute, storage, networking basics, identity, permissions, and billing to non-cloud engineers.85 minarticle5 pages
Welcome and Learning Objectives
Introduce core cloud concepts.
8 min
Regions, Availability and Managed Services
Explain cloud infrastructure basics.
20 min
Compute, Storage, Networking and Identity
Explain core layers.
22 min
Billing Basics for Data Engineers
Explain cost awareness.
18 min
Exercise - Explain Cloud Pipeline Components
Students explain cloud components to a non-cloud engineer.
37 min
Module 2: Cloud Service MappingMap storage, integration, warehouse/query, orchestration, and monitoring services across AWS, Azure/Microsoft Fabric, and Google Cloud.5 lessons
Lesson 1: Storage ServicesMap AWS S3, Azure Data Lake Storage/OneLake, Google Cloud Storage, buckets, containers, prefixes, lifecycle rules, and object metadata.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Data Integration ServicesCompare AWS Glue, Azure Data Factory, Fabric Data Factory, Google Dataflow, managed connectors, transformation engines, and serverless jobs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Warehouse and Query ServicesCompare Amazon Redshift, Athena, BigQuery, Synapse/Fabric Warehouse, serverless query, warehouse compute, and lakehouse patterns.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Orchestration and Monitoring ServicesMap MWAA, Cloud Composer, Data Factory pipelines, EventBridge, Step Functions, CloudWatch-style monitoring, Azure Monitor, and Google Cloud Monitoring.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 5: Mini Project 1 - Cloud Data Platform BlueprintStudents produce a cloud platform blueprint with AWS-first service choices and multi-cloud comparison.120 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
2Phase 2 - Cloud Storage and Data Lake DesignDesign cloud object storage, data lake zones, file formats, partitions, and AWS S3 implementation/documentation.2 modules9 lessons1–2 weeks
Module 1: Object Storage for Data EngineeringDesign object storage and data lake structures with zones, formats, partitions, and performance tradeoffs.4 lessons
Lesson 1: Object Storage FundamentalsUnderstand buckets, objects, prefixes, metadata, durability, storage classes, lifecycle, and data lake patterns.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Data Lake ZonesDesign landing, raw, staging, cleaned, curated, serving, archive, and quarantine zones.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: File FormatsChoose CSV, JSON, Parquet, Avro concept, compression, row vs columnar storage, performance, and tradeoffs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Partitioning in Data LakesDesign date, source, region, event-type partitions, pruning, depth, bad partitions, and small file controls.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
37 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
30 min
Module 2: Hands-On AWS S3 Data LakeImplement or design an AWS S3 data lake with upload, organization, lifecycle, and documentation.5 lessons
Lesson 1: Creating Data Lake BucketsSet up S3 buckets, naming, prefixes, environments, dev/staging/prod concept, and ownership.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Uploading and Organizing DataUse manual upload, AWS CLI, boto3 basics, batch uploads, source naming, batch dates, and raw evidence preservation.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Lifecycle and RetentionDesign retention, archive storage, deletion rules, cost control, compliance, and temporary data cleanup.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Data Lake DocumentationWrite source inventory, zone definitions, naming standards, partition standards, ownership, retention, and access notes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 5: Milestone Project 1 - Cloud Data Lake SetupBuild or design an AWS-first data lake with zones, datasets, partitions, formats, lifecycle, access assumptions, and documentation.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
3Phase 3 - Cloud Ingestion and Managed ETLBuild cloud batch ingestion, Python-to-cloud storage workflows, managed ETL with Glue, cataloging, incremental loads, and ETL monitoring.2 modules10 lessons1–2 weeks
Module 1: Cloud Batch IngestionDesign file, Python, API, database, and managed ETL ingestion into cloud storage.4 lessons
Lesson 1: File-Based Cloud IngestionDesign landing zones, file drops, source folders, upload automation, validation, archive patterns, and rejected files.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Python to Cloud StorageUse boto3 upload/download, S3 reads/writes, credentials, environment variables, error handling, and metadata.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
37 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
30 min
Lesson 3: Database and API to Cloud StorageDesign extracts to cloud storage, raw landing files, batch IDs, timestamps, incremental outputs, and secure credentials.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Managed ETL ServicesCompare AWS Glue, Azure Data Factory/Fabric, Google Dataflow, serverless transformation, connectors, and custom Python tradeoffs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Module 2: Cloud ETL with AWS GlueUse AWS Glue concepts, crawlers, cataloging, ETL jobs, incremental logic, and monitoring.6 lessons
Lesson 1: AWS Glue ConceptsExplain Glue jobs, Studio, crawlers, Data Catalog, connections, triggers, bookmarks, and PySpark concept.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Glue Crawlers and CatalogingUse crawlers, schema discovery, tables, partitions, Data Catalog, schema drift risks, and limitations.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Glue ETL JobsDesign source, transform, target, PySpark concept, parameters, output formats, and curated outputs.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
37 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
30 min
Lesson 4: Incremental Loads in GlueDesign job bookmarks, new files, changed data, partitions, incremental processing, and rerun risk controls.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 5: Glue Job MonitoringReview job runs, logs, failures, retries, metrics, debugging, and failed job recovery.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 6: Milestone Project 2 - Cloud Ingestion and ETL PipelineBuild or simulate cloud ingestion, raw landing, cataloging, managed ETL, curated outputs, logs, and documentation.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
4Phase 4 - Metadata Catalogs and Serverless SQLDesign metadata catalogs, schema evolution handling, partition metadata, governance, serverless SQL, Athena patterns, performance, and quality checks.2 modules9 lessons1 week
Module 1: Metadata Catalogs and SchemasBuild catalog thinking for discoverability, schemas, partitions, ownership, and governance.4 lessons
Lesson 1: Why Catalogs MatterExplain discoverability, schema management, table definitions, partition metadata, query engines, governance, and ownership.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Schema EvolutionManage new columns, changed types, missing fields, drift, backward compatibility, consumer impact, and breaking changes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Partition MetadataHandle partition registration, repair, date partitions, partition projection concept, stale and missing partitions.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Data Catalog GovernanceDocument ownership, table descriptions, column descriptions, sensitive labels, access notes, and stewardship.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Module 2: Serverless SQL on Data LakesQuery files with SQL, use Athena-style external tables, optimize serverless queries, and perform lake quality checks.5 lessons
Lesson 1: Querying Files with SQLUse external tables, SQL over object storage, schema-on-read, serverless query engines, lakehouse patterns, and query cost.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Athena FundamentalsCreate databases, tables, external locations, SQL queries, result locations, history, workgroups, and cost awareness.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Serverless Query PerformanceOptimize Parquet, compression, partition pruning, fewer columns, avoiding full scans, result reuse, and small file issues.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Lake Query Quality ChecksWrite row count, null, duplicate, partition, freshness, and relationship checks using serverless SQL.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 5: Milestone Project 3 - Serverless Data Lake AnalyticsCreate cataloged lake tables, serverless SQL queries, quality checks, cost/performance notes, and BI-ready outputs.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
5Phase 5 - Cloud Data Warehousing and PerformanceDesign cloud warehouses, compare platforms, load lake data, build schemas/marts, use Redshift concepts, optimize performance, and control cost.3 modules13 lessons1–2 weeks
Module 1: Cloud Warehouse FoundationsUnderstand cloud warehouse architecture, platform choices, loading, and schema design.4 lessons
Lesson 1: What Makes Cloud Warehouses DifferentExplain elastic compute, storage/compute separation, serverless options, concurrency, scaling, managed operations, and billing patterns.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Warehouse Platform ComparisonCompare Amazon Redshift, BigQuery, Azure Synapse/Fabric Warehouse, Snowflake optional, workload fit, pricing, and ecosystem.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Loading Data into WarehousesDesign bulk loading, COPY-style commands, external tables, staged files, incremental loads, data types, and metadata.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Warehouse Schema DesignDesign datasets/schemas, facts/dimensions, marts, partitions, clustering/sort keys, naming, and access boundaries.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
37 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
30 min
Module 2: AWS Redshift and Warehouse PatternsUse Redshift concepts, lakehouse/external tables, warehouse transformations, and reconciliation.4 lessons
Lesson 1: Redshift ConceptsUnderstand clusters/serverless concept, databases, schemas, tables, COPY from S3, distribution/sort concepts, and workload management high level.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Lakehouse and External Table PatternsChoose query-in-lake vs load-into-warehouse, external schemas, lakehouse concepts, materialization tradeoffs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Cloud Warehouse TransformationsBuild staging, dimensional tables, marts, aggregate tables, incremental transformations, and materialized outputs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Warehouse Quality and ReconciliationValidate source-to-target counts, financial totals, duplicates, freshness, BI outputs, and reconciliation reports.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Module 3: Cloud Warehouse Performance and CostOptimize cloud warehouse performance and cost through partitioning, clustering, materialization, and cost governance.5 lessons
Lesson 1: Query Performance in Cloud WarehousesReview large scans, joins, aggregations, sorting, filters, concurrency, and expensive dashboards.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Partitioning, Clustering and SortingChoose partitioning, clustering, sort keys, distribution concepts, pruning, data skipping, and cloud-specific differences.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Materialization and Pre-AggregationChoose views, tables, materialized views, aggregate tables, dashboard acceleration, and refresh cost.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Warehouse Cost ControlControl compute, storage, query scans, concurrency, scheduling, result caching, wide tables, and retention.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 5: Milestone Project 4 - Cloud Warehouse MartBuild or design warehouse schemas, marts, validations, performance strategy, cost notes, and BI-ready outputs.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
6Phase 6 - Streaming and Event-Driven Data PipelinesUnderstand batch vs streaming, event-driven architecture, streaming services, failure modes, AWS/GCP/Azure patterns, and monitoring.2 modules9 lessons1–2 weeks
Module 1: Streaming FoundationsUnderstand event data, streaming architecture, cloud streaming services, and streaming risks.4 lessons
Lesson 1: Batch vs Streaming RevisitedClassify event data, near-real-time analytics, queues, streams, latency requirements, cost/complexity, and when not to stream.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Event-Driven ArchitectureDesign producers, topics/streams, consumers, messages, event schemas, delivery guarantees high level, and replay.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Streaming Ingestion ServicesMap AWS Kinesis, Firehose, Azure Event Hubs, Google Pub/Sub, managed ingestion, sinks, and service comparison.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Streaming Data Quality RisksHandle duplicates, late events, out-of-order events, schema changes, poison messages, dead-letter queues, and replay.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Module 2: Streaming Pipeline DesignDesign AWS, GCP, and Azure streaming patterns and monitoring metrics.5 lessons
Lesson 1: AWS Streaming PatternDesign producer, Kinesis/Firehose concept, S3 landing, Glue catalog, Athena query, and Redshift load concept.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 2: Google Streaming PatternCompare Pub/Sub, Dataflow, BigQuery, Cloud Storage, templates, and streaming analytics pattern.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Azure Streaming PatternCompare Event Hubs, Data Factory/Fabric concepts, Synapse/Fabric/Power BI integration, and stream processing concepts.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Streaming MonitoringDefine lag, failed events, throughput, error rates, dead-letter queues, replay strategy, and consumer failures.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 5: Mini Project 2 - Streaming Architecture DesignStudents design a streaming architecture for a realistic event use case.120 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
7Phase 7 - Security, Governance, Monitoring and Cost ControlSecure, govern, monitor, operate, and control cost in cloud data platforms.4 modules17 lessons1–2 weeks
Module 1: Cloud Security for Data EngineersApply IAM, access control, encryption, secrets, privacy, and sensitive data protections.4 lessons
Lesson 1: IAM FundamentalsDesign users, roles, policies, least privilege, service roles, temporary credentials, boundaries, and human/service access.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: Data Access ControlDesign bucket policies, object permissions, warehouse permissions, table/column/row access, and environment isolation.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 3: Encryption and SecretsHandle encryption at rest/in transit, KMS concept, secrets managers, environment variables, rotation, and leaks.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Privacy and Sensitive DataProtect PII using masking, anonymization, retention, deletion, audit logs, sensitive marts, and least-data principle.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Module 2: Governance and Data ContractsDesign governance ownership, data contracts, retention, lifecycle rules, and audit evidence.4 lessons
Lesson 1: Cloud Data GovernanceDefine ownership, stewardship, data definitions, catalog responsibilities, quality ownership, and access approval.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Data Contracts in Cloud PipelinesWrite schema expectations, producer/consumer agreements, breaking changes, contract testing, ownership, and alerts.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Retention and Lifecycle GovernanceCreate retention, archive, deletion, compliance, temporary expiry, cost, and governance policies.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: AuditabilityDesign access logs, job logs, load metadata, change history, lineage, and incident evidence.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Module 3: Monitoring and OperationsDefine logs, metrics, monitoring, incident response, and runbooks.4 lessons
Lesson 1: Cloud Logging and MetricsDefine job logs, query logs, pipeline logs, service metrics, dashboards, alerts, and retention.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Pipeline MonitoringMonitor freshness, run success, row counts, schema drift, failed jobs, quality failures, and query failures.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Incident ResponseRespond to failed ETL jobs, bad published data, access leaks, cost spikes, outages, failed loads, and recovery.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: RunbooksWrite normal operation, failed runs, reruns, backfills, access requests, troubleshooting, and escalation steps.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Module 4: Cost Engineering / FinOps for DataControl storage, query, ETL, warehouse, streaming, logs, transfer, and idle resource costs.5 lessons
Lesson 1: Cloud Data Cost DriversIdentify storage, query scans, ETL compute, warehouse compute, streaming ingestion, logs, transfer, and idle resources.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
30 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
22 min
Lesson 2: Storage Cost ControlUse lifecycle policies, compression, retention, archive tiers, temp deletion, small file control, and partitions.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Query and Warehouse Cost ControlOptimize partitions, columnar formats, query limits, pre-aggregation, sizing, scheduling, and dashboards.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 4: Cost Monitoring and BudgetsUse budgets, alerts, tagging, allocation, reports, chargeback/showback concept, and cost review cadence.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 5: Milestone Project 5 - Cloud Data Operations PackageProduce security, governance, monitoring, incident response, runbook, and cost-control assets.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
8Phase 8 - Multi-Cloud Translation and CapstoneTranslate AWS architectures to Azure and Google Cloud, choose a cloud strategy, prepare career answers, and complete the final capstone.2 modules7 lessons1–2 weeks
Module 1: Multi-Cloud TranslationTranslate AWS-first cloud data architectures into Azure/Microsoft Fabric and Google Cloud equivalents.4 lessons
Lesson 1: AWS to Azure TranslationTranslate S3 to ADLS/OneLake, Glue to Data Factory/Fabric, Redshift/Athena to Synapse/Fabric/SQL endpoints, IAM to Azure RBAC, CloudWatch to Azure Monitor, and Kinesis to Event Hubs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 2: AWS to Google Cloud TranslationTranslate S3 to Cloud Storage, Glue/Dataflow, Athena/Redshift to BigQuery, Kinesis to Pub/Sub, MWAA to Composer, CloudWatch to Cloud Monitoring.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
32 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
25 min
Lesson 3: Choosing a Cloud StrategyEvaluate company ecosystem, tools, talent, cost, compliance, BI integration, credits, maturity, and hiring market.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Lesson 4: Cloud Data Engineering Career ReadinessPrepare for portfolio, certifications awareness, interview scenarios, architecture explanation, cost/security questions, and production thinking.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional cloud platform workflow.
35 min
Practice Activity
Apply the lesson through a guided cloud data engineering exercise.
27 min
Module 2: Cloud Data Engineering CapstoneStudents design and build a cloud data platform MVP using an AWS-first implementation with Azure and Google Cloud translation notes.3 lessons
Lesson 1: Capstone OptionsChoose a realistic cloud data engineering capstone domain.55 minarticle1 pages
Choose Your Cloud Data Engineering Capstone
Review approved capstone options.
55 min
Lesson 2: Final Capstone - Cloud Data Engineering CapstoneStudents design and build a cloud data platform MVP using AWS-first implementation with Azure and Google Cloud translation notes.220 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
Lesson 3: Graduation Requirements and Portfolio OutcomeClarify completion requirements and portfolio outcomes.55 minarticle1 pages
Requirements and Portfolio Checklist
Summarize graduation requirements and portfolio assets.
55 min
Tools and skills
Build skill with the tools used in the work.
Understand how data engineering works in cloud environments.Use cloud storage concepts for raw and processed data.Understand managed databases and warehouse-style services.Choose basic compute options for data workloads.Apply access control and security basics.Deploy data scripts and pipeline workflows.Monitor logs, failures, and pipeline health.Understand cost-aware cloud data architecture.Prepare analytics-ready cloud data workflows.Build portfolio-ready cloud data engineering projects.
Projects and exercises
- Understand how data engineering works in cloud environments.
- Structured exercises
- Portfolio practice
Resources included
- Course resources
- Project guidance
- Learners building practical tech skills
- A willingness to practice consistently
Career relevance
Cloud Data Engineering supports practical career readiness.
Related Professional Diploma
View Professional DiplomaData Engineering
Learn how to build the pipelines, data models, warehouses, orchestration workflows, and cloud data systems that power analytics, reporting, machine learning, and AI products.
FAQ
Questions about this Short Course.
Short Course answers about scope, projects, support, and next steps.
No advanced cloud experience is required, but basic understanding of data pipelines, SQL, and Python will help.
Related Short Courses
View all Short CoursesContinue building connected skills.
School of Data & AIData AnalyticsBeginner to Intermediate
SQL for Data Analytics
Query databases, join tables, summarize records, and uncover business insights with SQL.
Learn the SQL skills data analysts use to extract, filter, join, group, and analyze data from relational databases.
From₦65,000
7 weeks - 6-8 hours/week
Understand tables, columns, rows, keys, and relationships.
Project included
Mentor review available
Related Professional Diploma
Data Engineering
School of Data & AIData & AIBeginner to Intermediate
Excel for Data Analytics
Turn raw spreadsheets into clean analysis, useful reports, and business-ready insights.
Master the Excel skills used by data analysts to clean, organize, calculate, summarize, visualize, and report business data with confidence.
From₦50,000
6 weeks - 5–8 hours /week
Clean and organize messy spreadsheet data.
Project included
Mentor review available
School of Data & AIData AnalyticsIntermediate
Power BI for Business Intelligence
Build interactive dashboards and business reports that make performance clear.
Learn to connect, clean, model, measure, visualize, and present business data using Power BI.
From₦85,000
8 weeks - 6-8 hours/week
Connect Power BI to different data sources.
Project included
Mentor review available
Professional Diploma application
Continue through Data Engineering.
This course is included in a Professional Diploma, so tuition enrollment is handled after the diploma application flow.