Plan complete data engineering projects.
School of Data & AIData EngineeringIntermediateIncluded in a Professional Diploma
Data Engineering Studio
Turn your data engineering skills into complete portfolio projects.
Apply Python, SQL, data modeling, warehousing, ETL, ELT, orchestration, dbt, cloud workflows, and documentation to build end-to-end data engineering projects.
Duration
8 weeks - 6-8 hours/week
Project
Plan complete data engineering projects.
Support
Pricing and enrolment are handled through the Professional Diploma
Overview
A practical Short Course built around a visible project.
Apply Python, SQL, data modeling, warehousing, pipelines, orchestration, dbt, and cloud workflows to build portfolio-ready data engineering projects.
Extract data from files, APIs, databases, and source systems.
Clean, validate, and transform raw datasets.
Design data models for analytics and reporting.
Build pipeline workflows from raw data to trusted outputs.
Apply warehouse and transformation concepts.
Use orchestration thinking to manage workflow steps.
Document data sources, assumptions, models, and decisions.
Prepare outputs for dashboards, analytics, or machine learning.
Build portfolio-ready data engineering projects.
Course roadmap
What you will work through.
The sequence below is specific to this course. It shows the phases, modules, lessons, and page outlines that move you toward Plan complete data engineering projects..
1Phase 1 - Capstone Discovery and Data Platform ScopeChoose and scope the right capstone problem before building: domain, sources, consumers, MVP boundaries, non-goals, risks, and concept brief.1 modules5 lessons1 week
Module 1: Choosing the Right Data Engineering ProblemSelect a realistic, valuable, and portfolio-worthy data engineering capstone problem.5 lessons
Lesson 1: What Makes a Strong Data Engineering Capstone?Learn what separates a serious data engineering capstone from a small script project: realistic sources, downstream consumers, warehouse outputs, quality risks, operations, and portfolio strength.90 minarticle5 pages
Welcome and Learning Objectives
Introduce the standard for a strong studio capstone.
8 min
Signals of a Strong Capstone
Explain the core ingredients.
20 min
Realism, Difficulty and Business Value
Teach how to score project ideas.
18 min
Examples of Strong Capstone Directions
Give realistic domain examples.
18 min
Exercise - Capstone Idea Ranking
Students review and score project ideas.
26 min
Lesson 2: Avoiding Weak Data Engineering ProjectsIdentify weak capstone ideas and upgrade them into stronger platform projects with validation, orchestration, modeling, incremental logic, consumers, monitoring, and documentation.85 minarticle4 pages
Welcome and Learning Objectives
Introduce weak project patterns.
8 min
Weak Project Patterns
Explain common weak capstone mistakes.
20 min
How to Upgrade a Weak Project
Show upgrade strategy.
22 min
Exercise - Weak Project Rewrite
Students rewrite weak pipeline ideas into stronger platform projects.
35 min
Lesson 3: Choosing a Business DomainChoose a capstone domain and identify realistic sources, consumers, outputs, quality risks, and platform scope.85 minarticle4 pages
Welcome and Learning Objectives
Introduce domain choice.
8 min
Approved Studio Domains
List and explain domain options.
22 min
Domain Selection Criteria
Explain how to choose.
20 min
Exercise - Domain Selection Brief
Students choose a capstone domain and identify sources and consumers.
35 min
Lesson 4: Defining Scope and Non-GoalsDefine MVP scope, must-have pipelines, nice-to-have features, simulations, build decisions, documentation scope, and overbuild boundaries.85 minarticle4 pages
Welcome and Learning Objectives
Introduce scope control.
8 min
MVP Scope for a Data Platform
Explain MVP thinking.
20 min
Must-Haves, Nice-to-Haves and Non-Goals
Teach scope categories.
22 min
Exercise - Capstone v1 Scope and Non-Goals
Students define scope and non-goals.
35 min
Lesson 5: Mini Project 1 - Data Platform Concept BriefStudents define their capstone concept, business value, scope, risks, sources, consumers, and expected outputs.110 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
2Phase 2 - Requirements, Architecture and Data ModelingConvert the capstone idea into requirements, architecture, tooling decisions, source models, warehouse layers, dimensional models, marts, KPIs, and risk register.3 modules13 lessons1–2 weeks
Module 1: Data Platform RequirementsCapture business, source, quality, and operational requirements for the capstone platform.4 lessons
Lesson 1: Business and Analytics RequirementsDefine stakeholders, BI requirements, analytics requirements, AI/ML downstream needs, freshness, granularity, and metrics.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Source System RequirementsDocument source ownership, schema expectations, data contracts, extraction frequency, reliability, drift risk, and access.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 3: Data Quality RequirementsDefine completeness, validity, uniqueness, freshness, consistency, reconciliation, accepted values, and relationships.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 4: Operational RequirementsDefine schedule, retries, monitoring, alerts, backfills, reruns, support ownership, and incident response.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 2: Architecture DesignDesign source-to-consumer architecture, architecture style, tooling matrix, and environment strategy.4 lessons
Lesson 1: Source-to-Consumer ArchitectureDraw source systems, ingestion, raw storage, staging, transformations, warehouse, marts, consumers, and monitoring.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: Lake, Warehouse, or Hybrid ArchitectureChoose data lake, warehouse, lakehouse concept, serverless SQL, marts, and cost/complexity tradeoffs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Tooling DecisionsChoose Python, SQL, dbt, Airflow, cloud storage, serverless query, warehouse, BI output, and monitoring tools.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 4: Environment and Deployment DesignPlan local dev, cloud dev, staging/production concepts, environment variables, secrets, service accounts, and reproducible setup.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 3: Data Modeling BlueprintCreate source models, warehouse layer models, dimensional models, and marts/KPI designs.5 lessons
Lesson 1: Source ModelingDocument source tables, files, APIs, schemas, keys, relationships, and source limitations.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Warehouse Layer ModelingMap raw, staging, intermediate, curated, mart layers, naming standards, and lineage.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Dimensional ModelingDesign facts, dimensions, grain, conformed dimensions, date dimensions, factless facts, and snapshots.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 4: Metric and Mart DesignDefine KPI definitions, aggregate tables, reporting marts, BI-ready outputs, consistency, and dashboard consumers.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 5: Milestone Project 1 - Data Platform BlueprintProduce the complete blueprint for the capstone platform.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
3Phase 3 - Build Sprint 1: Ingestion and Raw LayerSet up the repository, simulate sources, build ingestion pipelines, load raw data, capture metadata, and demonstrate ingestion evidence.2 modules9 lessons1–2 weeks
Module 1: Project Setup and Source SimulationPrepare the repository, source datasets, configuration, secrets, and sprint plan.4 lessons
Lesson 1: Repository SetupSet up project structure, src, dags, dbt project, configs, tests, docs, samples, and logs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Source Data PreparationPrepare sample files, simulated API, source database, event logs, seed data, and source realism.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Configuration and SecretsUse config files, .env, credentials, local vs cloud config, safe defaults, and secret safety.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 4: Build Plan and Sprint BoardCreate build milestones, tasks, dependencies, blockers, review points, and acceptance criteria.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 2: Ingestion LayerBuild file, API, database, and raw loading workflows with metadata, errors, and logs.5 lessons
Lesson 1: File IngestionImplement discovery, naming patterns, batch dates, archive folders, bad file handling, and source metadata.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: API or Simulated API IngestionImplement REST extraction, pagination, retries, rate limits, incremental extraction, and API logs.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 3: Database ExtractionImplement source database connection, query extraction, incremental extraction, chunked reads, and connection error handling.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 4: Raw Layer LoadingLoad raw files/tables using batch ID, loaded_at, source tracking, and immutable raw concept.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 5: Milestone Project 2 - Ingestion and Raw Layer DemoDemonstrate ingestion from multiple sources and raw layer loading with logs and metadata.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
4Phase 4 - Build Sprint 2: Staging, Transformations and Warehouse ModelsBuild staging models, validation, rejected record handling, intermediate models, facts/dimensions, marts, and lineage.2 modules9 lessons1–2 weeks
Module 1: Staging LayerCreate source-to-staging models, schema checks, rejected record handling, and staging documentation.4 lessons
Lesson 1: Source-to-Staging ModelsStandardize names, cast types, perform light cleaning, preserve traceability, and follow staging naming conventions.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: Schema and Contract ChecksCheck required columns, expected types, missing columns, schema drift, and source contract violations.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Rejected RecordsHandle invalid rows, rejection reasons, quarantine table/file, bad data review, and reprocessing.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 4: Staging DocumentationDocument source mapping, column dictionary, source assumptions, known issues, and owner notes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 2: Transformation LayerBuild intermediate models, dimensional models, marts, dependencies, and lineage.5 lessons
Lesson 1: Intermediate ModelsBuild reusable joins, business logic, entity models, event models, and avoid repeated SQL.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 2: Dimensional ModelsBuild fact tables, dimension tables, grain, surrogate keys, date dimension, and conformed dimensions.80 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
40 min
Practice Activity
Apply the lesson through a guided studio activity.
32 min
Lesson 3: Mart ModelsBuild BI-ready tables, KPI tables, aggregate tables, friendly columns, and dashboard consumption outputs.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 4: Lineage and Model DependenciesCreate source-to-target mapping, dbt refs, dependency graph, model order, and downstream impact notes.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 5: Milestone Project 3 - Warehouse Model and Mart DemoDemonstrate staging, intermediate, dimensional, and mart outputs with lineage and queries.150 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
5Phase 5 - Build Sprint 3: Quality, Incremental Logic and OrchestrationAdd data quality tests, reconciliation, quality reports, incremental logic, safe reruns, backfills, and Airflow orchestration.3 modules13 lessons1–2 weeks
Module 1: Data Quality LayerDesign and implement data quality tests, reconciliation, and quality reporting.4 lessons
Lesson 1: Quality Test DesignDesign uniqueness, not-null, accepted values, relationships, row counts, freshness, and reconciliation tests.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: dbt Tests and Custom SQL TestsImplement generic tests, custom tests, severity, warnings vs errors, and test documentation.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 3: ReconciliationReconcile source totals, staging totals, mart totals, financial totals, thresholds, and mismatches.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 4: Data Quality ReportingGenerate quality summary, failed tests, warnings, rejected records, run-level metrics, and stakeholder visibility.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Module 2: Incremental Loads, Backfills and RerunsImplement incremental strategy, safe reruns, and backfill planning.4 lessons
Lesson 1: Incremental StrategyChoose append-only, upsert, merge, high-watermark, partition reload, and full-refresh fallback strategies.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: Implementing Incremental ModelsImplement updated_at filters, unique keys, incremental dbt models, source state tracking, and late-arriving data handling.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 3: Safe RerunsDesign idempotency, rerunning failed batches, duplicate prevention, temp/staging swap, and batch IDs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 4: Backfill PlanPlan historical reload, date windows, dependency order, validation after backfill, and rollback.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Module 3: Airflow OrchestrationCreate DAG design, ingestion tasks, dbt orchestration, notifications, and run tracking.5 lessons
Lesson 1: DAG DesignDesign task boundaries, dependencies, scheduling, retries, task groups, and quality gates.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 2: Orchestrating Ingestion and LoadingAdd extract tasks, raw load tasks, metadata logging, failure handling, and retries.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 3: Orchestrating dbtAdd dbt run, dbt test, dbt docs, source freshness, and failure behavior.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
37 min
Practice Activity
Apply the lesson through a guided studio activity.
30 min
Lesson 4: Notifications and Run TrackingAdd success alerts, failure alerts, run summary, task logs, metadata table, and operational visibility.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 5: Milestone Project 4 - Orchestrated Quality-Controlled PipelineDemonstrate quality-controlled, incremental, orchestrated pipeline behavior.150 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
6Phase 6 - Cloud, Security, Monitoring and OperationsDesign cloud deployment, multi-cloud translation, access/security, governance, monitoring, incidents, cost review, and runbook.3 modules13 lessons1–2 weeks
Module 1: Cloud Deployment DesignMap local platform design to cloud storage, query, warehouse, deployment, and multi-cloud architecture.4 lessons
Lesson 1: Cloud Storage LayoutDesign raw, staging, curated, archive zones, partitions, and lifecycle rules.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Cloud Warehouse / Query LayerMap warehouse schemas, external tables, serverless SQL, marts, BI access, and compute/cost tradeoffs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Deployment PlanCreate infrastructure choices, environment variables, secrets, service accounts, network considerations, and managed services.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 4: Multi-Cloud TranslationTranslate AWS implementation, Azure equivalent, GCP equivalent, tradeoffs, and employer ecosystem.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 2: Security and GovernanceCreate access control, sensitive data handling, governance ownership, and risk notes.4 lessons
Lesson 1: Access ControlDesign least privilege, role-based access, bucket/table permissions, department access, and service accounts.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Sensitive Data HandlingIdentify PII, masking, anonymization, retention, deletion, and audit logs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 3: Data GovernanceDefine ownership, data definitions, approval process, quality responsibility, documentation standards, and data contracts.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 4: Compliance and Risk NotesWrite privacy expectations, auditability, retention, source agreements, user/customer data risk, and operational risk.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 3: Monitoring and Cost ControlDefine monitoring metrics, incident response, cost review, and operations handoff.5 lessons
Lesson 1: Monitoring MetricsTrack pipeline success, freshness, row count anomalies, quality failures, duration, volume, and cost.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Alerts and Incident ResponseRespond to failed pipeline, stale mart, failed checks, bad data, cost spike, and access issue.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 3: Cost ReviewEstimate storage, compute, query, orchestration, log costs, and optimization options.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 4: Runbook and Operations HandoffDocument normal run, failed run, manual rerun, backfill, access request, troubleshooting, and ownership.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 5: Milestone Project 5 - Production Readiness ReviewProduce a production readiness package for cloud, security, monitoring, cost, and operations.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
7Phase 7 - BI, Analytics, AI Readiness and Portfolio PackagingPrepare downstream consumer outputs, documentation, case study, technical evidence, defense answers, and final demo.2 modules8 lessons1 week
Module 1: Serving Downstream ConsumersValidate BI, analytics, data science, AI readiness, and consumer-facing documentation.4 lessons
Lesson 1: BI ReadinessValidate Power BI-ready marts, semantic considerations, dashboard queries, metrics, date dimensions, and friendly columns.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 2: Analytics ReadinessSupport self-service, documented datasets, clean joins, query examples, safe filters, and known limitations.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 3: Data Science and AI ReadinessIdentify feature-ready datasets, RAG/AI preparation, event data, history, quality expectations, and lineage.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 4: Consumer DocumentationWrite table guide, KPI guide, query examples, dashboard notes, freshness notes, and caveats.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Module 2: Portfolio Case Study and Technical DefensePrepare the case study, evidence, technical defense, and final demo.4 lessons
Lesson 1: Writing the Case StudyDraft problem, architecture, sources, transformations, orchestration, checks, cloud plan, outcomes, and limitations.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 2: Showing Technical EvidenceSelect architecture diagrams, DAG screenshots, dbt lineage, SQL models, quality report, logs, and outputs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
32 min
Practice Activity
Apply the lesson through a guided studio activity.
25 min
Lesson 3: Technical Review PreparationPrepare answers on architecture, model choices, reruns, data quality, failures, and cost controls.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
Lesson 4: Final Demo PreparationRehearse demo flow, repo walkthrough, DAG walkthrough, dbt walkthrough, marts, operations package, and clarity.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a capstone delivery workflow.
35 min
Practice Activity
Apply the lesson through a guided studio activity.
27 min
8Phase 8 - Final Capstone and Technical DefenseBuild and defend a complete data platform MVP combining ingestion, warehousing, transformations, orchestration, quality, monitoring, cloud readiness, documentation, and downstream consumption.1 modules3 lessons1 week
Module 1: Data Engineering CapstoneComplete and present the final end-to-end data engineering capstone.3 lessons
Lesson 1: Capstone OptionsChoose or confirm a serious final data platform option.60 minarticle1 pages
Choose Your Data Engineering Capstone
Review approved capstone options.
60 min
Lesson 2: Final Project - Data Engineering CapstoneStudents build and defend a complete data platform MVP.260 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
Lesson 3: Graduation Requirements and Portfolio OutcomeClarify studio completion requirements and portfolio assets.60 minarticle1 pages
Requirements and Portfolio Checklist
Summarize graduation requirements and portfolio outcomes.
60 min
Tools and skills
Build skill with the tools used in the work.
Plan complete data engineering projects.Extract data from files, APIs, databases, and source systems.Clean, validate, and transform raw datasets.Design data models for analytics and reporting.Build pipeline workflows from raw data to trusted outputs.Apply warehouse and transformation concepts.Use orchestration thinking to manage workflow steps.Document data sources, assumptions, models, and decisions.Prepare outputs for dashboards, analytics, or machine learning.Build portfolio-ready data engineering projects.
Projects and exercises
- Plan complete data engineering projects.
- Structured exercises
- Portfolio practice
Resources included
- Course resources
- Project guidance
- Learners building practical tech skills
- A willingness to practice consistently
Career relevance
Data Engineering Studio supports practical career readiness.
Related Professional Diploma
View Professional DiplomaData Engineering
Learn how to build the pipelines, data models, warehouses, orchestration workflows, and cloud data systems that power analytics, reporting, machine learning, and AI products.
FAQ
Questions about this Short Course.
Short Course answers about scope, projects, support, and next steps.
It is a project-focused course where learners combine Python, SQL, pipelines, warehousing, orchestration, and cloud concepts to build complete data engineering projects.
Related Short Courses
View all Short CoursesContinue building connected skills.
School of Data & AIData AnalyticsBeginner to Intermediate
SQL for Data Analytics
Query databases, join tables, summarize records, and uncover business insights with SQL.
Learn the SQL skills data analysts use to extract, filter, join, group, and analyze data from relational databases.
From₦65,000
7 weeks - 6-8 hours/week
Understand tables, columns, rows, keys, and relationships.
Project included
Mentor review available
Related Professional Diploma
Data Engineering
School of Data & AIData & AIBeginner to Intermediate
Excel for Data Analytics
Turn raw spreadsheets into clean analysis, useful reports, and business-ready insights.
Master the Excel skills used by data analysts to clean, organize, calculate, summarize, visualize, and report business data with confidence.
From₦50,000
6 weeks - 5–8 hours /week
Clean and organize messy spreadsheet data.
Project included
Mentor review available
School of Data & AIData AnalyticsIntermediate
Power BI for Business Intelligence
Build interactive dashboards and business reports that make performance clear.
Learn to connect, clean, model, measure, visualize, and present business data using Power BI.
From₦85,000
8 weeks - 6-8 hours/week
Connect Power BI to different data sources.
Project included
Mentor review available
Professional Diploma application
Continue through Data Engineering.
This course is included in a Professional Diploma, so tuition enrollment is handled after the diploma application flow.