Understand data orchestration and workflow scheduling.
School of Data & AIData EngineeringIntermediateIncluded in a Professional Diploma
Orchestration with Airflow & dbt
Manage data workflows that run reliably, transform cleanly, and support analytics teams.
Learn how to use Airflow for scheduling and monitoring workflows, and dbt for structured SQL transformations, testing, documentation, and analytics-ready data models.
Duration
8 weeks - 6-8 hours/week
Project
Understand data orchestration and workflow scheduling.
Support
Pricing and enrolment are handled through the Professional Diploma
Overview
A practical Short Course built around a visible project.
Learn how to manage reliable data workflows using Airflow for orchestration and dbt for structured, tested, analytics-ready transformations.
Build Airflow workflows with tasks and dependencies.
Schedule and monitor pipeline runs.
Handle retries, failures, and workflow visibility.
Understand dbt models, sources, tests, and documentation.
Transform warehouse data into clean analytics layers.
Use SQL models for repeatable data transformations.
Apply testing and documentation to data workflows.
Connect orchestration and transformation concepts.
Build portfolio-ready Airflow and dbt workflow projects.
Course roadmap
What you will work through.
The sequence below is specific to this course. It shows the phases, modules, lessons, and page outlines that move you toward Understand data orchestration and workflow scheduling..
1Phase 1 - Orchestration and Modern Data Stack FoundationsBuild orchestration thinking: manual pipeline risks, task coordination, tool responsibilities, modern data stack design, and Airflow/dbt strategy.2 modules9 lessons1–2 weeks
Module 1: Why Orchestration MattersUnderstand why manual pipelines fail and how orchestration creates scheduled, visible, dependency-aware workflows.4 lessons
Lesson 1: The Problem with Manual PipelinesUnderstand why manual scripts, cron jobs, hidden failures, dependency confusion, and weak ownership create unreliable data workflows.85 minarticle6 pages
Welcome and Learning Objectives
Introduce why manual pipelines become dangerous.
8 min
Manual Scripts and Cron Jobs
Explain why early workflows are often manual.
18 min
Hidden Failures and No Visibility
Explain operational blindness.
18 min
Dependency Confusion
Explain why task order matters.
18 min
Weak Operational Ownership
Explain people and support issues.
18 min
Exercise - Manual Pipeline Risk Review
Students review a manual pipeline process and identify operational risks.
23 min
Lesson 2: What Is Orchestration?Understand orchestration as task scheduling, dependency management, retries, monitoring, state, history, ownership, and operational visibility.85 minarticle4 pages
Welcome and Learning Objectives
Introduce orchestration.
8 min
Orchestration Building Blocks
Explain the core concepts.
22 min
Why Airflow Is Useful
Connect orchestration to Airflow.
20 min
Exercise - Orchestrated Sales Workflow
Students draw an orchestrated workflow.
35 min
Lesson 3: Orchestration vs TransformationSeparate orchestration, transformation, Airflow, dbt, warehouse, extraction, loading, testing, reporting, and monitoring responsibilities.85 minarticle4 pages
Welcome and Learning Objectives
Introduce tool responsibility separation.
8 min
Layer Responsibilities
Explain the difference.
24 min
Avoiding Tool Misuse
Teach boundaries.
20 min
Exercise - Responsibility Classifier
Students classify pipeline steps.
35 min
Lesson 4: Modern Data Stack OverviewDesign a modern data stack with source systems, ingestion, raw/staging layers, warehouse/lake, dbt transformations, orchestration, BI, monitoring, documentation, and consumers.85 minarticle4 pages
Welcome and Learning Objectives
Introduce the modern data stack.
8 min
Modern Data Stack Components
Explain common components.
24 min
Domain Stack Examples
Show domain examples.
20 min
Exercise - Modern Data Stack Design
Students design a stack for a domain.
33 min
Module 2: Tooling StrategyUnderstand Airflow, dbt, Python scripts, setup needs, and tool-selection decisions.5 lessons
Lesson 1: Airflow OverviewExplain Airflow DAGs, operators, scheduler, web UI, logs, retries, and backfills.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: dbt OverviewExplain dbt SQL models, refs, sources, tests, docs, lineage, and analytics engineering workflow.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 3: Airflow vs dbt vs Python ScriptsChoose when to use Python scripts, Airflow, dbt, or combined workflows.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 4: Local Development SetupSet up Python environment, Docker overview, Airflow, dbt, database/warehouse target, project folders, and environment variables.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 5: Mini Project 1 - Orchestration Design BriefStudents choose a pipeline scenario and produce an orchestration design brief.110 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
2Phase 2 - Airflow FundamentalsBuild Airflow foundations: architecture, DAGs, tasks, operators, dependencies, scheduling, runs, catchup, backfills, and UI debugging.2 modules9 lessons1–2 weeks
Module 1: Airflow Core ConceptsUnderstand Airflow architecture, DAGs, tasks, operators, and dependency graph design.4 lessons
Lesson 1: Airflow ArchitectureExplain scheduler, webserver, metadata database, workers/executors, DAG files, task instances, logs, and UI.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: DAGsCreate DAGs with DAG ID, start date, schedule, catchup, tags, default args, and readable structure.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Tasks and OperatorsUse task concept, PythonOperator, BashOperator, EmptyOperator, SQL operators conceptually, custom tasks, and task boundaries.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 4: DependenciesDesign upstream/downstream tasks, linear dependencies, branching dependencies, task groups, and readable graph design.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Module 2: Scheduling and ExecutionUse schedules, DAG runs, task instances, states, retries, backfills, catchup, and Airflow UI debugging.5 lessons
Lesson 1: Scheduling ConceptsWrite schedules using schedule intervals, cron expressions, daily/weekly/monthly schedules, manual triggers, data interval, execution date, and timezone awareness.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 2: DAG Runs and Task InstancesInspect DAG run, task instance, states, queued/running/success/failed, retries, skipped tasks, and failures.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Backfills and CatchupUnderstand catchup behavior, historical runs, backfills, date ranges, risks, and rerun safety.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 4: Airflow UIUse DAG list, grid view, graph view, logs, task duration, run history, manual trigger, and debugging workflow.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 5: Milestone Project 1 - First Scheduled Airflow PipelineBuild a first scheduled Airflow DAG with tasks, dependencies, logs, failure handling, manual trigger support, and README.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
3Phase 3 - Airflow for Production Data PipelinesUse Airflow for real data workflows: extraction, loading, validation, transformation, retries, connections, sensors, branching, structure, logs, alerts, and operational risks.3 modules13 lessons1–2 weeks
Module 1: Building Real Data Workflows with AirflowOrchestrate extraction, loading, validation, and transformation tasks.4 lessons
Lesson 1: Orchestrating Extraction TasksCreate file, API, and database extraction tasks with boundaries, logs, and source metadata.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 2: Orchestrating Loading TasksAdd raw load, staging load, load metadata, file-to-database, database-to-warehouse, and run tracking.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Orchestrating Validation TasksAdd row count, schema, null, freshness, reconciliation checks, and task failure on quality issues.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 4: Orchestrating Transformation TasksAdd Python, SQL, dbt preview transformations, dependencies, logs, and validation before publishing.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Module 2: Airflow Reliability PatternsUse retries, variables, connections, sensors, branching, and conditional logic.4 lessons
Lesson 1: Retries and Failure HandlingConfigure retries, retry delay, exponential backoff, failure callbacks, permanent/transient failures, and when not to retry.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 2: Parameters, Variables and ConnectionsUse Airflow Variables, Connections, environment config, secrets concept, no hardcoded credentials, and environment separation.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 3: Sensors and External DependenciesUse file sensors, external task sensors, upstream data waiting, sensor risks, timeout settings, and missing data scenarios.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 4: Branching and Conditional LogicUse branching, conditional paths, skip states, quality gates, alert vs stop decisions, and downstream protection.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Module 3: Airflow Project Structure and Deployment AwarenessOrganize DAG projects, improve observability, design alerts, and diagnose operational risks.5 lessons
Lesson 1: Organizing DAG ProjectsStructure DAG folder, plugins, scripts, configs, reusable functions, and avoid giant DAG files.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: Logging and ObservabilityImprove task logs, structured messages, run IDs, row counts, duration, error messages, and traces.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Notifications and AlertsDesign email alerts, Slack/webhook concepts, failure alerts, SLA miss alerts, escalation, and alert fatigue controls.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: Airflow Operational RisksDiagnose scheduler issues, queued tasks, bad DAG imports, timezone issues, overloaded workers, dependency mistakes, and unsafe backfills.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 5: Milestone Project 2 - Airflow ETL Orchestration ProjectBuild an Airflow DAG that orchestrates extraction, raw load, validation, transformation, quality gates, final output, and failure behavior.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
4Phase 4 - dbt FundamentalsBuild dbt foundations: project structure, models, refs, sources, layered models, staging, intermediate, and marts.2 modules9 lessons1–2 weeks
Module 1: Introduction to dbtUnderstand why dbt exists and how it brings discipline to SQL transformations.4 lessons
Lesson 1: Why dbt ExistsExplain transformation chaos, scattered SQL, no tests, no lineage, no documentation, analytics engineering, and transformation as code.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: dbt Project StructureInspect dbt project, models folder, sources, seeds, snapshots, macros, profiles, and project naming.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: dbt ModelsCreate SQL select models, materializations, tables, views, incremental preview, naming, and dependency clarity.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 4: refs and sourcesUse source definitions, ref, source, dependency graph, lineage, no hardcoded table names, and dependency thinking.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Module 2: Building dbt Model LayersBuild source declarations, staging, intermediate, and mart models.5 lessons
Lesson 1: Source ModelsDeclare sources, freshness concept, source metadata, documentation, and ownership.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: Staging ModelsBuild one-to-one cleanup, renaming, type casting, standardizing, light cleaning, and source traceability.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Intermediate ModelsCreate reusable joins, business logic, entity models, avoid repeated logic, dependency layering, and modular SQL.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 4: Mart ModelsBuild reporting-ready tables, fact models, dimension models, BI-friendly outputs, and KPI-ready tables.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 5: Milestone Project 3 - dbt Transformation LayerBuild a dbt project with sources, staging, intermediate, mart models, refs, lineage, organization, and README.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
5Phase 5 - dbt Testing, Documentation and LineageAdd dbt tests, source freshness, model docs, lineage, exposures, standards, and quality reports.2 modules9 lessons1 week
Module 1: dbt TestingBuild generic tests, custom tests, freshness checks, and testing strategies.4 lessons
Lesson 1: Generic TestsUse not_null, unique, accepted_values, relationships, model-level, column-level, and severity.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 2: Custom TestsWrite custom SQL tests for business rules, metrics, reconciliation, thresholds, and test interpretation.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 3: Source FreshnessConfigure freshness, loaded_at, warnings, stale data, trust, and freshness SLAs.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: Testing StrategyChoose what to test, where to test, staging vs mart tests, warning vs error, avoid overload, protect outputs.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Module 2: dbt Documentation and LineageDocument models, generate docs, inspect lineage, define exposures, and create documentation standards.5 lessons
Lesson 1: Model DocumentationWrite model descriptions, column descriptions, definitions, owners, assumptions, examples, and limitations.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: dbt DocsGenerate docs, lineage graph, dependencies, documentation site, trust, and impact analysis.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 3: Exposures and BI ConsumersDocument dashboards as exposures, downstream dependencies, reporting impact, visibility, and BI ownership.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: Documentation StandardsCreate naming consistency, metric definitions, source descriptions, ownership, refresh notes, limitations, and consumer notes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 5: Milestone Project 4 - Tested and Documented dbt ProjectUpgrade the dbt project with tests, freshness, documentation, lineage, generated docs, and a quality report.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
6Phase 6 - Advanced dbt PatternsUse dbt materializations, incremental models, snapshots, selected runs, seeds, macros, packages, and project reviews.2 modules9 lessons1 week
Module 1: Materializations and PerformanceChoose dbt materializations and build incremental and snapshot models.4 lessons
Lesson 1: MaterializationsCompare view, table, incremental, ephemeral, performance tradeoffs, and cost impact.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: Incremental ModelsBuild incremental strategy, unique keys, updated_at filters, merge logic, full refresh, late data, and tests.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 3: SnapshotsUse SCD concepts, check strategy, timestamp strategy, current/historical records, and dimension history.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 4: Model Selection and RunsUse selected model runs, tags, dependencies, state-aware runs high level, build vs run vs test.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Module 2: Macros, Seeds and Reusable PatternsUse seeds, macros, packages, and project reviews to improve reuse and maintainability.5 lessons
Lesson 1: SeedsUse reference data, mapping tables, static datasets, accepted values, and business-controlled lists.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
30 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
22 min
Lesson 2: Macros and Reusable SQLCreate simple macros, DRY SQL, reusable transformations, audit columns, and readability review.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: Packages and ReuseEvaluate dbt packages, common utilities, package discipline, and dependency management.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
30 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
22 min
Lesson 4: dbt Project ReviewReview model structure, naming, tests, docs, materializations, performance, and maintainability.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 5: Milestone Project 5 - Advanced dbt WorkflowUpgrade the dbt project with materialization strategy, incremental model, snapshot/history, seed, macro, selected runs, performance notes, and docs.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
7Phase 7 - Airflow + dbt IntegrationOrchestrate dbt with Airflow and build integrated ELT workflows.2 modules9 lessons1 week
Module 1: Why Orchestrate dbt with Airflow?Understand when dbt alone is enough and when Airflow should coordinate broader workflows.4 lessons
Lesson 1: dbt Alone vs Airflow + dbtCompare dbt transformations, Airflow orchestration, extract/load before dbt, alerts, dependencies, and multi-system pipelines.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: Running dbt from AirflowRun dbt using BashOperator/Python operator approaches, commands, paths, environment config, failures, and logs.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 3: Orchestrating dbt StepsRun dbt run, test, docs generate, source freshness, model selection, task order, and quality gates.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 4: Handling dbt Failures in AirflowHandle failed tests, failed models, alerting, retries, stopping downstream tasks, warnings, and stakeholder impact.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Module 2: End-to-End ELT WorkflowBuild a complete workflow where Airflow extracts/loads data and dbt transforms/tests/publishes outputs.5 lessons
Lesson 1: Extract and Load Before dbtCreate extraction, raw loading, staging availability, handoff to dbt, completion checks, and freshness.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 2: dbt Transformation LayerRun source freshness, staging models, intermediate models, marts, tests, docs, and materializations as part of the workflow.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
37 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
30 min
Lesson 3: Publish and NotifyHandle final mart readiness, dashboard refresh trigger concept, success/failure notifications, consumers, and release notes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: End-to-End TraceabilityTrace through Airflow logs, dbt logs, dbt docs, lineage, run metadata, troubleshooting, and source-to-mart flow.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 5: Milestone Project 6 - Airflow + dbt ELT PipelineBuild an integrated ELT workflow where Airflow extracts/loads and dbt transforms/tests/documents marts.150 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
8Phase 8 - Monitoring, Reliability and OperationsOperate orchestrated data workflows with ownership, triage, runbooks, backfills, monitoring, alerts, change management, and production readiness.2 modules9 lessons1 week
Module 1: Operating Orchestrated Data WorkflowsDefine ownership, triage failures, write runbooks, and plan safe backfills.4 lessons
Lesson 1: Operational OwnershipDefine who owns DAGs, dbt models, failures, consumers, support process, and escalation.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
30 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
22 min
Lesson 2: Failure TriageDiagnose failed extraction, load, dbt model, dbt test, stale source, bad data, dashboard issue, and schema changes.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 3: RunbooksWrite normal run, manual rerun, backfill, failed task recovery, data quality failure, rollback, and alert response instructions.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: Backfills and ReprocessingPlan historical DAG runs, dbt full refresh, incremental backfills, partition reloads, risk controls, and validation.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Module 2: Monitoring and Data Quality OperationsDesign monitoring metrics, alerting, change management, and production readiness reviews.5 lessons
Lesson 1: Monitoring MetricsDefine DAG success rate, task duration, dbt test failure rate, source freshness, anomalies, rejected trends, and cost/performance signals.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 2: Alerting StrategyDesign failure, freshness, quality, warning vs critical, escalation routing, and alert fatigue controls.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 3: Change ManagementManage model changes, schema changes, source changes, downstream reports, release notes, stakeholder communication, and versioning.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
32 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
25 min
Lesson 4: Production Readiness ReviewReview scheduling, tests, docs, lineage, alerts, runbook, backfill plan, ownership, and maintainability.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional orchestration workflow.
35 min
Practice Activity
Apply the lesson through a guided orchestration exercise.
27 min
Lesson 5: Mini Project 2 - Data Workflow Operations PackageProduce operations assets for an Airflow + dbt workflow.120 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
9Phase 9 - CapstoneBuild a production-aware orchestrated data workflow using Airflow and dbt.1 modules3 lessons1–2 weeks
Module 1: Airflow and dbt Orchestration CapstoneStudents build a complete Airflow + dbt ELT workflow with extraction/loading, dbt transformations, tests, docs, lineage, marts, operations package, and presentation.3 lessons
Lesson 1: Capstone OptionsChoose a realistic Airflow and dbt orchestration capstone domain.55 minarticle1 pages
Choose Your Airflow and dbt Orchestration Capstone
Review approved capstone options.
55 min
Lesson 2: Final Capstone - Airflow and dbt Orchestration CapstoneBuild a production-aware orchestrated data workflow using Airflow and dbt.220 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
Lesson 3: Graduation Requirements and Portfolio OutcomeClarify completion requirements and portfolio assets.55 minarticle1 pages
Requirements and Portfolio Checklist
Summarize graduation requirements and portfolio outcomes.
55 min
Tools and skills
Build skill with the tools used in the work.
Understand data orchestration and workflow scheduling.Build Airflow workflows with tasks and dependencies.Schedule and monitor pipeline runs.Handle retries, failures, and workflow visibility.Understand dbt models, sources, tests, and documentation.Transform warehouse data into clean analytics layers.Use SQL models for repeatable data transformations.Apply testing and documentation to data workflows.Connect orchestration and transformation concepts.Build portfolio-ready Airflow and dbt workflow projects.
Projects and exercises
- Understand data orchestration and workflow scheduling.
- Structured exercises
- Portfolio practice
Resources included
- Course resources
- Project guidance
- Learners building practical tech skills
- A willingness to practice consistently
Career relevance
Orchestration with Airflow & dbt supports practical career readiness.
Related Professional Diploma
View Professional DiplomaData Engineering
Learn how to build the pipelines, data models, warehouses, orchestration workflows, and cloud data systems that power analytics, reporting, machine learning, and AI products.
FAQ
Questions about this Short Course.
Short Course answers about scope, projects, support, and next steps.
Airflow is used to schedule, monitor, and manage workflows. In data engineering, it helps run pipeline tasks in the right order and track failures.
Related Short Courses
View all Short CoursesContinue building connected skills.
School of Data & AIData AnalyticsBeginner to Intermediate
SQL for Data Analytics
Query databases, join tables, summarize records, and uncover business insights with SQL.
Learn the SQL skills data analysts use to extract, filter, join, group, and analyze data from relational databases.
From₦65,000
7 weeks - 6-8 hours/week
Understand tables, columns, rows, keys, and relationships.
Project included
Mentor review available
Related Professional Diploma
Data Engineering
School of Data & AIData & AIBeginner to Intermediate
Excel for Data Analytics
Turn raw spreadsheets into clean analysis, useful reports, and business-ready insights.
Master the Excel skills used by data analysts to clean, organize, calculate, summarize, visualize, and report business data with confidence.
From₦50,000
6 weeks - 5–8 hours /week
Clean and organize messy spreadsheet data.
Project included
Mentor review available
School of Data & AIData AnalyticsIntermediate
Power BI for Business Intelligence
Build interactive dashboards and business reports that make performance clear.
Learn to connect, clean, model, measure, visualize, and present business data using Power BI.
From₦85,000
8 weeks - 6-8 hours/week
Connect Power BI to different data sources.
Project included
Mentor review available
Professional Diploma application
Continue through Data Engineering.
This course is included in a Professional Diploma, so tuition enrollment is handled after the diploma application flow.