Understand ETL and ELT pipeline workflows.
School of Data & AIData EngineeringIntermediateIncluded in a Professional Diploma
ETL & ELT Pipelines
Build the pipelines that move raw data into useful analytics-ready datasets.
Learn how to extract, load, transform, validate, and prepare data from files, APIs, databases, and source systems for reporting, dashboards, and data products.
Duration
8 weeks - 6-8 hours/week
Project
Understand ETL and ELT pipeline workflows.
Support
Pricing and enrolment are handled through the Professional Diploma
Overview
A practical Short Course built around a visible project.
Learn how to move data from source systems into databases, warehouses, and analytics layers using practical ETL and ELT pipeline workflows.
Extract data from files, APIs, and databases.
Load data into databases and warehouse-style systems.
Transform raw data into clean, structured datasets.
Validate data quality during pipeline processing.
Handle pipeline errors, logs, and failed records.
Understand full refresh and incremental load patterns.
Prepare pipeline outputs for analytics and BI tools.
Design repeatable data movement workflows.
Build portfolio-ready data pipeline projects.
Course roadmap
What you will work through.
The sequence below is specific to this course. It shows the phases, modules, lessons, and page outlines that move you toward Understand ETL and ELT pipeline workflows..
1Phase 1 - Pipeline FoundationsBuild core pipeline thinking: pipeline components, ETL vs ELT, timing patterns, consumers, architecture, design principles, and failure modes.2 modules9 lessons1–2 weeks
Module 1: Understanding Data PipelinesUnderstand what pipelines are, how ETL/ELT works, timing choices, and downstream consumers.4 lessons
Lesson 1: What Is a Data Pipeline?Understand a data pipeline as a repeatable workflow that moves data from sources through extraction, loading, transformation, validation, scheduling, monitoring, and downstream consumption.85 minarticle6 pages
Welcome and Learning Objectives
Introduce the full pipeline concept.
8 min
Pipeline Building Blocks
Explain core pipeline components.
18 min
Pipeline Flow Example
Show a realistic data pipeline flow.
18 min
BI, Analytics, AI and ML Consumers
Connect pipelines to downstream consumers.
18 min
Reliability, Validation and Monitoring
Explain why pipeline operations matter.
18 min
Exercise - Draw a Business Data Pipeline
Students draw a pipeline for a realistic business domain.
23 min
Lesson 2: ETL vs ELTCompare extract-transform-load and extract-load-transform workflows, where transformation happens, warehouse-first design, Python-heavy ETL, SQL-heavy ELT, and tradeoffs.85 minarticle5 pages
Welcome and Learning Objectives
Introduce ETL and ELT.
8 min
ETL Explained
Explain extract-transform-load.
20 min
ELT Explained
Explain extract-load-transform.
20 min
Tradeoffs and Decision Rules
Compare ETL and ELT decisions.
18 min
Exercise - ETL or ELT Decision Matrix
Students choose ETL or ELT for different scenarios.
39 min
Lesson 3: Batch, Near Real-Time and StreamingClassify pipelines by latency needs, scheduled jobs, event streams, cost, complexity, and why batch is often enough.80 minarticle4 pages
Welcome and Learning Objectives
Introduce pipeline latency patterns.
8 min
Three Pipeline Timing Patterns
Explain timing patterns.
24 min
Cost and Complexity Tradeoffs
Explain tradeoffs.
18 min
Exercise - Timing Pattern Classifier
Students classify use cases.
30 min
Lesson 4: Pipeline ConsumersMap pipeline outputs to BI dashboards, analytics notebooks, ML, AI systems, operational reporting, data marts, reverse ETL, and stakeholder expectations.80 minarticle4 pages
Welcome and Learning Objectives
Introduce downstream consumers.
8 min
Common Consumers
Explain pipeline consumers.
20 min
Consumer Quality Expectations
Explain quality expectation mapping.
22 min
Exercise - Consumer Output Mapping
Students map outputs to consumers and expectations.
30 min
Module 2: Pipeline Architecture and DesignDesign source-to-target flows, layers, principles, and failure controls before writing pipeline code.5 lessons
Lesson 1: Source-to-Target ThinkingMap source systems to raw, staging, intermediate, curated, and mart targets with lineage and ownership.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 2: Pipeline LayersBreak pipelines into ingestion, staging, transformation, validation, publishing, monitoring, archiving, and quarantine layers.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 3: Pipeline Design PrinciplesApply reliability, repeatability, idempotency, observability, modularity, recoverability, documentation, and auditability.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Pipeline Failure ModesPlan for missing files, API failures, bad data, schema drift, duplicates, partial failures, broken transformations, stale dashboards, and wrong logic.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 5: Mini Project 1 - Pipeline Design BriefStudents choose one domain and produce a professional pipeline design brief.100 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
2Phase 2 - Data Extraction PatternsExtract data reliably from files, APIs, and databases with metadata, pagination, incremental logic, performance, retries, and logs.3 modules13 lessons1–2 weeks
Module 1: File-Based ExtractionDesign and build reliable file-drop pipelines with validation, archiving, reprocessing, and extraction logs.4 lessons
Lesson 1: File Drop PipelinesDesign daily files, folder conventions, naming, batch dates, source tracking, inventory, and missing-file handling.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Reading Files Reliably with PythonRead CSV, Excel, JSON with encoding issues, malformed files, missing columns, empty files, and schema checks.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 3: File Archiving and ReprocessingDesign raw archive, processed folder, rejected folder, reruns, originals preservation, auditability, and source evidence.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: File Extraction LogsGenerate logs with filename, size, row count, checksum/hash concept, timestamp, status, and error reason.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 2: API ExtractionExtract API data using endpoints, pagination, incremental filters, retries, and failure handling.4 lessons
Lesson 1: API Data SourcesInspect REST APIs, endpoints, query parameters, headers, keys, JSON responses, and source contracts.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: PaginationHandle page-based, cursor-based, limit/offset, next links, stopping conditions, and duplicate page risks.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 3: Incremental API ExtractionUse date filters, updated_since, cursors, high-watermarks, last successful run, and backfill windows.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 4: Rate Limits, Retries and FailuresHandle 429 errors, timeouts, retries, exponential backoff, partial success, provider outages, and retry limits.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Module 3: Database ExtractionExtract from relational databases safely with incremental logic, CDC awareness, and performance controls.5 lessons
Lesson 1: Extracting from Relational DatabasesUse connection strings, credentials, SELECT extraction, filtered extraction, chunked reads, query limits, and source safety.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 2: Incremental Database ExtractionUse updated_at, created_at, high-watermark table, last successful run, deleted records problem, and late updates.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 3: Change Data Capture ConceptsCompare CDC, inserts, updates, deletes, log-based capture, and simple incremental extraction.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Extraction PerformanceOptimize indexes, filters, date partitions, batch sizes, source load, extraction windows, and operational safety.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 5: Milestone Project 1 - Multi-Source Extraction PipelineBuild or simulate extraction from files, API, and database sources.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
3Phase 3 - Loading, Raw Storage and StagingLoad raw data safely, design raw storage, build production staging layers, add metadata, validate staging, and quarantine bad records.2 modules10 lessons1–2 weeks
Module 1: Loading Raw DataPreserve source data and load raw outputs into files, databases, or staging environments with metadata.4 lessons
Lesson 1: Raw Data Loading PrinciplesPreserve source data, load first, transform later, raw tables/files, source traceability, batch IDs, timestamps, and immutable raw concept.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Loading to FilesWrite CSV, JSON, Parquet concept, folder partitions, batch-based storage, naming, and source/date partitions.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Loading to DatabasesLoad raw tables, staging tables, append/replace loads, bulk loading, data type mapping, and handle failures.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Load MetadataTrack load_id, batch_id, run_id, source name, loaded_at, row count, status, error message, and duration.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 2: Production Staging LayerBuild staging models with standards, SQL implementation, validation, and rejected-record workflows.6 lessons
Lesson 1: Why Staging ExistsExplain raw vs staging, schema standardization, type casting, renaming, light cleaning, lineage, and one-to-one cleanup.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Staging Table StandardsDefine naming, data types, source columns, audit columns, unique keys, record status, and tracking standards.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Staging SQL ImplementationUse SELECT from raw, aliases, casting, trimming, date parsing, null normalization, and CASE expressions.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 4: Staging ValidationsWrite schema, row count, null, duplicate, accepted value, and source consistency checks.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 5: Rejected and Quarantined DataHandle invalid records, rejection reasons, quarantine tables, error files, review process, reprocessing, and reporting.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 6: Milestone Project 2 - Raw-to-Staging LoadBuild a loading workflow that stores raw extracted data, creates staging tables, validates data, quarantines bad records, and records load status.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
4Phase 4 - Production SQL and Transformation PatternsBuild production transformation logic with modular SQL/Python, deduplication, standardization, safe joins, marts, tests, contracts, and reviews.3 modules15 lessons2 weeks
Module 1: Transformation DesignDecide transformation responsibilities, tool placement, modular logic, and lineage documentation.4 lessons
Lesson 1: Transformation ResponsibilitiesClassify cleaning, standardization, deduplication, joins, rules, aggregations, intermediate models, and mart creation.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Python vs SQL TransformationsChoose Python or SQL based on task, location, warehouse compute, maintainability, performance, and team workflow.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Modular Transformation LogicDesign small transformation units, reusable functions, SQL models, intermediate outputs, and dependency management.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Transformation LineageDocument source-to-target mapping, dependencies, column mapping, business rules, and downstream impact.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 2: Production SQL Transformation PatternsUse SQL patterns for deduplication, standardization, business rules, safe joins, aggregates, and marts.5 lessons
Lesson 1: Deduplication with SQLUse ROW_NUMBER, partitioning, latest record logic, source priority, and deterministic deduplication.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 2: Standardization SQLStandardize category, code, status, date, currency, country/region, and null-like strings.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 3: Business Rule TransformationsUse CASE expressions, customer segments, order status flags, transaction risk flags, completion flags, and derived columns.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Joining Sources SafelyHandle join keys, one-to-many issues, unmatched records, many-to-many traps, relationship validation, and fanout risk.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 5: Aggregation and Mart CreationCreate daily, customer, product, cohort, KPI, and BI-ready mart aggregates.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Module 3: Transformation Testing and ReviewTest transformation logic, data contracts, SQL reviews, and transformation models.6 lessons
Lesson 1: Transformation Unit TestsTest inputs, expected outputs, edge cases, pure transformations, regression tests, and SQL model tests.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 2: Business Rule TestsTest statuses, categories, formulas, KPIs, thresholds, and business expectations.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Data Contract ChecksWrite source expectations, required columns, expected types, allowed changes, and producer-consumer agreements.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: SQL Code Review for PipelinesReview logic, joins, grain, performance, naming, and business rules in SQL transformations.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 5: Transformation ReviewReview logic, performance, downstream impact, documentation, and stakeholder approval.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 6: Milestone Project 3 - Production Transformation PipelineTransform staged data into curated and reporting-ready outputs.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
5Phase 5 - Incremental Pipelines and Change HandlingUpgrade full-refresh pipelines using high-watermarks, append-only design, merge/upsert logic, schema drift handling, change detection, deletes, history, idempotency, reruns, backfills, and recovery.3 modules13 lessons2 weeks
Module 1: Incremental Pipeline PatternsDesign incremental pipelines for growing data, cost, speed, reruns, freshness, and performance.4 lessons
Lesson 1: Why Incremental Pipelines MatterUnderstand growing data, cost, speed, reruns, daily loads, historical data, freshness, and warehouse performance.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: High-Watermark PatternUse last successful run, updated_at, created_at, extraction state, missed updates, late records, and state tables.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 3: Append-Only PipelinesDesign event, transaction, log, immutable data, duplicate risk, partitioning, and late-record handling.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Merge and Upsert PipelinesDesign changing records, natural keys, updates, conflicts, idempotency, merge strategy, and staging-to-final pattern.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Module 2: Change Detection and HistoryDetect schema drift, changed records, deletes, and historical changes.4 lessons
Lesson 1: Schema DriftDetect new columns, missing columns, changed types, renamed fields, breaking changes, and contract violations.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 2: Change DetectionCompare records, checksums/hashes, updated_at logic, changed fields, source-of-truth, and record_hash.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 3: Deletes and Soft DeletesHandle hard deletes, soft deletes, is_deleted flags, tombstones, tracking, compliance, and reporting impact.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Historical TrackingDesign snapshots, SCD, Type 1 updates, Type 2 inserts, audit history, and record versions.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Module 3: Idempotency and Safe RerunsDesign safe reruns, backfills, rollback, resume, and recovery from partial failure.5 lessons
Lesson 1: What Idempotency MeansExplain safe reruns, duplicate prevention, deterministic outputs, overwrite vs append, run IDs, and batch IDs.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Designing Rerunnable LoadsUse delete-and-reload partition, merge logic, temp tables, staging-to-final swap, transaction boundaries, and rerun audit.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 3: BackfillsPlan historical reloads, date windows, batch ranges, dependencies, tests, and rollback.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Recovery from Partial FailureRecover from failed extract/load/transformation using rollback, retry, resume, and checkpoints.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 5: Milestone Project 4 - Incremental Pipeline UpgradeUpgrade a full-refresh pipeline to support incremental extraction, change handling, safe reruns, and backfills.140 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
6Phase 6 - Data Quality, Reconciliation and ReliabilityAdd quality gates, validation, rejected record reporting, reconciliation, auditability, retries, checkpoints, dependencies, and operational readiness.3 modules13 lessons1–2 weeks
Module 1: Data Quality in PipelinesDefine quality dimensions, gates, automated validation, rejected records, and reports.4 lessons
Lesson 1: Data Quality DimensionsDefine completeness, uniqueness, validity, consistency, freshness, accuracy, timeliness, and lineage quality.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Quality GatesDesign warning vs failure, hard stops, soft alerts, thresholds, tolerance, downstream protection, and gate placement.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Automated ValidationBuild schema, row count, null, uniqueness, accepted value, relationship, and reconciliation checks.75 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
37 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
30 min
Lesson 4: Rejected Records and Quality ReportsReport invalid rows, rejection reasons, failed/warning checks, rejected records, counts, and stakeholder visibility.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Module 2: Reconciliation and AuditabilityReconcile row counts, financial totals, metrics, audit columns, and reports.4 lessons
Lesson 1: Row Count ReconciliationCompare source, loaded, transformed, rejected counts, batch reconciliation, and mismatch detection.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 2: Financial and Metric ReconciliationCompare source totals, transformed totals, dashboard totals, tolerance thresholds, and trust reports.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 3: Audit ColumnsAdd created_at, updated_at, loaded_at, processed_at, batch_id, run_id, source_file, record_hash, and effective dates.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Reconciliation ReportsGenerate validation summaries, failures, warnings, pass/fail thresholds, visibility, and audit trail.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 3: Pipeline Reliability EngineeringAdd retries, checkpoints, dependency management, operational readiness, and runbooks.5 lessons
Lesson 1: Retries and TimeoutsHandle transient failures, retries, exponential backoff, timeouts, retry limits, and escalation.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 2: CheckpointsSave progress, resume from steps, intermediate outputs, pipeline state, and avoid full restart.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Dependency ManagementManage upstream/downstream dependencies, task order, availability, dependency failures, and readiness checks.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Operational ReadinessWrite runbooks, ownership, support process, known failure modes, windows, rollback, and escalation.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 5: Milestone Project 5 - Reliable Pipeline Operations PackageAdd operational reliability to a pipeline.130 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
7Phase 7 - Monitoring, Documentation and HandoffDocument, monitor, hand off, review, and package pipelines for consumers, source owners, reviewers, and portfolios.3 modules13 lessons1 week
Module 1: Logging, Monitoring and AlertingAdd structured logs, run tracking, monitoring metrics, alerts, and incident response.4 lessons
Lesson 1: Pipeline LoggingUse structured logs, run IDs, task logs, error logs, row counts, duration, status, and context.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Pipeline Run TrackingCreate pipeline_run and task_run tables with status, start/end time, errors, retries, and records processed.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Monitoring MetricsTrack success rate, failure rate, duration, freshness, row anomalies, quality failures, and rejected counts.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Alerts and Incident ResponseDesign missing data, failed run, freshness, anomaly alerts, escalation, runbook, and ownership.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 2: Pipeline DocumentationWrite README, source-to-target mapping, data dictionary, and runbook documentation.4 lessons
Lesson 1: Pipeline READMEDocument purpose, sources, outputs, schedule, setup, run instructions, dependencies, and limitations.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Source-to-Target MappingDocument source fields, target fields, transformations, rules, owners, assumptions, and lineage.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 3: Data DictionaryCreate table descriptions, column descriptions, types, definitions, examples, quality notes, and refresh notes.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 4: Pipeline RunbookDocument normal run, failed run, backfill, rerun, alert response, ownership, and troubleshooting.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Module 3: Handoff and CollaborationCollaborate with analysts, BI teams, source owners, reviewers, and portfolio audiences.5 lessons
Lesson 1: Working with Analysts and BI TeamsUnderstand reporting needs, metric definitions, dashboards, change communication, expectations, and consumer docs.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 2: Working with Source System OwnersHandle schema changes, reliability, data contracts, ownership, communication, and breaking changes.60 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
30 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
22 min
Lesson 3: Code Review for PipelinesReview extraction, transformation, quality checks, idempotency, error handling, SQL, and documentation.70 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
35 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
27 min
Lesson 4: Portfolio PackagingPackage project story, architecture diagram, flow, screenshots/logs, sample outputs, README, limitations, and next improvements.65 minarticle3 pages
Overview and Learning Objectives
Introduce the lesson and clarify expected outcomes.
8 min
Concepts and Professional Workflow
Explain the concept through a professional pipeline workflow.
32 min
Practice Activity
Apply the lesson through a guided pipeline exercise.
25 min
Lesson 5: Mini Project 2 - Pipeline Handoff PackageCreate a complete handoff package for a pipeline.110 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
8Phase 8 - CapstoneBuild a complete production-aware ETL or ELT capstone pipeline.1 modules3 lessons1–2 weeks
Module 1: ETL and ELT Pipeline CapstoneStudents build a complete production-aware ETL/ELT pipeline with extraction, loading, staging, transformation, quality, reconciliation, monitoring, documentation, and presentation.3 lessons
Lesson 1: Capstone OptionsChoose a realistic ETL/ELT capstone domain.55 minarticle1 pages
Choose Your ETL and ELT Pipeline Capstone
Review approved capstone options.
55 min
Lesson 2: Final Capstone - ETL and ELT Pipeline CapstoneBuild a complete production-aware ETL or ELT pipeline from realistic sources to analytics-ready outputs with reliability and handoff documentation.220 minarticle2 pages
Project Brief
Explain the project scenario and expected output.
20 min
Review Checklist
Checklist for project quality.
20 min
Lesson 3: Graduation Requirements and Portfolio OutcomeClarify completion requirements and portfolio outputs.55 minarticle1 pages
Requirements and Portfolio Checklist
Summarize graduation requirements and portfolio assets.
55 min
Tools and skills
Build skill with the tools used in the work.
Understand ETL and ELT pipeline workflows.Extract data from files, APIs, and databases.Load data into databases and warehouse-style systems.Transform raw data into clean, structured datasets.Validate data quality during pipeline processing.Handle pipeline errors, logs, and failed records.Understand full refresh and incremental load patterns.Prepare pipeline outputs for analytics and BI tools.Design repeatable data movement workflows.Build portfolio-ready data pipeline projects.
Projects and exercises
- Understand ETL and ELT pipeline workflows.
- Structured exercises
- Portfolio practice
Resources included
- Course resources
- Project guidance
- Learners building practical tech skills
- A willingness to practice consistently
Career relevance
ETL & ELT Pipelines supports practical career readiness.
Related Professional Diploma
View Professional DiplomaData Engineering
Learn how to build the pipelines, data models, warehouses, orchestration workflows, and cloud data systems that power analytics, reporting, machine learning, and AI products.
FAQ
Questions about this Short Course.
Short Course answers about scope, projects, support, and next steps.
ETL means extract, transform, then load. ELT means extract, load, then transform. Modern warehouses often use ELT because transformation can happen inside the warehouse.
Related Short Courses
View all Short CoursesContinue building connected skills.
School of Data & AIData AnalyticsBeginner to Intermediate
SQL for Data Analytics
Query databases, join tables, summarize records, and uncover business insights with SQL.
Learn the SQL skills data analysts use to extract, filter, join, group, and analyze data from relational databases.
From₦65,000
7 weeks - 6-8 hours/week
Understand tables, columns, rows, keys, and relationships.
Project included
Mentor review available
Related Professional Diploma
Data Engineering
School of Data & AIData & AIBeginner to Intermediate
Excel for Data Analytics
Turn raw spreadsheets into clean analysis, useful reports, and business-ready insights.
Master the Excel skills used by data analysts to clean, organize, calculate, summarize, visualize, and report business data with confidence.
From₦50,000
6 weeks - 5–8 hours /week
Clean and organize messy spreadsheet data.
Project included
Mentor review available
School of Data & AIData AnalyticsIntermediate
Power BI for Business Intelligence
Build interactive dashboards and business reports that make performance clear.
Learn to connect, clean, model, measure, visualize, and present business data using Power BI.
From₦85,000
8 weeks - 6-8 hours/week
Connect Power BI to different data sources.
Project included
Mentor review available
Professional Diploma application
Continue through Data Engineering.
This course is included in a Professional Diploma, so tuition enrollment is handled after the diploma application flow.