Software Development
Python Data Pipelines with Apache Airflow
Cron jobs and hand-run scripts stop scaling the moment one load depends on another. This course teaches Airflow as an orchestration discipline: DAG authoring in Python, idempotent tasks, data intervals and backfills, sensors and external dependencies, retry and SLA policy, testing, and running a scheduler you can trust overnight.
Overview
Practical learning for workplace transfer.
The pipeline problem is rarely a missing transformation. It is the 3 a.m. failure that nobody sees until the morning report is empty, the rerun that double-counts because the task appends instead of replacing, and the history that cannot be reprocessed because the code assumes today's date. Airflow solves these only if the DAGs are written for them. This course builds that habit deliberately: every task is made idempotent against its data interval, dependencies are declared rather than implied by ordering, deferrable operators replace polling loops, and retry, timeout, and alerting policy is set per task. Participants also unit test DAGs so a broken import cannot stall the scheduler.
Prerequisites
Solid Python programming, SQL familiarity, and basic command-line and container concepts.
Objectives
- Explain the Airflow scheduler, executor, and metadata database interaction model.
- Author readable DAGs using the TaskFlow API, decorators, and task groups.
- Design idempotent tasks that produce the same result on every rerun and backfill.
- Coordinate cross-DAG and external dependencies with sensors and deferrable operators.
- Configure retries, timeouts, SLAs, and alerting so failures are seen before users are.
- Test, version, and deploy DAGs through a controlled promotion process.
Target audience
- Data engineers building and operating batch pipelines
- ETL developers migrating from cron, shell scripts, or legacy schedulers
- Analytics engineers responsible for warehouse refresh reliability
- Platform and DevOps engineers hosting and scaling an Airflow deployment
- ML engineers scheduling feature generation and retraining jobs
- Technical leads defining data reliability and on-call standards
Program outline
A clear structure for the learning journey.
Program outline
Outline points are grouped in one designed block instead of being treated as separate module cards.
Module 1: Airflow Architecture and the Scheduling Model
Scheduler, webserver, workers, metadata database, and the DAG parsing loop
LocalExecutor, CeleryExecutor, and KubernetesExecutor selection criteria
Logical date, data interval, and catchup: the source of most scheduling confusion
Pools, priority weights, and concurrency limits at DAG and task level
Module 2: Authoring DAGs in Python
TaskFlow decorators versus classic operators and when each reads better
PythonOperator, BashOperator, SQL operators, and provider packages
Passing data with XCom and knowing when to pass a reference instead
Task groups, dynamic task mapping, and generating DAGs from configuration
Module 3: Idempotency, Backfills, and Reprocessing
Writing tasks that replace a partition rather than append to a table
Templating the data interval into paths, filters, and SQL predicates
Running a controlled backfill without overwhelming the source system
Late-arriving data, clearing task instances, and safe partial reruns
Module 4: Dependencies, Sensors, and External Systems
Connections, hooks, and keeping credentials out of DAG source code
File, table, and API sensors and the worker-slot cost of poking
Deferrable operators and the triggerer for long waits without holding a slot
Cross-DAG coordination with datasets, ExternalTaskSensor, and TriggerDagRun
Module 5: Reliability, Alerting, and Testing
Retry counts, exponential backoff, execution timeouts, and circuit breaking
SLA misses, callbacks, and routing alerts to the team that can act
Data-quality checks as first-class tasks that stop a bad load downstream
Unit testing DAG integrity, operator logic, and rendered templates in CI
Module 6: Deployment, Environments, and Operations
Running Airflow with Docker Compose, Helm, or a managed service
DAG distribution, dependency isolation, and avoiding a shared requirements file
Promoting DAGs from development to staging to production with version control
Scheduler health metrics, task duration trends, and capacity planning
Materials provided
- Course workbook, annotated code samples, and reference notes
- Hands-on lab environment and starter repositories
- Exercises, checklists, and reusable code templates
- 4D Certificate of Completion
- Post-course technical guidance
Training Options
Programs can be delivered in-house, online, or in a blended format depending on your team's schedule, location, and learning objectives. When an external certificate or exam is included, certification rules and fees remain under the relevant awarding body's policies, while 4D provides the training and preparation support.
Why choose 4D
4D maps your existing schedule before writing a single DAG — the cron entries, the shell scripts, the load that someone starts by hand each Sunday — and converts the most fragile of them during the workshop. Trainers work with the engineers who carry the pager, so the retry, SLA, and alert routing agreed in class reflects who actually responds at 3 a.m.
Related courses
Python Programming Fundamentals
A hands-on introduction to Python for engineers and analysts who need working code rather than tutorial fragments. Participants build fluency with data types, collections, functions, files, and error handling, then assemble a small program that reads real data and produces a usable report.
View courseIntermediate Python and Clean Code Practices
Designed for developers whose Python works but is difficult to change. The course covers dataclasses, generators, decorators, context managers, and type hints, then applies the refactoring moves that turn a nine-hundred-line module into small tested units guarded by black, ruff, and mypy.
View coursePython Automation and Scripting for Work Tasks
Aimed at teams losing hours every week to repetitive file, spreadsheet, and reporting work. Participants automate those tasks with Python: batch file handling, Excel and CSV processing, REST API calls, email alerts, and scheduled jobs that run unattended and report clearly when something fails.
View course