Software Development
Python Data Pipelines with Apache Airflow
Cron jobs and hand-run scripts stop scaling the moment one load depends on another. This course teaches Airflow as an orchestration discipline: DAG authoring in Python, idempotent tasks, data intervals and backfills, sensors and external dependencies, retry and SLA policy, testing, and running a scheduler you can trust overnight.
Overview
Practical learning for workplace transfer.
The pipeline problem is rarely a missing transformation. It is the 3 a.m. failure that nobody sees until the morning report is empty, the rerun that double-counts because the task appends instead of replacing, and the history that cannot be reprocessed because the code assumes today's date. Airflow solves these only if the DAGs are written for them. This course builds that habit deliberately: every task is made idempotent against its data interval, dependencies are declared rather than implied by ordering, deferrable operators replace polling loops, and retry, timeout, and alerting policy is set per task. Participants also unit test DAGs so a broken import cannot stall the scheduler.
Prerequisites
Solid Python programming, SQL familiarity, and basic command-line and container concepts.
Objectives
- Explain the Airflow scheduler, executor, and metadata database interaction model.
- Author readable DAGs using the TaskFlow API, decorators, and task groups.
- Design idempotent tasks that produce the same result on every rerun and backfill.
- Coordinate cross-DAG and external dependencies with sensors and deferrable operators.
- Configure retries, timeouts, SLAs, and alerting so failures are seen before users are.
- Test, version, and deploy DAGs through a controlled promotion process.
Target audience
- Data engineers building and operating batch pipelines
- ETL developers migrating from cron, shell scripts, or legacy schedulers
- Analytics engineers responsible for warehouse refresh reliability
- Platform and DevOps engineers hosting and scaling an Airflow deployment
- ML engineers scheduling feature generation and retraining jobs
- Technical leads defining data reliability and on-call standards
Program outline
A clear structure for the learning journey.
Outline points are grouped in one designed block instead of being treated as separate module cards.
01Airflow Architecture and the Scheduling Model4 topics
- Scheduler, webserver, workers, metadata database, and the DAG parsing loop
- LocalExecutor, CeleryExecutor, and KubernetesExecutor selection criteria
- Logical date, data interval, and catchup: the source of most scheduling confusion
- Pools, priority weights, and concurrency limits at DAG and task level
02Authoring DAGs in Python4 topics
- TaskFlow decorators versus classic operators and when each reads better
- PythonOperator, BashOperator, SQL operators, and provider packages
- Passing data with XCom and knowing when to pass a reference instead
- Task groups, dynamic task mapping, and generating DAGs from configuration
03Idempotency, Backfills, and Reprocessing4 topics
- Writing tasks that replace a partition rather than append to a table
- Templating the data interval into paths, filters, and SQL predicates
- Running a controlled backfill without overwhelming the source system
- Late-arriving data, clearing task instances, and safe partial reruns
04Dependencies, Sensors, and External Systems4 topics
- Connections, hooks, and keeping credentials out of DAG source code
- File, table, and API sensors and the worker-slot cost of poking
- Deferrable operators and the triggerer for long waits without holding a slot
- Cross-DAG coordination with datasets, ExternalTaskSensor, and TriggerDagRun
05Reliability, Alerting, and Testing4 topics
- Retry counts, exponential backoff, execution timeouts, and circuit breaking
- SLA misses, callbacks, and routing alerts to the team that can act
- Data-quality checks as first-class tasks that stop a bad load downstream
- Unit testing DAG integrity, operator logic, and rendered templates in CI
06Deployment, Environments, and Operations4 topics
- Running Airflow with Docker Compose, Helm, or a managed service
- DAG distribution, dependency isolation, and avoiding a shared requirements file
- Promoting DAGs from development to staging to production with version control
- Scheduler health metrics, task duration trends, and capacity planning
Materials provided
- Course workbook, annotated code samples, and reference notes
- Hands-on lab environment and starter repositories
- Exercises, checklists, and reusable code templates
- 4D Certificate of Completion
- Post-course technical guidance
Training Options
Programs can be delivered in-house, online, or in a blended format depending on your team's schedule, location, and learning objectives. When an external certificate or exam is included, certification rules and fees remain under the relevant awarding body's policies, while 4D provides the training and preparation support.
Why choose 4D
4D maps your existing schedule before writing a single DAG — the cron entries, the shell scripts, the load that someone starts by hand each Sunday — and converts the most fragile of them during the workshop. Trainers work with the engineers who carry the pager, so the retry, SLA, and alert routing agreed in class reflects who actually responds at 3 a.m.
Related courses
Python Programming Fundamentals
A hands-on introduction to Python for engineers and analysts who need working code rather than tutorial fragments. Participants build fluency with data types, collections, functions, files, and error handling, then assemble a small program that reads real data and produces a usable report.
View courseIntermediate Python and Clean Code Practices
Designed for developers whose Python works but is difficult to change. The course covers dataclasses, generators, decorators, context managers, and type hints, then applies the refactoring moves that turn a nine-hundred-line module into small tested units guarded by black, ruff, and mypy.
View coursePython Automation and Scripting for Work Tasks
Aimed at teams losing hours every week to repetitive file, spreadsheet, and reporting work. Participants automate those tasks with Python: batch file handling, Excel and CSV processing, REST API calls, email alerts, and scheduled jobs that run unattended and report clearly when something fails.
View course