Software Development
Data Wrangling with Python and pandas
A hands-on pandas course for teams that spend more time repairing data than analysing it. Participants work through dtypes and memory, vectorized operations, merge and join semantics, missing-value strategy, reshaping, and time-series alignment, ending with cleaning code that runs unchanged next month.
Overview
Practical learning for workplace transfer.
The common failure mode is not a missing feature in pandas — it is a notebook that worked once and breaks the moment the source file changes. A column arrives as text instead of a number, a left join silently multiplies rows, a chained assignment updates a copy nobody sees, and the reported total quietly drifts. This course rebuilds the foundations that prevent all four: what an Index actually guarantees, why dtype selection decides both correctness and memory, how merge validation catches duplicate keys before they reach a report, and how to express a transformation without a Python loop. Participants finish by refactoring a recurring cleaning task into a tested script.
Prerequisites
Basic Python syntax including functions, lists, and dictionaries. No prior pandas experience is required.
Objectives
- Select dtypes deliberately so datasets stay correct and fit comfortably in memory.
- Replace row-by-row loops and apply calls with vectorized pandas operations.
- Apply merge, join, and concat with validated keys and predictable row counts.
- Choose a defensible missing-data strategy and document its effect on results.
- Reshape between wide and long form using pivot, melt, stack, and groupby.
- Package a cleaning workflow as a parameterised script with data-quality checks.
Target audience
- Data analysts moving from spreadsheets to Python
- Business intelligence and reporting specialists
- Finance, planning, and commercial analysts handling recurring data files
- Junior data engineers building ingestion and cleaning routines
- Researchers and scientists preparing experimental or field datasets
- Operations and quality teams consolidating data from multiple systems
Program outline
A clear structure for the learning journey.
Program outline
Outline points are grouped in one designed block instead of being treated as separate module cards.
Module 1: Series, DataFrames, dtypes, and Memory
The Index contract: alignment, duplicates, and why operations silently reorder
int, float, object, category, and nullable dtypes compared on real columns
Measuring memory with memory_usage and cutting it with category and downcasting
Views versus copies, chained indexing, and the SettingWithCopy trap
Module 2: Loading Messy Sources and Selecting Data
read_csv, read_excel, and JSON options for encodings, separators, and bad rows
Declaring dtype and parse_dates at load time instead of repairing afterwards
loc, iloc, boolean masks, query, and when each is the readable choice
Reading files larger than memory with chunksize and column selection
Module 3: Cleaning, Missing Data, and Type Correction
NaN, None, NaT, and pd.NA: how each propagates through arithmetic and comparison
Drop, fill, forward-fill, and interpolate: choosing by business meaning, not habit
String cleaning with the .str accessor, regex extraction, and normalisation
Detecting duplicates, outliers, and impossible values before they reach a report
Module 4: Combining Datasets Without Losing Rows
Inner, left, right, and outer joins and the row count each one implies
The validate argument as a contract against many-to-many key explosions
indicator, suffixes, and reconciling columns that exist on both sides
concat, merge_asof, and combining files that share only part of a schema
Module 5: Grouping, Reshaping, and Time Series
groupby split-apply-combine with agg, transform, and filter
pivot_table, melt, stack, and unstack for wide and long reshaping
DatetimeIndex, time zones, resample, and rolling windows
Window functions for running totals, ranks, and period-over-period change
Module 6: From Notebook to Repeatable Cleaning Pipeline
Refactoring notebook cells into named, tested transformation functions
Assertion-based data-quality gates on row counts, ranges, and key uniqueness
Parameterising paths, dates, and filters so the script runs unattended
Writing to Parquet and CSV with stable schemas and audit columns
Materials provided
- Course workbook, annotated code samples, and reference notes
- Hands-on lab environment and starter repositories
- Exercises, checklists, and reusable code templates
- 4D Certificate of Completion
- Post-course technical guidance
Training Options
Programs can be delivered in-house, online, or in a blended format depending on your team's schedule, location, and learning objectives. When an external certificate or exam is included, certification rules and fees remain under the relevant awarding body's policies, while 4D provides the training and preparation support.
Why choose 4D
4D runs this course on your own extracts — the ERP export with mixed date formats, the monthly file whose headers move, the join that keeps inflating totals. Trainers profile those files live, rebuild the cleaning logic in pandas alongside the analysts who own it, and leave the team with a tested script that replaces the manual repair step in their reporting cycle.
Related courses
Python Programming Fundamentals
A hands-on introduction to Python for engineers and analysts who need working code rather than tutorial fragments. Participants build fluency with data types, collections, functions, files, and error handling, then assemble a small program that reads real data and produces a usable report.
View courseIntermediate Python and Clean Code Practices
Designed for developers whose Python works but is difficult to change. The course covers dataclasses, generators, decorators, context managers, and type hints, then applies the refactoring moves that turn a nine-hundred-line module into small tested units guarded by black, ruff, and mypy.
View coursePython Automation and Scripting for Work Tasks
Aimed at teams losing hours every week to repetitive file, spreadsheet, and reporting work. Participants automate those tasks with Python: batch file handling, Excel and CSV processing, REST API calls, email alerts, and scheduled jobs that run unattended and report clearly when something fails.
View course