AI and Data in Business
Databricks Lakehouse Data Engineering
This in-depth course develops directly applicable capability in Databricks Lakehouse Data Engineering. It connects Lakehouse Architecture and Workspace Control, Ingestion with Auto Loader and Delta Lake, and Transformation Pipelines to the decisions, controls, and activities participants need to perform in their workplace.
Overview
Practical learning for workplace transfer.
This in-depth course develops directly applicable capability in Databricks Lakehouse Data Engineering. It connects Lakehouse Architecture and Workspace Control, Ingestion with Auto Loader and Delta Lake, and Transformation Pipelines to the decisions, controls, and activities participants need to perform in their workplace. The five-module curriculum progresses toward Lakehouse Pipeline Workshop, using evidence, scenarios, and work products appropriate to the subject.
Objectives
- Analyze lakehouse architecture and workspace control, including unity catalog, metastores, catalogs, schemas, and managed storage.
- Configure or structure ingestion with auto loader and delta lake, including cloudfiles discovery, schema inference, and rescued data.
- Evaluate transformation pipelines, including spark dataframe transformations and sql workflows.
- Manage orchestration and production operations, including databricks workflows jobs, tasks, parameters, and notifications.
- Apply lakehouse pipeline workshop, including design bronze, silver, and gold tables for a business dataset.
Target audience
- Professionals responsible for this subject area
- Managers, supervisors, and team leaders
- Analysts, specialists, engineers, or coordinators working with the relevant processes
- Project, implementation, assurance, or improvement team members
- Professionals preparing for broader responsibilities in this field
Program outline
A clear structure for the learning journey.
Program outline
Outline points are grouped in one designed block instead of being treated as separate module cards.
Module 1: Lakehouse Architecture and Workspace Control
Unity Catalog, metastores, catalogs, schemas, and managed storage
Compute policies, serverless options, and workspace boundaries
Medallion architecture and data-product ownership
Module 2: Ingestion with Auto Loader and Delta Lake
CloudFiles discovery, schema inference, and rescued data
Delta transaction log, schema enforcement, and time travel
Batch and streaming ingestion design choices
Module 3: Transformation Pipelines
Spark DataFrame transformations and SQL workflows
Delta Live Tables expectations and pipeline dependencies
Incremental processing, change data feed, and merge patterns
Module 4: Orchestration and Production Operations
Databricks Workflows jobs, tasks, parameters, and notifications
Cluster sizing, Photon, partitioning, and cost controls
Monitoring failed runs, data quality, lineage, and recovery
Module 5: Lakehouse Pipeline Workshop
Design bronze, silver, and gold tables for a business dataset
Configure quality rules and an incremental load
Review performance evidence and production-readiness controls
Materials provided
- ○ Course-specific presentation slides
- ○ Guided exercises, scenarios, or configured-environment activities appropriate to the subject
- ○ Course-specific worksheets, checklists, or calculation templates
- ○ Applied workplace case materials
- ○ 4D Certificate of Completion issued by 4D Training & Consultancy
- ○ Post-course support for implementation questions
Training Options
Programs can be delivered in-house, online, or in a blended format depending on your team's schedule, location, and learning objectives. When an external certificate or exam is included, certification rules and fees remain under the relevant awarding body's policies, while 4D provides the training and preparation support.
Why choose 4D
4D Training & Consultancy adapts the program to the client’s operating environment. Delivery combines structured explanation with subject-specific analysis, exercises, and implementation decisions so participants can transfer the learning to real responsibilities without implying vendor authorization.
Related courses
AI Agents and Workflow Automation for Business Operations
This course helps business and operations teams understand how AI agents and workflow automation can reduce repetitive work, improve handoffs, and support faster execution. Participants learn how to map processes, identify automation candidates, design human-in-the-loop controls, and manage risks before scaling AI-enabled workflows.
View courseAI Change Management and Adoption for Managers
This course helps managers lead teams through AI adoption with clarity, confidence, and responsible use. Participants learn how to address resistance, redesign work, set expectations, coach employees, define safe-use rules, and measure adoption without creating fear or unrealistic expectations.
View courseAI for Business Leaders and Department Managers
This course helps business leaders and department managers understand how artificial intelligence can be used responsibly across departments. Participants explore practical AI use cases, productivity opportunities, governance requirements, implementation risks, and decision-making considerations without needing a technical background.
View course