AI and Data in Business
Databricks Lakehouse Data Engineering
This in-depth course develops directly applicable capability in Databricks Lakehouse Data Engineering. It connects Lakehouse Architecture and Workspace Control, Ingestion with Auto Loader and Delta Lake, and Transformation Pipelines to the decisions, controls, and activities participants need to perform in their workplace.
Overview
Practical learning for workplace transfer.
This in-depth course develops directly applicable capability in Databricks Lakehouse Data Engineering. It connects Lakehouse Architecture and Workspace Control, Ingestion with Auto Loader and Delta Lake, and Transformation Pipelines to the decisions, controls, and activities participants need to perform in their workplace. The five-module curriculum progresses toward Lakehouse Pipeline Workshop, using evidence, scenarios, and work products appropriate to the subject.
Prerequisites
Relevant workplace familiarity is recommended; exact technical prerequisites can be adapted to the participant group.
Objectives
- Analyze lakehouse architecture and workspace control, including unity catalog, metastores, catalogs, schemas, and managed storage.
- Configure or structure ingestion with auto loader and delta lake, including cloudfiles discovery, schema inference, and rescued data.
- Evaluate transformation pipelines, including spark dataframe transformations and sql workflows.
- Manage orchestration and production operations, including databricks workflows jobs, tasks, parameters, and notifications.
- Apply lakehouse pipeline workshop, including design bronze, silver, and gold tables for a business dataset.
Target audience
- Professionals responsible for this subject area
- Managers, supervisors, and team leaders
- Analysts, specialists, engineers, or coordinators working with the relevant processes
- Project, implementation, assurance, or improvement team members
- Professionals preparing for broader responsibilities in this field
Program outline
A clear structure for the learning journey.
Outline points are grouped in one designed block instead of being treated as separate module cards.
01Lakehouse Architecture and Workspace Control3 topics
- Unity Catalog, metastores, catalogs, schemas, and managed storage
- Compute policies, serverless options, and workspace boundaries
- Medallion architecture and data-product ownership
02Ingestion with Auto Loader and Delta Lake3 topics
- CloudFiles discovery, schema inference, and rescued data
- Delta transaction log, schema enforcement, and time travel
- Batch and streaming ingestion design choices
03Transformation Pipelines3 topics
- Spark DataFrame transformations and SQL workflows
- Delta Live Tables expectations and pipeline dependencies
- Incremental processing, change data feed, and merge patterns
04Orchestration and Production Operations3 topics
- Databricks Workflows jobs, tasks, parameters, and notifications
- Cluster sizing, Photon, partitioning, and cost controls
- Monitoring failed runs, data quality, lineage, and recovery
05Lakehouse Pipeline Workshop3 topics
- Design bronze, silver, and gold tables for a business dataset
- Configure quality rules and an incremental load
- Review performance evidence and production-readiness controls
Materials provided
- ○ Course-specific presentation slides
- ○ Guided exercises, scenarios, or configured-environment activities appropriate to the subject
- ○ Course-specific worksheets, checklists, or calculation templates
- ○ Applied workplace case materials
- ○ 4D Certificate of Completion issued by 4D Training & Consultancy
- ○ Post-course support for implementation questions
Training Options
Programs can be delivered in-house, online, or in a blended format depending on your team's schedule, location, and learning objectives. When an external certificate or exam is included, certification rules and fees remain under the relevant awarding body's policies, while 4D provides the training and preparation support.
Why choose 4D
4D Training & Consultancy adapts the program to the client’s operating environment. Delivery combines structured explanation with subject-specific analysis, exercises, and implementation decisions so participants can transfer the learning to real responsibilities without implying vendor authorization.
Related courses
AI Agents and Workflow Automation for Business Operations
This course helps business and operations teams understand how AI agents and workflow automation can reduce repetitive work, improve handoffs, and support faster execution. Participants learn how to map processes, identify automation candidates, design human-in-the-loop controls, and manage risks before scaling AI-enabled workflows.
View courseAI Change Management and Adoption for Managers
This course helps managers lead teams through AI adoption with clarity, confidence, and responsible use. Participants learn how to address resistance, redesign work, set expectations, coach employees, define safe-use rules, and measure adoption without creating fear or unrealistic expectations.
View courseAI for Business Leaders and Department Managers
This course helps business leaders and department managers understand how artificial intelligence can be used responsibly across departments. Participants explore practical AI use cases, productivity opportunities, governance requirements, implementation risks, and decision-making considerations without needing a technical background.
View course