Summary Why traditional entity-by-entity pipelines do not scale for enterprise data engineering How a config-driven ingestion framework eliminates the need to write new code for every new data entity How two generic pipelines handle authentication, pagination, and incremental upsert loading for any D365 entity How this was built as part of a Microsoft Fabric implementation for a D365 Finance & Operations data platform Table of Contents 1. Let’s Start Here 2. The Challenge 3. The Solution — Config-Driven Ingestion 4. Technical Deep-Dive 5. Business Impact 6. Conclusion Let’s Start Here Every data engineering team eventually hits the same wall. You build a pipeline for one entity — opportunities, invoices, time entries. It works well. Then another entity gets added, and another. Before long, you have a collection of pipelines that each do roughly the same thing but are written differently, maintained separately, and break in different ways. The question is not whether this happens — it always does. The question is whether your framework is designed to prevent it from the start. This is Part 1 of a two-part series on building an end-to-end data engineering framework on Microsoft Fabric connected to Dynamics 365. Before we get into the detail, here is how the full architecture fits together: The Full Architecture — D365 to AI D365 / F&O  →  Bronze Layer (raw ingestion)  →  Silver Layer (cleansed)  →  Gold Layer (business-ready)  →  Power BI Reports + Fabric Data Agent Part 1 covers the ingestion step — pulling data from D365 into the Bronze layer of the Fabric Lakehouse using a config-driven pipeline framework. Part 2 covers everything after Bronze — transformation through Silver and Gold, Direct Lake reporting in Power BI, and conversational AI through the Fabric Data Agent. This part focuses on the ingestion layer — how a single config-driven framework pulls data from any D365 entity and lands it in the Bronze layer of a Fabric Lakehouse, without writing new code for each entity. The Challenge Traditional data engineering approaches treat each entity as a unique problem. A pipeline is built for accounts, another for contacts, another for time entries. Each one has its own authentication logic, its own pagination handling, its own incremental-load approach. This creates a set of problems that compound over time: a.   Onboarding a new entity requires a new pipeline build, test cycle, and deployment — work that can take days b.   Incremental load logic is duplicated across pipelines, often inconsistently, leading to missed records or duplicates c.   When upstream systems change — authentication, API structure, column names — the blast radius is wide d.   There is no single place to look to understand what is being ingested and how The answer is not better pipelines. It is a framework where the pipeline is generic and the entity-specific details live in configuration. The Solution — Config-Driven Ingestion on Microsoft Fabric The ingestion layer of this framework runs entirely from a single Fabric workspace containing one Lakehouse, two pipelines, and one master configuration file. Every entity, its key column, its watermark column, and its source details live in a CSV — not in pipeline code. The Lakehouse is structured using a Medallion architecture — three table layers: Bronze (raw data exactly as it arrives from D365), Silver (cleansed and standardised), and Gold (business-ready, joined, and logic-applied). The ingestion framework is responsible for the first step — getting data from D365 into the Bronze layer reliably, incrementally, and without entity-specific code. The Fabric workspace — one Lakehouse, two pipelines, and a config file that together handle ingestion for any number of D365 entities Adding a new entity to the framework means adding one row to a CSV. No new pipeline. No new deployment. No code change. Technical Deep-Dive The Lakehouse and Config Files Inside CRM_Lakehouse, the Tables area is organised by Medallion layer. The Files area holds the configuration CSVs that drive every pipeline and notebook. Three files do all the work: ingestion.csv — controls which entities are ingested, how they are filtered, and where they land b2s_columnconfig.csv — controls Bronze-to-Silver column mapping, aliasing, and type casting gold_fixed.csv — controls Gold-layer business logic and join definitions The Lakehouse holds both the data layers and the config files that drive every pipeline and notebook — everything in one place The Master Config — ingestion.csv Every entity is described in a single row of ingestion.csv. Each row contains: Entity name and OData logical name — what to call and where to find it in D365 Primary key column — used for upsert to prevent duplicates Incremental filter column — the watermark field used to fetch only changed records Partition key — supports multi-source ingestion Checkpoint — stores the last watermark value applied so each run picks up exactly where the last one left off The pipeline reads this file, builds its incremental filter dynamically, and upserts records using the configured key. No entity-specific code exists anywhere. ingestion.csv — every entity is a single self-describing row. Adding a new D365 entity is a config change, not a code change The Trigger Pipeline The Data Ingestion Trigger pipeline is the orchestrator. It works in three steps: Lookup — reads ingestion.csv and returns the full list of configured entities ForEach — loops through every entity row in the config Invoke Pipeline — calls the Entity pipeline once per row, passing the full config object as a parameter One lightweight orchestrator pipeline controls an unlimited number of entities — no changes needed when a new entity is added. The Trigger pipeline — Lookup config, loop through entities, invoke the Entity pipeline once per row The Entity Pipeline The Data Ingestion Entity pipeline is the reusable worker. It receives a single entityConfig parameter and executes three steps: Get Access Token — authenticates against D365/F&O via OAuth using the entity config Set Variable — stores the token for use in the next activity Copy Data — calls the correct OData endpoint, handles pagination, and loads the result into the … Continue reading Microsoft Fabric Part 1 – Building a Config-Driven Data Ingestion Framework for D365 →