Microsoft Fabric Part 1: Building a Config-Driven Data Ingestion Framework for Dynamics 365

Every data engineering team eventually hits the same wall. You build a pipeline for one entity — opportunities, invoices, time entries. It works well. Then another entity gets added, and another. Before long, you have a collection of pipelines that each do roughly the same thing but are written differently, maintained separately, and break in different ways.

The question is not whether this happens — it always does. The question is whether your framework is designed to prevent it from the start.

This is Part 1 of a two-part series on building an end-to-end data engineering framework on Microsoft Fabric connected to Dynamics 365. Before we get into the detail, here is how the full architecture fits together:

Traditional data engineering approaches treat each entity as a unique problem. A pipeline is built for accounts, another for contacts, another for time entries. Each has its own authentication logic, its own pagination handling, its own incremental-load approach.

This creates a set of problems that compound over time:

The answer is not better pipelines. It is a framework where the pipeline is generic and the entity-specific details live in configuration.

The ingestion layer of this framework runs entirely from a single Fabric workspace containing one Lakehouse, two pipelines, and one master configuration file. Every entity, its key column, its watermark column, and its source details live in a CSV — not in pipeline code.

The Lakehouse is structured using a Medallion architecture — three table layers: Bronze (raw data exactly as it arrives from D365), Silver (cleansed and standardised), and Gold (business-ready, joined, and logic-applied). The ingestion framework is responsible for the first step — getting data from D365 into the Bronze layer reliably, incrementally, and without entity-specific code.

The Fabric workspace — one Lakehouse, two pipelines, and a config file that together handle ingestion for any number of D365 entities

Adding a new entity to the framework means adding one row to a CSV. No new pipeline. No new deployment. No code change.

Inside CRM_Lakehouse, the Tables area is organised by Medallion layer. The Files area holds the configuration CSVs that drive every pipeline and notebook. Three files do all the work:

The Lakehouse holds both the data layers and the config files that drive every pipeline and notebook — everything in one place

Every entity is described in a single row of ingestion.csv. Each row contains:

The pipeline reads this file, builds its incremental filter dynamically, and upserts records using the configured key. No entity-specific code exists anywhere.

ingestion.csv — every entity is a single self-describing row. Adding a new D365 entity is a config change, not a code change

The Data Ingestion Trigger pipeline is the orchestrator. It works in three steps:

One lightweight orchestrator pipeline controls an unlimited number of entities — no changes needed when a new entity is added.

The Trigger pipeline — Lookup config, loop through entities, invoke the Entity pipeline once per row

The Data Ingestion Entity pipeline is the reusable worker. It receives a single entityConfig parameter and executes three steps:

Raw data lands in Bronze exactly as it arrives from D365 — no transformation, no filtering. The same pipeline handles every entity with no hardcoded entity-specific logic.

The Entity pipeline — authenticate, extract, and load. Three activities handle any entity without a line of entity-specific code

The Copy Data activity is configured to page through large D365 result sets automatically:

No manual offset logic is required — pagination works identically for every entity regardless of record count.

Pagination via OData’s native @odata.nextLink — the pipeline pages through any result set automatically, regardless of size

Each run filters the OData query using the entity’s configured IncrementalFilterColumn and last checkpoint value. Results are written to Bronze using an Upsert table action:

Upsert mode with config-driven key columns — Bronze stays accurate across every incremental run without duplicates or missed records

Building a new pipeline for every new data entity is one of the most common and quietly expensive habits in enterprise data engineering. It feels manageable at first — one entity, one pipeline — until the number of entities grows and the cost of maintaining, updating, and troubleshooting a collection of near-identical pipelines starts to outweigh the value they deliver.

The config-driven ingestion framework on Microsoft Fabric eliminates that pattern entirely. Any D365 entity — whether it is time entries today, invoices tomorrow, or a new custom entity next month — is onboarded by adding a single row to a configuration file. The same two pipelines handle authentication, pagination, incremental loading, and upsert logic for every entity, every time.

The result is a data ingestion layer that is faster to extend, easier to maintain, and more reliable to operate — with every entity landing in Bronze consistently, traceably, and without duplicates.