How to Build an Incremental Data Pipeline with Azure Logic Apps

Why Incremental Loads Matter

When integrating data from external systems, whether it’s a CRM, an ERP like Business Central, or an HR platform like Zoho People, pulling all data every time is expensive, slow, and unnecessary. The smarter approach is to track what has changed since the last successful run and fetch only that delta.

This is the core idea behind an incremental data pipeline: identify a timestamp or sequence field in your source system, persist the last-known watermark, and use it as a filter on your next API call. Azure Logic Apps, paired with Azure Table Storage as a lightweight checkpoint store, gives you everything you need to implement this pattern without managing any infrastructure.

Architecture Overview

Instead of one large workflow doing everything, we separate responsibilities. One Logic App handles scheduling and orchestration. Another handles actual data extraction.

Core components:

• a. Azure Table Storage – stores entity configuration and checkpoint
• b. Logic App 1 – scheduler and orchestrator
• c. Logic App 2 – incremental processor
• d. Source API – Business Central, D365, or any REST API
• e. Target – Data Lake, Blob, or Database
  

3. Metadata Design (Azure Table)

Instead of hardcoding entity names and fields inside Logic Apps, we define them in Azure Table Storage.

Example structure:

Briefly, this table answers three questions:

– What entity should be extracted?
– Which column defines incremental logic?
– What was the last successful checkpoint?

When you want to onboard a new entity, you add a row. No redesign needed.

4. Logic App 1 – Scheduler

Trigger: Recurrence (for example, every 15 minutes)

Steps:

1. Read entities from Azure Table
2. Filter by source if required
3. For each entity, call Logic App 2 via HTTP

This Logic App should not call APIs directly. Its only job is orchestration.

Keep it light.

5. Logic App 2 – Incremental Processor

Trigger: HTTP (called from Logic App 1)

Functional steps:

1. Receive entity details and incremental field
2. Get access token (if required)
3. Read last checkpoint from Azure Table
4. Call API using filter

Example:

?$filter=modifiedOn gt 'LastCheckpoint'

5. Handle pagination
6. Store data in staging
7. Capture max(modifiedOn)
8. Update checkpoint

This is where the real work happens.

6. Checkpoint Strategy

Each entity must maintain:

– LastSuccessfulRunTime
– Status
– LastRecordTimestamp

After successful extraction:

Checkpoint = max(modifiedOn) from extracted data.

This ensures:

• a. No duplicates
• b. No missed records
• c. Predictable behavior

Checkpoint management is the backbone of incremental loading. If this fails, everything fails.

This pattern gives you a production-grade incremental data pipeline entirely within Azure’s managed services. By centralizing entity configuration and watermarks in Azure Table Storage, you create a data-driven pipeline where adding a new integration is as simple as inserting a row — no code deployment required.

The two-Logic-App architecture cleanly separates orchestration from execution, enables parallel processing, and ensures your pipeline is resilient to failures through checkpoint-based watermark management. Whether you’re pulling from Business Central, Zoho People, or any REST API that exposes a timestamp field, this architecture scales gracefully with your data needs.

Explore the case study below to learn how Logic Apps were implemented to solve key business challenges:

1. https://www.cloudfronts.com/case-studies/tinius-olsen-migrating-from-tibco-to-azure-logic-apps-for-seamless-integration-between-d365-field-service-and-finance-operations/
2. https://www.cloudfronts.com/case-studies/buchi-customer-success-story/

Ready to deploy AIS to seamlessly connect systems and improve operational cost and efficiency? Get in touch with CloudFronts at transform@cloudfronts.com.