Error Handling in Azure Data Factory (ADF): Part 1

Azure Data Factory (ADF) is a powerful ETL tool, but when it comes to error handling, things can get tricky—especially when you’re dealing with parallel executions or want to notify someone on failure.

In this two-part blog series, we’ll walk through how to build intelligent error handling into your ADF pipelines. This post—Part 1—focuses on the planning phase: understanding ADF’s behavior, the common pitfalls, and how to set your pipelines up for reliable error detection and notification. In Part 2, we’ll implement everything you’ve planned to use ADF control flows.

Part 1: Planning for Failures

Step 1: Understand ADF Dependency Behavior

In ADF, activities can be connected via dependency conditions like:

• On Success
• On Failure
• On Skip

When multiple dependencies are attached to a single activity, ADF uses an OR condition.

However, if you have parallel branches, ADF uses an AND condition for the following activity—meaning the next activity runs only if all parallel branches succeed.

Step 2: Identify the Wrong Approach

Many developers attempt to add a “failure email” activity after each pipeline activity, assuming it will trigger if any activity fails.

This doesn’t work as expected:

• Parallel activities will require all branches to fail before the next activity (e.g., email alert) is triggered.
• This leads to missed alerts when only one activity fails and others succeed.

Step 3: Design with a Centralized Failure Handler in Mind

So, what’s the right approach?

Plan your pipeline in a way that allows you to handle any failure from a centralized point—a dedicated failure handler.

Here’s how:

1. For sequential pipelines: Add a final step (like a Logic App call) that is triggered On Failure or On Skip from the last major activity. This ensures the handler catches any interruption upstream.
   
   
   
   
2. For complex or parallel pipelines: Design an architecture that either:

• Monitors each branch independently
• Or joins all branches with a logic gate (like a control activity) before passing failure signals forward
  
  Think of this handler like an emergency responder—not just patching the issue but informing the right people.

Step 4: Plan Your Notification Strategy

Error detection is one half of the equation. The other half is communication.

Ask yourself:

• How will stakeholders be notified? Email? Teams? Dashboard?
• What info should they receive? Activity name, error message, pipeline name?
• How fast should alerts go out? Real-time? Batched?

To conclude, start thinking about Logic Apps, Webhooks, or Azure Functions that you can plug in later to send customized notifications. We’ll cover the “how” in the next blog, but the “what” needs to be defined now.

Planning for failure isn’t pessimism—it’s smart architecture.
By understanding ADF’s behavior and avoiding common mistakes with parallel executions, you can build pipelines that fail gracefully, alert intelligently, and recover faster.

In Part 2, we’ll take this plan and show you how to implement it step-by-step using ADF’s built-in tools.

Please refer to our case study https://www.cloudfronts.com/case-studies/city-council/ to know more about how we used the Azure Data Factory and other AIS to deliver seamless integration.

We hope you found this blog post helpful! If you have any questions or want to discuss further, please contact us at transform@cloudfronts.com.