Summary Clean ERP data sitting in a data lake doesn’t answer the question every operations leader eventually asks: where exactly is our process breaking down? We built an automated pipeline that connects a client-facing web portal, Azure Table Storage, and Azure Databricks to a leading process mining platform, turning validated ERP data into a living view of how work actually flows. The pipeline is fully status-driven: every record is tracked from submission through processing to completion, with no manual exports or spreadsheet hand-offs. Purchase order data is modeled through a medallion architecture and delivered to the process mining platform, where AI-driven analysis automatically surfaces bottlenecks and deviations from the expected process. Business impact: process owners moved from static, after-the-fact reporting to a near real-time, evidence-based view of process performance. Table of Contents 01  About the Customer 05  The Six-Step Pipeline 02  The Challenge 06  Architecture Overview 03  The Solution 07  Business Impact 04  AI-Driven Process Mining 08  FAQs About the Customer Customer Spotlight A Leading Digital Transformation Partner — Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Standardized, clean data answers “what happened.” It rarely answers “why is this taking so long” or “where exactly is this process breaking down.” The business kept running into the same limitations: 1Why do purchase orders take longer to close in some regions than others? 2Which approval step is quietly adding the most delay to the process? 3How do we get validated ERP data into a process analysis tool without manual exports every time? 4How do we know, at any point in time, what has been processed, what’s pending, and what failed? 5Can this insight be generated automatically, instead of requiring a manual investigation every quarter? The Solution We extended the existing Databricks-based data platform with an automated, status-driven delivery layer connecting a client web portal, Azure Table Storage, Azure Databricks, and a leading process mining platform, orchestrated end-to-end with minimal manual intervention. Status-Driven Orchestration Every record carries a live status, from initial submission through sync completion, tracked in Azure Table Storage. Automated Bulk Processing Azure Logic Apps trigger the pipeline through APIs, so batches of records are processed without manual intervention. Reusable Databricks Framework The same medallion pipeline used for data standardization models Purchase Order data for process mining. AI-Driven Process Analysis The process mining platform’s AI reconstructs the real, as-executed process and highlights bottlenecks automatically. The Six-Step Pipeline Here’s how a single record moves from submission to a fully synced, process-mining-ready state: ⚙ Client Web PortalEnd-to-end data pipeline · Azure + Databricks 6 steps 🌐 1) Website Input The user submits data via the client web portal, a form or API request initiates the pipeline. ↓ 🗃 2) Azure Table Sync Incoming data is written and synced into Azure Table Storage. ↓ 📁 3) Status Filter Records from Azure Table are filtered where status matches: ✓ Perfect🕑 Queue ↓ ⚡ 4) Databricks Pipeline The framework is executed through the Databricks pipeline, processing all filtered records in batch. ↓ 🔄 5) Azure Table Update Once the Databricks sync completes, status is updated in Azure Table: Queue→✓ Synced ↓ 📊 6) UI Reflection Synced data is reflected back to the client web portal UI for the end user. Architecture Overview Once records reach the “Synced” state, the same medallion architecture used for data standardization models Purchase Order Details and Purchase Order Lines and delivers them into the process mining platform: ERPExtracts Row-header files → Bronze Raw landing → Silver Cleansed & standardized → Gold Business-ready models → DeltaLake Parquet delivery → ProcessMining AI-driven analysis Because the framework is configuration-driven, the same architecture can extend to additional ERP data lake sources, SFTP feeds, or other cloud storage without a redesign. AI-Driven Process Mining Analysis With Purchase Order Details and Purchase Order Lines modeled and delivered on a reliable, automated cadence, the process mining platform’s AI reconstructs the real, as-executed purchase order process directly from the underlying event data. Instead of relying on assumptions about how the process should work, process owners see how it actually works: where orders stall, which approval paths deviate from the intended flow, and where cycle time is quietly being lost. “A purchase order may look fine on paper, but the process data tells you exactly where it got stuck, and that gap surfaces automatically.” Business Impact Before After Manual exports required to analyze process performance Fully automated, status-driven pipeline from intake to process mining No visibility into where a record stood in processing Live status tracking from submission through sync completion Process bottlenecks discovered through manual investigation AI-driven analysis surfaces deviations and delays automatically Static, after-the-fact process reporting Near real-time, evidence-based process visibility One-off integration effort per process area Reusable framework, extendable to other business processes Frequently Asked Questions Does this require a specific process mining platform? No. The pipeline delivers modeled, business-ready data through Delta Lake and Parquet, which can be connected to most modern process mining platforms. How often is data refreshed in the process mining platform? The pipeline is designed for batch processing on a defined schedule, and can be tuned toward near real-time delivery depending on business needs and source system constraints. Can this be extended beyond Purchase Order data? Yes. Because the framework is configuration-driven, the same approach can extend to other process areas such as order-to-cash or procure-to-pay. What happens if a record fails validation? Records that don’t meet the status criteria simply remain in a pending state and are not passed downstream, so failures are visible and traceable rather than silently dropped. Conclusion Clean data is the foundation, but process visibility is where the business … Continue reading From ERP Data to Process Mining Insights: Building an Automated Pipeline for Real-Time Process Visibility →

Summary Enterprises running large, multi-country ERP systems often extract data that is technically complete but practically unusable, split across duplicate language columns and encoded with undocumented numeric values. We built a self-service data platform on Azure so that business users, not just data engineers, could define, validate, and process ERP extracts without writing a single line of code. The solution resolves two of the most common ERP data problems: a single field like “Item Description” spread across nine language-specific columns, and reference fields like “Order Status” stored only as numeric codes. A custom web portal puts business users in control of table specifications, validation rules, and processing status, while Azure Databricks and Delta Lake quietly do the heavy lifting behind the scenes. Business impact: dozens of ERP tables moved from raw, multi-language, code-heavy extracts to a single, trusted, human-readable data layer, without adding headcount to the data engineering team. Table of Contents 01  About the Customer 05  Self-Service Data Onboarding 02  The Challenge 06  Medallion Architecture 03  The Solution 07  Business Impact 08  FAQs 09  Conclusion About the Customer Customer Spotlight A Leading Digital Transformation Partner — Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Most organizations extracting data from a large ERP system successfully get the data out. The problem isn’t extraction, it’s making that data mean something the moment it lands. Business and IT teams found themselves asking the same questions on repeat: 1Why does the same field appear nine times, with a different value in each column? 2What does “Order Status= 3” actually mean, and who is the source of truth for that mapping? 3How much manual translation and lookup work happens before a single report can be trusted? 4Can business users resolve these issues themselves, without waiting weeks on an IT backlog? 5How do we scale this across dozens of tables without writing dozens of one-off scripts? Two problems came up again and again, and both are far more common across ERP implementations than most leadership teams realize. Multi-Language Columns Because the ERP system was configured for every Order Status the business operates in, a single logical field such as “Item Description” existed as up to nine separate columns, one per language: English, French, German, Spanish, and more. Reports built directly on top of the raw extract had no reliable way of knowing which column to use for which record. In practice, this meant a plant manager in France could open a report and see item names in German, while a sales report for the Spanish market silently pulled blank fields because the Spanish-language column hadn’t been populated for that record. The data was all there; it just wasn’t usable without someone manually deciding, table by table, which language column to trust. Undocumented Domain Values Reference fields like Country, Currency, and Order Status were stored as raw numeric codes rather than readable labels, for example Order Status: 1 = Completed , 2 = In Progress, 3 = Shipped. These mappings lived inside ERP configuration screens, not in the extracted data itself. That meant every downstream report, dashboard, or spreadsheet needed its own copy of the same lookup table, manually kept in sync. When a code changed or a new Order Status was added in the ERP, there was no guarantee every report using it would be updated at the same time, which meant leadership could be looking at the performance chart that was quietly wrong. The Solution Rather than writing custom transformation logic for every table (a solution that ages badly the moment a new table or region gets added), we designed a configuration-driven pipeline built on Azure Databricks, fronted by a self-service web application that puts control directly in the hands of business and functional users. Self-Service Web Portal Business users upload table specifications, review validation results, and queue tables for processing, entirely through a browser. Medallion Architecture Azure Databricks and Delta Lake refine raw extracts through Bronze, Silver, and Gold layers, without table-specific code. Automated Language Resolution Multi-language columns are detected and normalized automatically based on the specification, not hardcoded per table. Centralized Domain Mapping Numeric and coded reference values are resolved against a single, maintained lookup layer instead of scattered spreadsheets. Self-Service Data Onboarding: No Databricks Knowledge Required The centerpiece of the solution is a custom web application that lets a business or functional analyst, not a Databricks engineer, onboard a new ERP table from start to finish. Here’s what that looks like in practice: A business user uploads an Excel-based table specification defining the expected columns, data types, which fields are multi-language, and which fields are domain-coded and how to decode them. The portal validates the specification instantly, flagging missing mandatory columns or mismatches before any data is processed, so problems are caught at the source rather than three reports downstream. Once validation passes, the same user queues the table for processing with a single click. No notebook to open, no cluster to configure, no code to write or review. Behind the scenes, that specification feeds a generic, reusable Databricks framework that already knows how to apply the correct language resolution and domain-value decoding rules, so engineering effort doesn’t scale linearly with the number of tables. In effect, the portal turns “add a new ERP table to the analytics environment” from a data engineering request into a form a finance or operations analyst can complete in minutes, while still enforcing the same rigor and consistency a hand-built pipeline would require. Medallion Architecture on Databricks Once a table is queued through the portal, Azure Databricks takes over: Bronze: Raw ERP extracts are landed as-is, preserving … Continue reading How a Self-Service Data Portal Solved Multi-Language and Domain Value Chaos in ERP Data →

In production Databricks environments, data issues such as accidental overwrites, faulty MERGE conditions, or incorrect backfills are common. Delta Lake’s Time Travel is not just a feature – it is a critical recovery and governance mechanism. This blog focuses only on practical recovery strategies that are actually used in real-world production systems. Why Time Travel Is Critical in Production Common failure scenarios include: •a. INSERT OVERWRITE wiping historical data • b. Incorrect MERGE conditions deleting valid records • c. Wrong filters during backfill corrupting data Reprocessing data is expensive and risky. Time Travel enables instant rollback with minimal impact. Version vs Timestamp (What You Should Use) Always prefer version-based time travel for recovery operations. Why version-based recovery is preferred: • a. Precise and deterministic • b. No time zone dependency • c. Safest option for production recovery Use timestamp-based queries only for auditing, not recovery. Identify the Last Safe State Before performing any recovery, always inspect the table history. DESCRIBE HISTORY crm_opportunities; Key fields to review: • a. version • b. timestamp • c. operation • d. userName This history acts as the single source of truth during incidents. Recovery Patterns That Actually Work 1. Partial Data Recovery (Recommended) Recover only the affected records instead of rolling back the entire table. Advantages: • a. No downtime • b. Safe for downstream reports • c. Most production-friendly approach 2. Full Table Restore (Use Carefully) Advantages: •a. Fast and atomic Risks: •a. Impacts all downstream consumers Use this approach only when the entire table is corrupted. Safe Validation Using CLONE Before restoring data in production, validate changes using a clone. Typical use cases: • a. Validate recovered data • b. Compare versions •c. Run business checks Retention & VACUUM (Most Common Mistake) The following command causes permanent data loss: Once vacuumed aggressively, time travel breaks and rollback becomes impossible. Production-Safe Retention Recommended retention: • a. Critical tables: 30 days • b. Reporting tables: 7–14 days Auditing & Root Cause Analysis (RCA) Track who changed data and when: Compare changes between versions: Key Best Practices • a. Capture table version before running risky jobs • b. Always use version-based time travel for recovery • c. Prefer partial recovery over full restores • d. Avoid aggressive VACUUM operations • e. Extend retention for critical tables • f. Validate using CLONE before restoring To conclude, Delta Lake Time Travel is not a backup mechanism, but it is the fastest and safest recovery tool in Databricks. When used correctly, it prevents downtime, reduces reprocessing cost, and improves production reliability. For enterprise Databricks pipelines, mastering this capability is mandatory, not optional. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Delta Lake is a storage layer that brings reliability, consistency, and flexibility to big data lakes. It enables advanced features such as Time Travel, Schema Evolution, and ACID Transactions, which are crucial for modern data pipelines. Feature Benefit Time Travel Access historical data for auditing, recovery, or analysis. Schema Evolution Adapt automatically to changes in the data schema. ACID Transactions Guarantee reliable and consistent data with atomic upserts. 1. Time Travel Time Travel allows you to access historical versions of your data, making it possible to “go back in time” and query past snapshots of your dataset. Use Cases:– Recover accidentally deleted or updated data.– Audit and track changes over time.– Compare dataset versions for analytics. How it works:Delta Lake maintains a transaction log that records every change made to the table. You can query a previous version using either a timestamp or a version number. Example: 2. Schema Evolution Schema Evolution allows your Delta table to adapt automatically to changes in the data schema without breaking your pipelines. Use Cases:– Adding new columns to your dataset.– Adjusting to evolving business requirements.– Simplifying ETL pipelines when source data changes. How it works:When enabled, Delta automatically updates the table schema if the incoming data contains new columns. Example: 3. ACID Transactions (with Atomic Upsert) ACID Transactions (Atomicity, Consistency, Isolation, Durability) ensure that all data operations are reliable and consistent, even in the presence of concurrent reads and writes. Atomic Upsert guarantees that an update or insert operation happens fully or not at all. Key Benefits:– No partial updates — either all changes succeed or none.– Safe concurrent updates from multiple users or jobs.– Consistent data for reporting and analytics.– Atomic Upsert ensures data integrity during merges. Atomic Upsert Example (MERGE): Here:– whenMatchedUpdateAll() updates existing rows.– whenNotMatchedInsertAll() inserts new rows.– The operation is atomic — either all updates and inserts succeed together or none. To conclude, Delta Lake makes data pipelines modern, maintainable, and error-proof. By leveraging Time Travel, Schema Evolution, and ACID Transactions, you can build robust analytics and ETL workflows with confidence, ensuring reliability, consistency, and adaptability in your data lake operations. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com