Summary Clean ERP data sitting in a data lake doesn’t answer the question every operations leader eventually asks: where exactly is our process breaking down? We built an automated pipeline that connects a client-facing web portal, Azure Table Storage, and Azure Databricks to a leading process mining platform, turning validated ERP data into a living view of how work actually flows. The pipeline is fully status-driven: every record is tracked from submission through processing to completion, with no manual exports or spreadsheet hand-offs. Purchase order data is modeled through a medallion architecture and delivered to the process mining platform, where AI-driven analysis automatically surfaces bottlenecks and deviations from the expected process. Business impact: process owners moved from static, after-the-fact reporting to a near real-time, evidence-based view of process performance. Table of Contents 01  About the Customer 05  The Six-Step Pipeline 02  The Challenge 06  Architecture Overview 03  The Solution 07  Business Impact 04  AI-Driven Process Mining 08  FAQs About the Customer Customer Spotlight A Leading Digital Transformation Partner — Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Standardized, clean data answers “what happened.” It rarely answers “why is this taking so long” or “where exactly is this process breaking down.” The business kept running into the same limitations: 1Why do purchase orders take longer to close in some regions than others? 2Which approval step is quietly adding the most delay to the process? 3How do we get validated ERP data into a process analysis tool without manual exports every time? 4How do we know, at any point in time, what has been processed, what’s pending, and what failed? 5Can this insight be generated automatically, instead of requiring a manual investigation every quarter? The Solution We extended the existing Databricks-based data platform with an automated, status-driven delivery layer connecting a client web portal, Azure Table Storage, Azure Databricks, and a leading process mining platform, orchestrated end-to-end with minimal manual intervention. Status-Driven Orchestration Every record carries a live status, from initial submission through sync completion, tracked in Azure Table Storage. Automated Bulk Processing Azure Logic Apps trigger the pipeline through APIs, so batches of records are processed without manual intervention. Reusable Databricks Framework The same medallion pipeline used for data standardization models Purchase Order data for process mining. AI-Driven Process Analysis The process mining platform’s AI reconstructs the real, as-executed process and highlights bottlenecks automatically. The Six-Step Pipeline Here’s how a single record moves from submission to a fully synced, process-mining-ready state: ⚙ Client Web PortalEnd-to-end data pipeline · Azure + Databricks 6 steps 🌐 1) Website Input The user submits data via the client web portal, a form or API request initiates the pipeline. ↓ 🗃 2) Azure Table Sync Incoming data is written and synced into Azure Table Storage. ↓ 📁 3) Status Filter Records from Azure Table are filtered where status matches: ✓ Perfect🕑 Queue ↓ ⚡ 4) Databricks Pipeline The framework is executed through the Databricks pipeline, processing all filtered records in batch. ↓ 🔄 5) Azure Table Update Once the Databricks sync completes, status is updated in Azure Table: Queue→✓ Synced ↓ 📊 6) UI Reflection Synced data is reflected back to the client web portal UI for the end user. Architecture Overview Once records reach the “Synced” state, the same medallion architecture used for data standardization models Purchase Order Details and Purchase Order Lines and delivers them into the process mining platform: ERPExtracts Row-header files → Bronze Raw landing → Silver Cleansed & standardized → Gold Business-ready models → DeltaLake Parquet delivery → ProcessMining AI-driven analysis Because the framework is configuration-driven, the same architecture can extend to additional ERP data lake sources, SFTP feeds, or other cloud storage without a redesign. AI-Driven Process Mining Analysis With Purchase Order Details and Purchase Order Lines modeled and delivered on a reliable, automated cadence, the process mining platform’s AI reconstructs the real, as-executed purchase order process directly from the underlying event data. Instead of relying on assumptions about how the process should work, process owners see how it actually works: where orders stall, which approval paths deviate from the intended flow, and where cycle time is quietly being lost. “A purchase order may look fine on paper, but the process data tells you exactly where it got stuck, and that gap surfaces automatically.” Business Impact Before After Manual exports required to analyze process performance Fully automated, status-driven pipeline from intake to process mining No visibility into where a record stood in processing Live status tracking from submission through sync completion Process bottlenecks discovered through manual investigation AI-driven analysis surfaces deviations and delays automatically Static, after-the-fact process reporting Near real-time, evidence-based process visibility One-off integration effort per process area Reusable framework, extendable to other business processes Frequently Asked Questions Does this require a specific process mining platform? No. The pipeline delivers modeled, business-ready data through Delta Lake and Parquet, which can be connected to most modern process mining platforms. How often is data refreshed in the process mining platform? The pipeline is designed for batch processing on a defined schedule, and can be tuned toward near real-time delivery depending on business needs and source system constraints. Can this be extended beyond Purchase Order data? Yes. Because the framework is configuration-driven, the same approach can extend to other process areas such as order-to-cash or procure-to-pay. What happens if a record fails validation? Records that don’t meet the status criteria simply remain in a pending state and are not passed downstream, so failures are visible and traceable rather than silently dropped. Conclusion Clean data is the foundation, but process visibility is where the business … Continue reading From ERP Data to Process Mining Insights: Building an Automated Pipeline for Real-Time Process Visibility →

Summary Enterprises running large, multi-country ERP systems often extract data that is technically complete but practically unusable, split across duplicate language columns and encoded with undocumented numeric values. We built a self-service data platform on Azure so that business users, not just data engineers, could define, validate, and process ERP extracts without writing a single line of code. The solution resolves two of the most common ERP data problems: a single field like “Item Description” spread across nine language-specific columns, and reference fields like “Order Status” stored only as numeric codes. A custom web portal puts business users in control of table specifications, validation rules, and processing status, while Azure Databricks and Delta Lake quietly do the heavy lifting behind the scenes. Business impact: dozens of ERP tables moved from raw, multi-language, code-heavy extracts to a single, trusted, human-readable data layer, without adding headcount to the data engineering team. Table of Contents 01  About the Customer 05  Self-Service Data Onboarding 02  The Challenge 06  Medallion Architecture 03  The Solution 07  Business Impact 08  FAQs 09  Conclusion About the Customer Customer Spotlight A Leading Digital Transformation Partner — Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Most organizations extracting data from a large ERP system successfully get the data out. The problem isn’t extraction, it’s making that data mean something the moment it lands. Business and IT teams found themselves asking the same questions on repeat: 1Why does the same field appear nine times, with a different value in each column? 2What does “Order Status= 3” actually mean, and who is the source of truth for that mapping? 3How much manual translation and lookup work happens before a single report can be trusted? 4Can business users resolve these issues themselves, without waiting weeks on an IT backlog? 5How do we scale this across dozens of tables without writing dozens of one-off scripts? Two problems came up again and again, and both are far more common across ERP implementations than most leadership teams realize. Multi-Language Columns Because the ERP system was configured for every Order Status the business operates in, a single logical field such as “Item Description” existed as up to nine separate columns, one per language: English, French, German, Spanish, and more. Reports built directly on top of the raw extract had no reliable way of knowing which column to use for which record. In practice, this meant a plant manager in France could open a report and see item names in German, while a sales report for the Spanish market silently pulled blank fields because the Spanish-language column hadn’t been populated for that record. The data was all there; it just wasn’t usable without someone manually deciding, table by table, which language column to trust. Undocumented Domain Values Reference fields like Country, Currency, and Order Status were stored as raw numeric codes rather than readable labels, for example Order Status: 1 = Completed , 2 = In Progress, 3 = Shipped. These mappings lived inside ERP configuration screens, not in the extracted data itself. That meant every downstream report, dashboard, or spreadsheet needed its own copy of the same lookup table, manually kept in sync. When a code changed or a new Order Status was added in the ERP, there was no guarantee every report using it would be updated at the same time, which meant leadership could be looking at the performance chart that was quietly wrong. The Solution Rather than writing custom transformation logic for every table (a solution that ages badly the moment a new table or region gets added), we designed a configuration-driven pipeline built on Azure Databricks, fronted by a self-service web application that puts control directly in the hands of business and functional users. Self-Service Web Portal Business users upload table specifications, review validation results, and queue tables for processing, entirely through a browser. Medallion Architecture Azure Databricks and Delta Lake refine raw extracts through Bronze, Silver, and Gold layers, without table-specific code. Automated Language Resolution Multi-language columns are detected and normalized automatically based on the specification, not hardcoded per table. Centralized Domain Mapping Numeric and coded reference values are resolved against a single, maintained lookup layer instead of scattered spreadsheets. Self-Service Data Onboarding: No Databricks Knowledge Required The centerpiece of the solution is a custom web application that lets a business or functional analyst, not a Databricks engineer, onboard a new ERP table from start to finish. Here’s what that looks like in practice: A business user uploads an Excel-based table specification defining the expected columns, data types, which fields are multi-language, and which fields are domain-coded and how to decode them. The portal validates the specification instantly, flagging missing mandatory columns or mismatches before any data is processed, so problems are caught at the source rather than three reports downstream. Once validation passes, the same user queues the table for processing with a single click. No notebook to open, no cluster to configure, no code to write or review. Behind the scenes, that specification feeds a generic, reusable Databricks framework that already knows how to apply the correct language resolution and domain-value decoding rules, so engineering effort doesn’t scale linearly with the number of tables. In effect, the portal turns “add a new ERP table to the analytics environment” from a data engineering request into a form a finance or operations analyst can complete in minutes, while still enforcing the same rigor and consistency a hand-built pipeline would require. Medallion Architecture on Databricks Once a table is queued through the portal, Azure Databricks takes over: Bronze: Raw ERP extracts are landed as-is, preserving … Continue reading How a Self-Service Data Portal Solved Multi-Language and Domain Value Chaos in ERP Data →

Organizations today generate massive volumes of raw data from multiple sources such as ERP systems, CRMs, APIs, logs, and IoT devices. However, raw data by itself holds little value unless it is properly processed, transformed, and optimized for analytics. In our data engineering journey, we faced challenges in building scalable and maintainable ETL pipelines that could handle growing data volumes while still delivering reliable insights. Azure Databricks helped us bridge the gap between raw data and business-ready insights. In this blog, we’ll walk through ETL best practices using Azure Databricks and how they helped us build efficient, production-grade data pipelines. Why ETL Best Practices Matter When working with large-scale data pipelines: – Raw data arrives in different formats and structures– Poorly designed ETL jobs lead to performance bottlenecks– Debugging and maintaining pipelines becomes difficult– Data quality issues propagate to downstream reports Key challenges we faced: – Tight coupling between ingestion and transformation– Reprocessing large datasets due to small logic changes– Lack of standardization across pipelines– Slow query performance on analytical layers Solution Architecture Overview Key Components: – Azure Data Lake Storage Gen2– Azure Databricks– Delta Lake– Power BI / Analytics Tools ETL Flow: – Ingest raw data from source systems into the Raw (Bronze) layer– Clean, validate, and standardize data in the Processed (Silver) layer– Apply business logic and aggregations in the Curated (Gold) layer– Expose curated datasets to reporting and analytics tools Step-by-Step ETL Best Practices with Azure Databricks Step 1: Separate Data into Layers (Bronze, Silver, Gold) – Bronze Layer: Store raw data exactly as received– Silver Layer: Apply cleansing, deduplication, and schema enforcement– Gold Layer: Create business-ready datasets and aggregations This separation ensures reusability and prevents unnecessary reprocessing. Step 2: Use Delta Lake for Reliability – Store tables in Delta format– Enable schema enforcement and schema evolution– Leverage time travel for data recovery and debugging Step 3: Build Incremental Pipelines – Process only new or changed data using watermarking– Avoid full reloads unless absolutely required– Design pipelines to safely re-run without duplications Step 4: Parameterize and Modularize Code – Use notebook parameters for environment-specific values– Create reusable functions for common transformations– Avoid hardcoding paths, table names, or business rules Step 5: Optimize Performance Early – Use partitioning based on query patterns– Apply Z-ORDER on frequently filtered columns– Cache datasets selectively for heavy transformations Step 6: Implement Data Quality Checks – Validate nulls, ranges, and duplicate records– Log rejected or invalid records separately– Fail pipelines early when critical checks fail Benefits of Following These ETL Best Practices – Scalability: Easily handle growing data volumes– Reliability: ACID-compliant pipelines with Delta Lake– Maintainability: Modular and reusable code structure– Performance: Faster queries and optimized storage– Cost Efficiency: Reduced compute usage through incremental processing Conclusion Transforming raw data into meaningful insights requires more than just moving data from one place to another. By following ETL best practices with Azure Databricks, we were able to build robust, scalable, and high-performing data pipelines that deliver reliable insights to the business. If your Databricks pipelines are becoming complex, slow, or difficult to maintain, it might be time to revisit your ETL design. Start applying these best practices today and turn your raw data into insights that truly drive decision-making. I hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.

As organizations scale, full data reload pipelines quickly become inefficient and risky. Reporting refresh windows grow longer, source systems experience increased load, and data duplication issues begin to surface. In our recent Unity Catalog-based Lakehouse implementation, we modernized incremental data processing using a structured Change Data Capture (CDC) strategy. Instead of reloading entire datasets daily, we captured only incremental changes across CRM, ERP, HR, and finance systems and governed them through Unity Catalog. This blog explains how we designed and implemented CDC in a production-ready Lakehouse architecture, the decisions behind our approach, and the technical patterns that made it scalable. One of the first challenges in CDC implementations is avoiding hardcoded logic for every entity. Centralized Incremental Control Using Metadata Configuration Instead of embedding incremental rules inside notebooks, we designed a centralized configuration table that drives CDC dynamically. Each record in this control table defines: This allowed us to manage incremental extraction logic centrally without modifying pipeline code for every new table. Fig – Azure Storage Table showing IncrementalField and Timestamp columns Why This Matters This configuration-driven design enabled: Most CDC blogs discuss theory. Few show how incremental control is actually governed in production. Bronze Layer: Append-Only Incremental Capture Once incremental records are identified, they land in the bronze layer in Delta format. Key design decisions: The Bronze layer acts as the immutable change log of the system. This ensures: Bronze is not for reporting. It is for reliability. Structuring CDC Layers with Unity Catalog To ensure proper governance and separation of concerns, we structured our Lakehouse using Unity Catalog with domain-based schemas. Each environment (dev, test, prod) had its own catalog. Within each catalog: (Unity Catalog Bronze schema view) Why Unity Catalog Was Critical Unity Catalog ensured: CDC without governance can become fragile. Unity Catalog added structure and security to the incremental architecture. Silver Layer: Applying CDC with Delta MERGE The Silver layer is where CDC logic is applied. We implemented Type 1 Change Data Capture using Delta Lake MERGE operations. The logic follows: If a job runs twice, the data remains consistent. We intentionally chose Type 1 because reporting required the latest operational state rather than historical tracking. Handling Late-Arriving Data One common CDC failure point is late-arriving records. If extraction logic strictly uses: modified_timestamp > last_run_timeSome records may be missed due to clock drift or processing delays. To mitigate this, we: This ensured no silent data loss. Governance and Power BI Integration A key architectural decision was limiting Power BI access strictly to Gold tables. Through Unity Catalog: This ensured reporting teams could not accidentally query raw incremental data. The result was a clean, governed reporting layer powered by curated Delta tables. Performance Optimization Considerations To maintain optimal performance: Compared to full data reloads, incremental CDC significantly reduced cluster runtime and improved refresh stability. Common CDC Mistakes We Avoided During implementation, we intentionally avoided: These mistakes often appear only after production failures. Designing CDC carefully from the start prevented costly refactoring later. Business Impact By implementing CDC within a Unity Catalog-governed Lakehouse: The architecture is now scalable and future ready. To encapsulates, change data capture is not just an incremental filter. It is a disciplined architectural pattern. When combined with: It becomes a powerful foundation for enterprise analytics. Organizations modernizing their reporting platforms must move beyond full reload pipelines and adopt structured CDC approaches that prioritize scalability, reliability, and governance. If you found this blog useful and would like to discuss, ,Get in touch with CloudFronts at transform@cloudfronts.com.

From a customer perspective, the hardest part of data engineering isn’t building pipelines-it’s ensuring that the data customers rely on is accurate, consistent, and trustworthy. When reports show incorrect revenue or missing customer information, confidence drops quickly. This is where Delta Live Tables in Databricks makes a real difference for customers. Instead of customers dealing with broken dashboards, manual fixes in BI tools, or delayed insights, Delta Live Tables enforces data quality at the pipeline level. Using a Bronze–Silver–Gold approach: Data validation rules are built directly into the pipeline, and customers gain visibility into data quality through built-in monitoring-without extra tools or manual checks. Quick Preview Building data pipelines is not the difficult part. The real challenge is building pipelines that are reliable, monitored, and enforce data quality automatically. That’s where Delta Live Tables in Databricks makes a difference. Instead of stitching together notebooks, writing custom validation scripts, and setting up separate monitoring jobs, Delta Live Tables lets you define your transformations once and handles the rest. Let’s look at a simple example. Imagine an e-commerce company storing raw order data in a Unity Catalog table called: cf.staging.orders_raw The problem? The data isn’t perfect. Some records have negative quantities. Some orders have zero amounts. Customer IDs may be missing. There might even be duplicate order IDs. If this raw data goes straight into reporting dashboards, revenue numbers will be wrong. And once business users lose trust in reports, it’s hard to win it back. Instead of fixing issues later in Power BI or during analysis, we fix them at the pipeline level. In Databricks, we create an ETL pipeline and define a simple three-layer structure: Bronze for raw data, Silver for cleaned data, and Gold for business-ready aggregation. The Bronze layer simply reads from Unity Catalog: Nothing complex here. We’re just loading data from Unity Catalog. No manual dependency setup required. The real value appears in the Silver layer, where we enforce data quality rules directly inside the pipeline: Here’s what’s happening behind the scenes. Invalid rows are automatically removed. Duplicate orders are eliminated. Data quality metrics are tracked and visible in the pipeline UI. There’s no need for separate validation jobs or manual checks. This is what simplifies pipeline development. You define expectations declaratively, and Delta Live Tables enforces them consistently. Finally, in the Gold layer, we create a clean reporting table: At this point, only validated and trusted data reaches reporting systems. Dashboards become reliable. Delta Live Tables doesn’t replace databases, and it doesn’t magically fix bad source systems. What it does is simplify how we build and manage reliable data pipelines. It combines transformation logic, validation rules, orchestration, monitoring, and lineage into one managed framework. Instead of reacting to data issues after reports break, we prevent them from progressing in the first place. For customers, trust in data is everything. Delta Live Tables helps organizations ensure that only validated, reliable data reaches customer-facing dashboards and analytics. Rather than reacting after customers notice incorrect numbers, Delta Live Tables prevents poor-quality data from moving forward. By unifying transformation logic, data quality enforcement, orchestration, monitoring, and lineage in one framework, it enables teams to deliver consistent, dependable insights. The result for customers is simple: accurate reports, faster decisions, and confidence that the data they see reflects reality. I Hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com.

If you’re new to Databricks, chances are someone told you “Everything starts with a Notebook.” They weren’t wrong. In Databricks, a Notebook is where your entire data engineering workflow begins from reading raw data, transforming it, visualizing trends, and even deploying jobs. It’s your coding lab, dashboard, and documentation space all in one. What Is a Databricks Notebook? A Databricks Notebook is an interactive environment that supports multiple programming languages such as Python, SQL, R, and Scala. Each Notebook is divided into cells you can write code, add text (Markdown), and visualize data directly within it. Unlike local scripts, Notebooks in Databricks run on distributed Spark clusters. That means even your 100 GB dataset is processed within seconds using parallel computation. So, Notebooks are more than just code editors they are collaborative data workspaces for building, testing, and documenting pipelines. How Databricks Notebooks Work Under the hood, every Notebook connects to a cluster a group of virtual machines managed by Databricks. When you run code in a cell, it’s sent to Spark running on the cluster, processed there, and results are sent back to your Notebook. This gives you the scalability of big data without worrying about servers or configurations. Setting Up Your First Cluster Before running a Notebook, you must create a cluster it’s like starting the engine of your car. Here’s how: Step-by-Step: Creating a Cluster in a Standard Databricks Workspace Once the cluster is active, you’ll see a green light next to its name now it’s ready to process your code. Creating Your First Notebook Now, let’s build your first Databricks Notebook: Your Notebook is now live ready to connect to data and start executing. Loading and Exploring Data Let’s say you have a sales dataset in Azure Blob Storage or Data Lake. You can easily read it into Databricks using Spark: df = spark.read.csv(“/mnt/data/sales_data.csv”, header=True, inferSchema=True)display(df.limit(5)) Databricks automatically recognizes your file’s schema and displays a tabular preview.Now, you can transform the data: from pyspark.sql.functions import col, sumsummary = df.groupBy(“Region”).agg(sum(“Revenue”).alias(“Total_Revenue”))display(summary) Or, switch to SQL instantly: %sqlSELECT Region, SUM(Revenue) AS Total_RevenueFROM sales_dataGROUP BY RegionORDER BY Total_Revenue DESC Visualizing DataDatabricks Notebooks include built-in charting tools.After running your SQL query:Click + → Visualization → choose Bar Chart.Assign Region to the X-axis and Total_Revenue to the Y-axis.Congratulations — you’ve just built your first mini-dashboard! Real-World Example: ETL Pipeline in a Notebook In many projects, Databricks Notebooks are used to build ETL pipelines: Each stage is often written in a separate cell, making debugging and testing easier.Once tested, you can schedule the Notebook as a Job running daily, weekly, or on demand. Best Practices To conclude, Databricks Notebooks are not just a beginner’s playground they’re the backbone of real data engineering in the cloud.They combine flexibility, scalability, and collaboration into a single workspace where ideas turn into production pipelines. If you’re starting your data journey, learning Notebooks is the best first step.They help you understand data movement, Spark transformations, and the Databricks workflow everything a data engineer need. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com

In production Databricks environments, data issues such as accidental overwrites, faulty MERGE conditions, or incorrect backfills are common. Delta Lake’s Time Travel is not just a feature – it is a critical recovery and governance mechanism. This blog focuses only on practical recovery strategies that are actually used in real-world production systems. Why Time Travel Is Critical in Production Common failure scenarios include: •a. INSERT OVERWRITE wiping historical data • b. Incorrect MERGE conditions deleting valid records • c. Wrong filters during backfill corrupting data Reprocessing data is expensive and risky. Time Travel enables instant rollback with minimal impact. Version vs Timestamp (What You Should Use) Always prefer version-based time travel for recovery operations. Why version-based recovery is preferred: • a. Precise and deterministic • b. No time zone dependency • c. Safest option for production recovery Use timestamp-based queries only for auditing, not recovery. Identify the Last Safe State Before performing any recovery, always inspect the table history. DESCRIBE HISTORY crm_opportunities; Key fields to review: • a. version • b. timestamp • c. operation • d. userName This history acts as the single source of truth during incidents. Recovery Patterns That Actually Work 1. Partial Data Recovery (Recommended) Recover only the affected records instead of rolling back the entire table. Advantages: • a. No downtime • b. Safe for downstream reports • c. Most production-friendly approach 2. Full Table Restore (Use Carefully) Advantages: •a. Fast and atomic Risks: •a. Impacts all downstream consumers Use this approach only when the entire table is corrupted. Safe Validation Using CLONE Before restoring data in production, validate changes using a clone. Typical use cases: • a. Validate recovered data • b. Compare versions •c. Run business checks Retention & VACUUM (Most Common Mistake) The following command causes permanent data loss: Once vacuumed aggressively, time travel breaks and rollback becomes impossible. Production-Safe Retention Recommended retention: • a. Critical tables: 30 days • b. Reporting tables: 7–14 days Auditing & Root Cause Analysis (RCA) Track who changed data and when: Compare changes between versions: Key Best Practices • a. Capture table version before running risky jobs • b. Always use version-based time travel for recovery • c. Prefer partial recovery over full restores • d. Avoid aggressive VACUUM operations • e. Extend retention for critical tables • f. Validate using CLONE before restoring To conclude, Delta Lake Time Travel is not a backup mechanism, but it is the fastest and safest recovery tool in Databricks. When used correctly, it prevents downtime, reduces reprocessing cost, and improves production reliability. For enterprise Databricks pipelines, mastering this capability is mandatory, not optional. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

A Databricks Cluster is a group of virtual machines (VMs) in the cloud that work together to process data using Apache Spark.It provides the memory, CPU, and compute power required to run your code efficiently. Clusters are used for: Each cluster has two main parts: Types of Clusters Databricks supports multiple cluster types, depending on how you want to work. Cluster Type Use Case Interactive (All-Purpose) Clusters Used for notebooks, ad-hoc queries, and development. Multiple users can attach their notebooks. Job Clusters Created automatically for scheduled jobs or production pipelines. Deleted after job completion. Single Node Clusters Used for small data exploration or lightweight development. No executors, only one driver node. How Databricks Clusters WorkWhen you execute a notebook cell, Databricks sends your code to the cluster.The cluster’s driver node divides your task into smaller jobs and distributes them to the executors.The executors process the data in parallel and send the results back to the driver.This distributed processing is what makes Databricks fast and scalable for handling massive datasets. Step-by-Step: Creating Your First Cluster Let’s create a cluster in your Databricks workspace. Step 1: Navigate to Compute In the Databricks sidebar, click Compute. You’ll see a list of existing clusters or an option to create a new one. Step 2: Create a New Cluster Click Create Compute in the top-right corner. Step 3: Configure Basic Settings Step 4: Select Node Type Choose the VM type based on your workload. For development, Standard_DS3_v2 or Standard_D4ds_v5 are cost-effective. Step 5: Auto-Termination Set the cluster to terminate after 10 or 20 minutes of inactivity. This prevents unnecessary cost when the cluster is idle. Step 6: Review and Create Click Create Compute. After a few minutes, your cluster will turn green, indicating it is ready to run code. Clusters in Unity Catalog-Enabled Workspaces If Unity Catalog is enabled in your workspace, there are a few additional configurations to note. Feature Standard Workspace Unity Catalog Workspace Access Mode Default is Single User. Must choose Shared, Single User, or No Isolation Shared. Data Access Managed by workspace permissions. Controlled through Catalog, Schema, and Table permissions. Data Hierarchy Database → Table Catalog → Schema → Table Example Query SELECT * FROM sales.customers; SELECT * FROM main.sales.customers; When you create a cluster with Unity Catalog, you will see a new Access Mode field in the configuration page. Choose “Shared” if multiple users need to access governed data under Unity Catalog. Managing Cluster Performance and CostClusters can become expensive if not managed properly. Follow these tips to optimize performance and cost: a. Use Auto-Termination to shut down idle clusters automatically.b. Choose the right VM size for your workload. Avoid oversizing.c. Use Job Clusters for production pipelines since they start and stop automatically.d. Leverage Autoscaling so Databricks can adjust the number of workers dynamically.e. Monitor with Ganglia metrics to identify performance bottlenecks. Common Cluster Issues and Fixes Issue Cause Fix Cluster stuck starting VM quota exceeded or region issue Change VM size or region. Slow performance Too few workers or data skew Increase worker count or repartition data. Access denied to data Missing storage credentials Use Databricks Secrets or Unity Catalog permissions. High cost Idle clusters running Enable auto-termination. Best Practices for Using Databricks Clusters1. Always attach your notebook to the correct cluster before running it.2. Use development, staging, and production clusters separately.3. Keep the cluster runtime version consistent across environments.4. Terminate unused clusters to reduce cost.5. If you use Unity Catalog, prefer Shared clusters for collaboration. To conclude, clusters are the heart of Databricks.They provide the compute power needed to process large-scale data efficiently. Without them, Databricks Notebooks and Jobs cannot run. Once you understand how clusters work, you will find it easier to manage costs, optimize performance, and build reliable data pipelines. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Databricks Time Travel is a powerful feature of Delta Lake that allows you to access older versions of your data. Whether you want to debug issues, recover deleted records, compare historical performance, or audit how data changed over time—Time Travel makes it effortless. It’s like having a complete rewind button for your tables, eliminating the fear of accidental updates or deletes. What is Time Travel? Time Travel enables you to query previous snapshots of a Delta table using either VERSION AS OF or TIMESTAMP AS OF. Delta automatically versions every transaction-UPDATE, MERGE, DELETE, INSERT. So, you can always go back to an earlier state without restoring backups manually. This versioning is stored in the Delta Log, making rewind operations efficient and reliable. Why Time Travel Matters (Use Cases) Debugging Pipelines: Quickly check what the data looked like before a bad job ran. Accidental Deletes: Recover records or entire tables. Audit & Compliance: Easily demonstrate how data has evolved. Root Cause Analysis: Compare two versions side by side. Model Re-training: Use historical datasets to retrain ML models. Data Quality Tracking: Validate when incorrect data first appeared. How Delta Stores Versions (Architecture Overview) Delta Lake stores metadata and version history inside the _delta_log folder. Each commit creates a new JSON or checkpoint Parquet file representing table state. When you run a query using Time Travel, Databricks does not rebuild the entire table. Instead, it directly reads the snapshot based on the transaction log. This architecture makes Time Travel extremely fast and scalable—even on very large datasets. Time Travel Commands Query older data: SELECT * FROM table VERSION AS OF 5; SELECT * FROM table TIMESTAMP AS OF ‘2024-11-20T10:00:00’; A. Example: DESCRIBE HISTORY Below is an example of using DESCRIBE HISTORY on a Delta table. B. Querying a Specific Version Here is how you can fetch an older snapshot using VERSION AS OF. C. Restoring a Table You can restore a Delta table to any older version using RESTORE TABLE. Retention Rules Delta keeps older versions based on two configs: `delta.logRetentionDuration` → How long commit logs are stored. `delta.deletedFileRetentionDuration`→ How long old data files are retained. By default, Databricks keeps 30 days of history. You can increase this if your compliance policy requires longer retention. Best Practices – Use Time Travel for debugging pipeline issues. – Increase retention for sensitive or audited datasets. – Use `DESCRIBE HISTORY` frequently during development. – Avoid unnecessarily large retention windows—they increase storage costs. – Use `RESTORE` carefully in production environments. To conclude, time Travel in Databricks brings reliability, auditability, and simplicity to modern data engineering. It protects teams from accidental data loss and gives full visibility into how datasets evolve. With just a few commands, you can analyze, compare, or restore historical data instantly making it one of the most useful features of Delta Lake. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

As data platforms mature, teams often face a familiar question:Should we continue with classic ETL pipelines, or move to Delta Live Tables (DLT)? Both approaches work. Both are widely used. The real challenge is knowing which one fits your use case, not which one is newer or more popular. In this blog, I’ll break down Delta Live Tables vs classic ETL from a practical, project-driven perspective, focusing on how decisions are actually made in real data engineering work. Classic ETL in Databricks Classic ETL in Databricks refers to pipelines where engineers explicitly control each stage of data movement and transformation. The pipeline logic is written imperatively, meaning the engineer decides how data is read, processed, validated, and written. Architecturally, classic ETL pipelines usually follow the Medallion pattern: Each step is executed explicitly, often as independent jobs or notebooks. Dependency management, error handling, retries, and data quality checks are all implemented manually or through external orchestration tools. This approach gives teams maximum freedom. Complex ingestion logic, conditional transformations, API integrations, and custom performance tuning are easier to implement because nothing is abstracted away. However, this flexibility also means consistency and governance depend heavily on engineering discipline. We implemented a Classic ETL pipeline in our internal Unity Catalog project, migrating 30+ Power BI reports from Dataverse into Unity Catalog to enable AI/BI capabilities. This architecture allows data to be consumed in two ways – through an agentic AI interface for ad-hoc querying and through Power BI for governed, enterprise-grade visualizations. We chose the ETL approach because it provides strong data quality control, schema stability, and predictable performance at scale. It also allows us to apply centralized transformations, enforce governance standards, optimize storage formats, and ensure consistent semantic models across reporting and AI workloads -making it ideal for production-grade analytics and enterprise adoption. Delta Live Tables Delta Live Tables is a managed, declarative pipeline framework provided by Databricks. Instead of focusing on execution steps, DLT encourages engineers to define what tables should exist and what rules the data must satisfy. From an architectural perspective, DLT formalizes the Medallion pattern. Pipelines are defined as a graph of dependent tables rather than a sequence of jobs. Databricks automatically understands lineage, manages execution order, applies data quality rules, and provides built-in monitoring. DLT pipelines are particularly well-suited for transformation and curation layers, where data is shared across teams and downstream consumers expect consistent, validated datasets. The platform takes responsibility for orchestration, observability, and failure handling, reducing operational overhead. In my next blog, I will demonstrate how to implement Delta Live Tables (DLT) in a hands-on, technical way to help you clearly understand how it works in real-world scenarios. We will walk through the creation of pipelines, data ingestion, transformation logic, data quality expectations, and automated orchestration. The Core Architectural Difference The fundamental difference between classic ETL and Delta Live Tables is how responsibility is divided between the engineer and the platform. In classic ETL, the engineer owns the full lifecycle of the pipeline. This provides flexibility but increases maintenance cost and risk. In Delta Live Tables, responsibility is shared: the engineer defines structure and intent, while Databricks enforces execution, dependencies, and quality. This shift changes how pipelines are designed. Classic ETL is optimized for control and customization. Delta Live Tables is optimized for consistency, governance, and scalability. When Classic ETL Makes More Sense Classic ETL is a strong choice when pipelines require complex logic, conditional execution, or tight control over performance. It is well suited for ingestion layers, API-based data sources, and scenarios where transformations are highly customized or experimental. Teams with strong engineering maturity may also prefer classic ETL for its transparency and flexibility, especially when governance requirements are lighter. When Delta Live Tables Is the Better Fit Delta Live Tables excels when pipelines are repeatable, standardized, and shared across multiple consumers. It is particularly effective for silver and gold layers where data quality, lineage, and operational simplicity matter more than low-level control. DLT is a good architectural choice for enterprise analytics platforms, certified datasets, and environments where multiple teams rely on consistent data definitions. A Practical Architectural Pattern In real-world platforms, the most effective design is often hybrid. Classic ETL is used for ingestion and complex preprocessing, while Delta Live Tables is applied to transformation and curation layers. This approach preserves flexibility where it is needed and enforces governance where it adds the most value. To conclude, Delta Live Tables is not a replacement for classic ETL. It is an architectural evolution that addresses governance, data quality, and operational complexity. The right question is not which tool to use, but where to use each. Mature Databricks platforms succeed by combining both approaches thoughtfully, rather than forcing a single pattern everywhere. Choosing wisely here will save significant rework as your data platform grows. Need help deciding which approach fits your use case? Reach out to us at transform@cloudfronts.com