Summary We built a Databricks Genie agent for our own PMO at CloudFronts, running on Dynamics 365 data held in a Databricks lakehouse. Project managers ask a question in plain English and get an answer back across resource utilization, time tracking, billing and milestones, tickets and cases, and project status. This blog covers what the agent does, what email sentiment analysis shows that the numbers do not, and how it works inside Microsoft Teams. Table of Contents Introduction The Challenge The Solution See It in Action Business Impact Frequently Asked Questions Conclusion Introduction This started inside our own PMO — the Billing and Delivery Excellence function at CloudFronts. We learned about a project risk when someone escalated it. The warning signs came earlier than that, in email threads and internal notes, but reading every thread across every project each week was not work anyone could take on. The rest of the picture was split across systems. Billing held the invoice that had passed its due date, delivery held the milestone that had moved, support held the ticket that had been open for weeks. No one screen put those next to each other, so the PMO opened each project every week and compiled the status by hand. So we built the agent for ourselves first: a Databricks Genie agent running on Dynamics 365 data held in a Databricks lakehouse, which project managers query in plain English. The Challenge Dynamics 365 Project Operations holds everything a project manager needs — resource assignments, logged hours, billing milestones, project budgets, and delivery timelines. The data is there. The challenge is that getting specific answers from it still requires navigating multiple modules, running reports manually, and in many cases, exporting to spreadsheets to piece things together. This created a set of questions that were surprisingly hard to answer: Identifying which resources are overutilized or sitting idle requires pulling allocation data and comparing it manually against actual hours logged Understanding whether a project is at risk means cross-referencing milestone progress, budget consumption, and team capacity — a process that can take hours Billing questions — what has been invoiced, what is pending, what is approaching a milestone — require moving between finance and project views that are not always aligned Status updates for leadership need to be manually compiled, often pulling from data that was accurate yesterday but may have shifted today The result is that project managers operate on a lag — making decisions based on reports that reflect the past, not the present, and spending time producing those reports instead of acting on them. The Solution — A Genie Agent Built on Databricks and D365 Project Operations We built a Genie agent on Azure Databricks, connected to Dynamics 365 Project Operations. Project managers can now ask questions in plain English and get answers drawn directly from their project data — without building a single report. The agent is designed around the areas that matter most to project managers on a daily basis: a. Resource UtilizationThe agent can answer questions about who is overallocated, which resources have capacity available, and how utilization is trending across the team or a specific project. What previously required pulling allocation reports and comparing them against timesheets can now be answered in a single question. b. Time TrackingProject managers can ask which team members have not logged hours for the week, where hours are being spent versus what was planned, and whether a specific project is tracking within its estimated effort. The agent surfaces this from logged timesheet data in D365. c. Billing and MilestonesThe agent connects billing milestone data with project progress, allowing project managers to ask what is due for invoicing, which milestones are approaching, and whether any billing triggers are at risk of being delayed. This brings finance and delivery into the same conversation. d. Tickets and CasesThe agent surfaces open tickets and cases linked to a project — how many are open, which are overdue, how they are distributed across team members, and whether any are blocking delivery. Project managers can ask for a snapshot of issue health across one or multiple projects without navigating case queues manually. e. Email Sentiment AnalysisOne of the more telling signals of how a project is going is often hiding in the inbox. The agent analyses email communication patterns and sentiment across project stakeholders — flagging when tone is shifting, when a client’s responses are becoming shorter or more urgent, or when concerns are being raised repeatedly. This gives project managers an early, qualitative read on relationship health before it shows up in a formal escalation. f. Project StatusInstead of assembling a status report, a project manager can ask for a summary of where a project stands — budget consumed, milestones completed, risks flagged, and remaining timeline. The agent compiles this from D365 data and presents it in plain language, ready to share or act on. The conversation does not stop at one question. A project manager can ask a follow-up — drill into a specific resource, filter by project phase, or compare two projects side by side — and the agent follows the thread, refining its response at each step. Available directly in Microsoft TeamsThe Genie agent is also available as a Databricks App inside Microsoft Teams — meaning project managers do not need to switch tools to get answers. They can ask questions about their projects, resources, and billing directly from the Teams interface they already work in every day. See It in Action Weekly Work Summary — Time Tracking in ActionA project manager asks Genie for a summary of work completed last week. The agent returns a full breakdown — total hours logged, billable vs non-billable split, project-wise distribution, and key observations — in seconds. Case Detail View — Tickets and Cases in ActionA project manager asks for details on a specific case. The agent surfaces the full case record — status, owner, priority, activity timeline, and a follow-up alert — without the manager needing to … Continue reading From Project Reporting to Project Intelligence: How AI is Transforming Project Management →

Microsoft Fabric Part 2 — Data Transformation, Real-Time Reporting and AI for D365 Summary With D365 data landed in Bronze (covered in Part 1), this blog covers the next three stages — transformation through a Medallion architecture, Direct Lake reporting in Power BI, and conversational AI via the Fabric Data Agent. Two generic PySpark notebooks — bronze_to_silver and silver_to_gold — transform raw Bronze data into cleansed Silver and business-ready Gold tables without entity-specific code. A Direct Lake semantic model built on the Gold layer gives Power BI real-time reporting without an import step or scheduled refresh cycle. The built-in Fabric Data Agent, grounded on the Gold layer, gives business users natural-language access to governed data — no separate AI platform, no additional licensing. The result is a complete data and AI layer — ingestion, transformation, reporting, and conversational AI — built on a single platform, driven by configuration, and designed to grow. Table of Contents 01 Quick Recap — Part 1 02 Technical Deep-Dive 03 Business Impact 04 Frequently Asked Questions 05 Conclusion Quick Recap — Part 1 In Part 1, we covered how a config-driven ingestion framework on Microsoft Fabric pulls data from any D365 Finance & Operations entity and lands it into the Bronze layer of a Fabric Lakehouse — using just two generic pipelines and a master CSV config file. No entity-specific code, no new pipeline per entity, and reliable incremental upsert loading driven entirely by configuration. Bronze is the raw layer — data arrives exactly as it comes from D365, unmodified. That is intentional. The Bronze layer is not for reporting. It is the foundation — a reliable, auditable record of everything that came in. What happens next is where the data becomes useful. Where Part 2 picks up Bronze Layer (raw — covered in Part 1)  →  Silver Layer (cleansed)  →  Gold Layer (business-ready)  →  Power BI (Direct Lake reporting)  +  Fabric Data Agent (conversational AI) This part walks through how two generic PySpark notebooks transform Bronze data into clean, business-ready Gold tables, how a Direct Lake semantic model exposes that Gold layer to Power BI without a refresh cycle, and how the built-in Fabric Data Agent gives business users conversational access to the same governed data. Technical Deep-Dive Bronze → Silver — Cleansing and Column Mapping The bronze_to_silver notebook is a parameterised PySpark notebook that promotes raw Bronze data into cleansed Silver tables. For each entity it: 1Reads the Bronze table for the configured entity 2Strips technical column prefixes added by D365 3Consults the b2s_columnconfig table to determine which columns are active and how they should be aliased 4Applies type casting and standardisation 5Writes the result to the Silver schema The same notebook promotes any Bronze table to Silver — changing the entity parameter is all it takes to onboard a new one. bronze_to_silver — strips prefixes, reads column config, applies aliasing. One notebook handles every Bronze-to-Silver promotion Silver → Gold — Business Logic and Joins The silver_to_gold notebook builds business-ready Gold tables by joining multiple Silver entities and applying business logic. For the Resource Time Tracking Gold table it: 1Joins six Silver entities — bookableresources, msdyn_timeentries, msdyn_projecttasks, msdyn_projects, accounts, and msdyn_transactioncategories 2Derives fields including Category, EntryDate, Weekday, TimeSheetStatus, and TimeSpent 3Translates D365 status codes into readable values 4Writes the result as a Delta table to the Gold schema The Gold table is the single, trusted, business-ready version of the data — the only layer exposed to reporting and AI consumers. silver_to_gold — joins Silver entities, applies business logic, and writes a clean Gold table ready for reporting and AI querying Direct Lake Semantic Model — Creation Once the Gold layer is ready, a Direct Lake semantic model is created directly from the Lakehouse home screen. The setup follows three simple steps: 1Click New semantic model from the Lakehouse toolbar 2Select only the Gold table — Bronze and Silver remain hidden from report authors 3Confirm — the model is created in Direct Lake mode, reading Delta Parquet files directly from OneLake No import step. No scheduled refresh. No data duplication. Only Gold is selected for the semantic model — reporting consumers never see raw or intermediate data Direct Lake Semantic Model — Power BI The finished Resource Time Tracking semantic model exposes Gold-layer fields ready for report authoring in Power BI: Category, Customer, EntryDate, ProjectName, ProjectTask ProjectType, ResourceName, TimeSheetStatus, TimeSpent Weekday, Week Number, Year Because it runs in Direct Lake mode, reports always reflect the latest Gold data without anyone needing to trigger a refresh. The finished semantic model in Power BI — Gold fields available for report authoring immediately, with no refresh cycle required Fabric Data Agent Beyond traditional BI, the built-in Fabric Data Agent extends the framework into conversational AI. Key characteristics of the agent: Grounded exclusively on the Gold layer — only curated, business-ready data is exposed Supports natural-language questions — trend analysis, outlier detection, resource summaries, and more Runs within the same Fabric workspace — no separate AI platform, no additional licensing, no integration work Business users and data teams access the same governed data whether they use Power BI or the agent The Fabric Data Agent grounded on the Gold layer — business users ask questions in plain English on the same data that powers Power BI Business Impact 1Consistent transformation governance — two notebooks enforce the same cleansing and business logic across every entity, from Bronze through Gold, with no per-entity exceptions or inconsistencies 2Near real-time reporting — Direct Lake semantic models read Gold Delta tables directly from OneLake, eliminating the import and refresh cycle that traditional Power BI datasets require 3Single trusted layer for all consumers — both Power BI reports and the Data Agent draw from the same governed Gold layer, ensuring consistent numbers across structured reporting and conversational queries 4Built-in AI access at no extra infrastructure cost — the Fabric Data Agent gives business users conversational access to governed data without a separate AI platform, additional licensing, or integration work 5End-to-end traceability — every … Continue reading Microsoft Fabric Part 2: Where Raw Data Becomes Business Intelligence and Conversational AI →

Microsoft Fabric Part 1 — Config-Driven D365 Ingestion into the Lakehouse Summary Building a new pipeline for every new data entity is one of the most common and quietly expensive habits in enterprise data engineering — this blog shows how to eliminate it entirely. A config-driven ingestion framework on Microsoft Fabric pulls data from any D365 Finance & Operations entity into the Bronze layer of a Fabric Lakehouse using just two generic pipelines and one master CSV config file. Adding a new entity requires no new pipeline, no new code, and no deployment — just a single row added to a configuration file. The framework handles OAuth authentication, OData pagination at 5,000 records per page, incremental watermark filtering, and key-based upsert loading — all driven by configuration. This is Part 1 of a two-part series. Part 2 covers transformation through Bronze → Silver → Gold, Direct Lake reporting in Power BI, and conversational AI via the Fabric Data Agent. Table of Contents 01 Let’s Start Here 02 The Challenge 03 The Solution 04 Technical Deep-Dive 05 Business Impact 06 Conclusion Let’s Start Here Every data engineering team eventually hits the same wall. You build a pipeline for one entity — opportunities, invoices, time entries. It works well. Then another entity gets added, and another. Before long, you have a collection of pipelines that each do roughly the same thing but are written differently, maintained separately, and break in different ways. The question is not whether this happens — it always does. The question is whether your framework is designed to prevent it from the start. This is Part 1 of a two-part series on building an end-to-end data engineering framework on Microsoft Fabric connected to Dynamics 365. Before we get into the detail, here is how the full architecture fits together: The Full Architecture — D365 to AI D365 / F&O  →  Bronze Layer (raw ingestion — this blog)  →  Silver Layer (cleansed)  →  Gold Layer (business-ready)  →  Power BI Reports + Fabric Data Agent Part 1 covers the ingestion step — pulling data from D365 into the Bronze layer of the Fabric Lakehouse using a config-driven pipeline framework. Part 2 covers everything after Bronze — transformation through Silver and Gold, Direct Lake reporting in Power BI, and conversational AI through the Fabric Data Agent. The Challenge Traditional data engineering approaches treat each entity as a unique problem. A pipeline is built for accounts, another for contacts, another for time entries. Each has its own authentication logic, its own pagination handling, its own incremental-load approach. This creates a set of problems that compound over time: 1Onboarding a new entity requires a new pipeline build, test cycle, and deployment — work that can take days 2Incremental load logic is duplicated across pipelines, often inconsistently, leading to missed records or duplicates 3When upstream systems change — authentication, API structure, column names — the blast radius is wide 4There is no single place to look to understand what is being ingested and how The answer is not better pipelines. It is a framework where the pipeline is generic and the entity-specific details live in configuration. The Solution — Config-Driven Ingestion on Microsoft Fabric The ingestion layer of this framework runs entirely from a single Fabric workspace containing one Lakehouse, two pipelines, and one master configuration file. Every entity, its key column, its watermark column, and its source details live in a CSV — not in pipeline code. The Lakehouse is structured using a Medallion architecture — three table layers: Bronze (raw data exactly as it arrives from D365), Silver (cleansed and standardised), and Gold (business-ready, joined, and logic-applied). The ingestion framework is responsible for the first step — getting data from D365 into the Bronze layer reliably, incrementally, and without entity-specific code. The Fabric workspace — one Lakehouse, two pipelines, and a config file that together handle ingestion for any number of D365 entities Adding a new entity to the framework means adding one row to a CSV. No new pipeline. No new deployment. No code change. Technical Deep-Dive The Lakehouse and Config Files Inside CRM_Lakehouse, the Tables area is organised by Medallion layer. The Files area holds the configuration CSVs that drive every pipeline and notebook. Three files do all the work: ingestion.csv — controls which entities are ingested, how they are filtered, and where they land b2s_columnconfig.csv — controls Bronze-to-Silver column mapping, aliasing, and type casting gold_fixed.csv — controls Gold-layer business logic and join definitions The Lakehouse holds both the data layers and the config files that drive every pipeline and notebook — everything in one place The Master Config — ingestion.csv Every entity is described in a single row of ingestion.csv. Each row contains: Entity name and OData logical name — what to call and where to find it in D365 Primary key column — used for upsert to prevent duplicates Incremental filter column — the watermark field used to fetch only changed records Partition key — supports multi-source ingestion Checkpoint — stores the last watermark value applied so each run picks up exactly where the last one left off The pipeline reads this file, builds its incremental filter dynamically, and upserts records using the configured key. No entity-specific code exists anywhere. ingestion.csv — every entity is a single self-describing row. Adding a new D365 entity is a config change, not a code change The Trigger Pipeline The Data Ingestion Trigger pipeline is the orchestrator. It works in three steps: 1Lookup — reads ingestion.csv and returns the full list of configured entities 2ForEach — loops through every entity row in the config 3Invoke Pipeline — calls the Entity pipeline once per row, passing the full config object as a parameter One lightweight orchestrator pipeline controls an unlimited number of entities — no changes needed when a new entity is added. The Trigger pipeline — Lookup config, loop through entities, invoke the Entity pipeline once per row The Entity Pipeline The Data Ingestion Entity pipeline is the reusable worker. It … Continue reading Microsoft Fabric Part 1: Building a Config-Driven Data Ingestion Framework for Dynamics 365 →

Stop Waiting on Reports — Databricks Genie | CloudFronts What You Will Learn Why dashboards alone are no longer enough for fast business decisions What Databricks Genie is and how it enables conversational access to your data How this changes the way finance, sales, and operations teams work What it means for your organization’s AI readiness and long-term decision-making Table of Contents 1. Let’s Start Here 2. The Challenge 3. The Solution — Databricks Genie 4. Business Impact 5. Frequently Asked Questions 6. Conclusion Let’s Start Here Organizations today are not short on data. They have dashboards, reports, and analytics tools in place. But when a business leader needs an answer to a specific question — one that no existing report covers — the usual path is to raise a request, wait for an analyst, and revisit it days later. That delay, small as it seems, adds up. Decisions get deferred. Opportunities get missed. And the data that was meant to drive the business ends up sitting behind a queue. Databricks Genie changes how organizations access their data — by making it conversational. The Challenge Dashboards were built to answer the questions someone thought of in the past. They are excellent for monitoring what is already defined — revenue trends, pipeline stages, operational metrics. But business does not move in straight lines. The moment a leader needs to investigate something outside of what was pre-built, the process breaks down: The question gets raised in a meeting — but no dashboard covers it It gets passed to a data analyst, who adds it to a queue behind other requests Days later, an answer arrives — often too late to influence the decision it was meant to support The result is a quiet, systemic gap between what the business senses and what the data can confirm in time. Leaders fill that gap with instinct. Risks go unspotted. Opportunities pass. Not because the data was not there — but because reaching it took too long. This pattern repeats across every function. Finance cannot investigate a cost anomaly until after month-end close. Sales leadership walks into a quarterly review with numbers someone else prepared. Operations learns about a supplier risk from a weekly report that arrives after the damage is done. The Solution — Databricks Genie Genie is the conversational AI interface built into Azure Databricks. It lets a business leader type a question in plain English — the same way they would ask a colleague — and get an answer drawn from the organization’s actual data, in seconds. There is no form to fill in. No report to request. No specialist to involve for every question. The leader asks, the data responds, and the conversation continues — narrowing, refining, following the next logical question — until the insight is clear enough to act on. The approach rests on three capabilities working together: Conversational access — questions in plain English return precise answers from live data, with no technical skill required from the business user Governed trust — Genie works within existing data permissions; every user sees only what they are authorized to access, and every answer shows the logic behind it Seamless fit — it connects to data the organization already holds, whether from ERP systems, CRM platforms, or operational sources, without requiring a new build This is not a replacement for dashboards. It is what happens between them — the investigative, in-the-moment layer that dashboards were never designed to provide. Business Impact The impact of conversational data access compounds across the organization over time: Decisions get made closer to the moment they matter — leaders investigate anomalies in real time, not after a two-day analysis cycle The right questions finally get asked — when the cost of asking drops to near zero, the volume and quality of insight-driven decisions goes up across every function Data teams focus on higher-value work — instead of fielding one-off requests, analysts build the data models and pipelines that generate lasting value Existing investments go further — Genie extends what the organization has already built, without requiring new infrastructure or a technology overhaul The organization becomes AI-ready — consistent, governed use of data at every level builds the foundation for more advanced AI capabilities to follow The organizations that embrace this shift early will not just be faster. They will be fundamentally better at acting on what they know — and that is an advantage that compounds over time. Frequently Asked Questions Do we need to replace our existing dashboards or BI tools? No. Genie works alongside what you already have. Dashboards remain the right tool for structured, recurring reporting. Genie handles the ad-hoc, investigative questions that dashboards were not built to answer. They complement each other. Does this require technical skills from business users? No. Genie is designed for business users who have no data or SQL background. Questions are asked in plain English — the same way you would ask a colleague — and answers are returned in a readable format without any technical input required. Is the data secure? Can users access data they should not see? Genie inherits the data permissions already configured in your organization’s data environment. Every user sees only what they are already authorized to access. There is no additional access granted by using Genie — governance is built in, not added on. Does our data need to be moved or rebuilt to use Genie? Not necessarily. If your organization’s data — from ERP, CRM, operational systems, or other sources — is already in the Databricks environment, Genie can work with it immediately. For organizations not yet on Databricks, CloudFronts can help assess the right path forward. How is this different from asking an AI chatbot a question about our business? A general AI chatbot answers from its training data — it does not know your organization’s numbers. Genie queries your actual data directly. Every answer is grounded in your real figures, with the source and logic visible, making … Continue reading Go Beyond Dashboards- How Databricks Genie Gives Every Business Leader Direct Access to Their Data →

Organizations today generate massive volumes of raw data from multiple sources such as ERP systems, CRMs, APIs, logs, and IoT devices. However, raw data by itself holds little value unless it is properly processed, transformed, and optimized for analytics. In our data engineering journey, we faced challenges in building scalable and maintainable ETL pipelines that could handle growing data volumes while still delivering reliable insights. Azure Databricks helped us bridge the gap between raw data and business-ready insights. In this blog, we’ll walk through ETL best practices using Azure Databricks and how they helped us build efficient, production-grade data pipelines. Why ETL Best Practices Matter When working with large-scale data pipelines: – Raw data arrives in different formats and structures– Poorly designed ETL jobs lead to performance bottlenecks– Debugging and maintaining pipelines becomes difficult– Data quality issues propagate to downstream reports Key challenges we faced: – Tight coupling between ingestion and transformation– Reprocessing large datasets due to small logic changes– Lack of standardization across pipelines– Slow query performance on analytical layers Solution Architecture Overview Key Components: – Azure Data Lake Storage Gen2– Azure Databricks– Delta Lake– Power BI / Analytics Tools ETL Flow: – Ingest raw data from source systems into the Raw (Bronze) layer– Clean, validate, and standardize data in the Processed (Silver) layer– Apply business logic and aggregations in the Curated (Gold) layer– Expose curated datasets to reporting and analytics tools Step-by-Step ETL Best Practices with Azure Databricks Step 1: Separate Data into Layers (Bronze, Silver, Gold) – Bronze Layer: Store raw data exactly as received– Silver Layer: Apply cleansing, deduplication, and schema enforcement– Gold Layer: Create business-ready datasets and aggregations This separation ensures reusability and prevents unnecessary reprocessing. Step 2: Use Delta Lake for Reliability – Store tables in Delta format– Enable schema enforcement and schema evolution– Leverage time travel for data recovery and debugging Step 3: Build Incremental Pipelines – Process only new or changed data using watermarking– Avoid full reloads unless absolutely required– Design pipelines to safely re-run without duplications Step 4: Parameterize and Modularize Code – Use notebook parameters for environment-specific values– Create reusable functions for common transformations– Avoid hardcoding paths, table names, or business rules Step 5: Optimize Performance Early – Use partitioning based on query patterns– Apply Z-ORDER on frequently filtered columns– Cache datasets selectively for heavy transformations Step 6: Implement Data Quality Checks – Validate nulls, ranges, and duplicate records– Log rejected or invalid records separately– Fail pipelines early when critical checks fail Benefits of Following These ETL Best Practices – Scalability: Easily handle growing data volumes– Reliability: ACID-compliant pipelines with Delta Lake– Maintainability: Modular and reusable code structure– Performance: Faster queries and optimized storage– Cost Efficiency: Reduced compute usage through incremental processing Conclusion Transforming raw data into meaningful insights requires more than just moving data from one place to another. By following ETL best practices with Azure Databricks, we were able to build robust, scalable, and high-performing data pipelines that deliver reliable insights to the business. If your Databricks pipelines are becoming complex, slow, or difficult to maintain, it might be time to revisit your ETL design. Start applying these best practices today and turn your raw data into insights that truly drive decision-making. I hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.

In production Databricks environments, data issues such as accidental overwrites, faulty MERGE conditions, or incorrect backfills are common. Delta Lake’s Time Travel is not just a feature – it is a critical recovery and governance mechanism. This blog focuses only on practical recovery strategies that are actually used in real-world production systems. Why Time Travel Is Critical in Production Common failure scenarios include: •a. INSERT OVERWRITE wiping historical data • b. Incorrect MERGE conditions deleting valid records • c. Wrong filters during backfill corrupting data Reprocessing data is expensive and risky. Time Travel enables instant rollback with minimal impact. Version vs Timestamp (What You Should Use) Always prefer version-based time travel for recovery operations. Why version-based recovery is preferred: • a. Precise and deterministic • b. No time zone dependency • c. Safest option for production recovery Use timestamp-based queries only for auditing, not recovery. Identify the Last Safe State Before performing any recovery, always inspect the table history. DESCRIBE HISTORY crm_opportunities; Key fields to review: • a. version • b. timestamp • c. operation • d. userName This history acts as the single source of truth during incidents. Recovery Patterns That Actually Work 1. Partial Data Recovery (Recommended) Recover only the affected records instead of rolling back the entire table. Advantages: • a. No downtime • b. Safe for downstream reports • c. Most production-friendly approach 2. Full Table Restore (Use Carefully) Advantages: •a. Fast and atomic Risks: •a. Impacts all downstream consumers Use this approach only when the entire table is corrupted. Safe Validation Using CLONE Before restoring data in production, validate changes using a clone. Typical use cases: • a. Validate recovered data • b. Compare versions •c. Run business checks Retention & VACUUM (Most Common Mistake) The following command causes permanent data loss: Once vacuumed aggressively, time travel breaks and rollback becomes impossible. Production-Safe Retention Recommended retention: • a. Critical tables: 30 days • b. Reporting tables: 7–14 days Auditing & Root Cause Analysis (RCA) Track who changed data and when: Compare changes between versions: Key Best Practices • a. Capture table version before running risky jobs • b. Always use version-based time travel for recovery • c. Prefer partial recovery over full restores • d. Avoid aggressive VACUUM operations • e. Extend retention for critical tables • f. Validate using CLONE before restoring To conclude, Delta Lake Time Travel is not a backup mechanism, but it is the fastest and safest recovery tool in Databricks. When used correctly, it prevents downtime, reduces reprocessing cost, and improves production reliability. For enterprise Databricks pipelines, mastering this capability is mandatory, not optional. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Databricks Time Travel is a powerful feature of Delta Lake that allows you to access older versions of your data. Whether you want to debug issues, recover deleted records, compare historical performance, or audit how data changed over time—Time Travel makes it effortless. It’s like having a complete rewind button for your tables, eliminating the fear of accidental updates or deletes. What is Time Travel? Time Travel enables you to query previous snapshots of a Delta table using either VERSION AS OF or TIMESTAMP AS OF. Delta automatically versions every transaction-UPDATE, MERGE, DELETE, INSERT. So, you can always go back to an earlier state without restoring backups manually. This versioning is stored in the Delta Log, making rewind operations efficient and reliable. Why Time Travel Matters (Use Cases) Debugging Pipelines: Quickly check what the data looked like before a bad job ran. Accidental Deletes: Recover records or entire tables. Audit & Compliance: Easily demonstrate how data has evolved. Root Cause Analysis: Compare two versions side by side. Model Re-training: Use historical datasets to retrain ML models. Data Quality Tracking: Validate when incorrect data first appeared. How Delta Stores Versions (Architecture Overview) Delta Lake stores metadata and version history inside the _delta_log folder. Each commit creates a new JSON or checkpoint Parquet file representing table state. When you run a query using Time Travel, Databricks does not rebuild the entire table. Instead, it directly reads the snapshot based on the transaction log. This architecture makes Time Travel extremely fast and scalable—even on very large datasets. Time Travel Commands Query older data: SELECT * FROM table VERSION AS OF 5; SELECT * FROM table TIMESTAMP AS OF ‘2024-11-20T10:00:00’; A. Example: DESCRIBE HISTORY Below is an example of using DESCRIBE HISTORY on a Delta table. B. Querying a Specific Version Here is how you can fetch an older snapshot using VERSION AS OF. C. Restoring a Table You can restore a Delta table to any older version using RESTORE TABLE. Retention Rules Delta keeps older versions based on two configs: `delta.logRetentionDuration` → How long commit logs are stored. `delta.deletedFileRetentionDuration`→ How long old data files are retained. By default, Databricks keeps 30 days of history. You can increase this if your compliance policy requires longer retention. Best Practices – Use Time Travel for debugging pipeline issues. – Increase retention for sensitive or audited datasets. – Use `DESCRIBE HISTORY` frequently during development. – Avoid unnecessarily large retention windows—they increase storage costs. – Use `RESTORE` carefully in production environments. To conclude, time Travel in Databricks brings reliability, auditability, and simplicity to modern data engineering. It protects teams from accidental data loss and gives full visibility into how datasets evolve. With just a few commands, you can analyze, compare, or restore historical data instantly making it one of the most useful features of Delta Lake. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Enterprises today are collecting more data than ever before, yet most leaders admit they don’t fully trust the insights derived from it. Inconsistent formats, missing values, and unreliable sources create what’s often called a data swamp an environment where data exists but can’t be used confidently for decision-making.  Clean, trusted data isn’t just a technical concern; it’s a business imperative. Without it, analytics, AI, and forecasting lose credibility and transformation initiatives stall before they start.  That’s where the Medallion Architecture comes in. It provides a structured, layered framework for transforming raw, unreliable data into consistent, analytics-ready insights that executives can trust.  At CloudFront’s, a Microsoft and Databricks partner, we’ve implemented this architecture to help enterprises modernize their data estates and unlock the full potential of their analytics investments.  Why Data Trust Matters More Than Ever  CIOs and data leaders today face a paradox: while data volumes are skyrocketing, confidence in that data is shrinking.  Poor data quality leads to:  In short, when data can’t be trusted, every downstream process from reporting to machine learning is compromised. The Medallion Architecture directly addresses this challenge by enforcing data quality, lineage, and governance at every stage.  What Is the Medallion Architecture?  The Medallion Architecture is a modern, layered data design framework introduced by Databricks. It organizes data into three progressive layers Bronze, Silver, and Gold each refining data quality and usability.  This approach ensures that every layer of data builds upon the last, improving accuracy, consistency, and performance at scale.  Inside Each Layer    Bronze Layer —> Raw and Untouched   The Bronze Layer serves as the raw landing zone for all incoming data. It captures data exactly as it arrives from multiple sources, preserving lineage and ensuring that no information is lost. This layer acts as a foundational source for subsequent transformations.   Silver Layer —> Cleansing and Transformation   At the Silver Layer, the raw data undergoes cleansing and standardization. Duplicates are removed, inconsistent formats are corrected, and business rules are applied. The result is a curated dataset that is consistent, reliable, and analytics ready.   Gold Layer —> Insights and Business Intelligence   The Gold Layer aggregates and enriches data around key business metrics. It powers dashboards, reporting, and advanced analytics, providing decision-makers with accurate and actionable insights.   Example: Data Transformation Across Layers   Layer   Data Example   Processing Applied   Outcome   Bronze   Customer ID: 123, Name: Null, Date: 12-03-24 / 2024-03-12   Raw data captured as-is   Unclean, inconsistent   Silver   Customer ID: 123, Name: Alex, Date: 2024-03-12   Standardization & de-duplication   Clean & consistent   Gold   Customer ID: 123, Name: Alex, Year: 2024   Aggregation for KPIs   Business-ready dataset   This layered approach ensures data becomes progressively more accurate, complete, and valuable.   Building Reliable, Performant Data Pipelines  By leveraging Delta Lake on Databricks, the Medallion Architecture enables enterprises to unify streaming and batch data, automate validations, and ensure schema consistency creating an end-to-end, auditable data pipeline.  This layered approach turns chaotic data flows into a structured, governed, and performant data ecosystem that scales as business needs evolve.  Client Example: Retail Transformation in Action  A leading hardware retailer in the Maldives faced challenges managing inventory and forecasting demand across multiple locations. They needed a unified data model that could deliver real-time visibility and predictive insights.  CloudFront’s implemented the Medallion Architecture using Databricks:  Results:  Key Benefits for Enterprise Leaders  Final Thoughts  Clean, trusted data isn’t a luxury, it’s the foundation of every successful analytics and AI strategy.  The Medallion Architecture gives enterprises a proven, scalable framework to transform disorganized, unreliable data into valuable, business-ready insights.  At CloudFront’s, we help organizations modernize their data foundations with Databricks and Azure delivering the clarity, consistency, and confidence needed for data-driven growth.  Ready to move from data chaos to clarity? Explore our Databricks Services or Talk to a Cloud Architect to start building your trusted analytics foundation today.  We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Azure Databricks has become the go-to platform for data engineering and analytics, while Power BI remains the most powerful visualization tool in the Microsoft ecosystem. Connecting Databricks to Power BI bridges the gap between your data lakehouse and business users, enabling real-time insights from curated Delta tables. In this blog, we’ll walk through the process of securely connecting Power BI to Databricks, covering both DirectQuery and Import mode, and sharing best practices for performance and governance. Architecture Overview The connection involves:– Azure Databricks → Your compute and transformation layer.– Delta Tables → Your curated and query-optimized data.– Power BI Desktop / Service → Visualization and sharing platform. Flow:1. Databricks processes and stores curated data in Delta format.2. Power BI connects directly to Databricks using the built-in connector.3. Users consume dashboards that are either refreshed on schedule (Import) or query live (DirectQuery). Step 1: Get Connection Details from Databricks In your Azure Databricks workspace:1. Go to the Compute tab and open your cluster (or SQL Warehouse if using Databricks SQL).2. Click on ‘Advanced → JDBC/ODBC’ tab.3. Copy the Server Hostname and HTTP Path — you’ll need these for Power BI. For example:– Server Hostname: adb-1234567890123456.7.azuredatabricks.net– HTTP Path: /sql/1.0/endpoints/1234abcd5678efgh Step 2: Configure Databricks Personal Access Token (PAT) Power BI uses this token to authenticate securely.1. In Databricks, click your profile icon → User Settings → Developer → Access Tokens.2. Click Generate New Token, provide a name and expiration, and copy the token immediately. (You won’t be able to view it again.) Step 3: Connect from Power BI Desktop 1. Open Power BI Desktop.2. Go to Get Data → Azure → Azure Databricks.3. In the connection dialog:   – Server Hostname: paste from Step 1   – HTTP Path: paste from Step 14. Click OK, and when prompted for credentials:   – Select Azure Databricks Personal Access Token   – Enter your token in the Password field. You’ll now see the list of Databricks tables and databases available for import. To conclude, you’ve successfully connected Power BI to Azure Databricks, unlocking analytical capabilities over your Lakehouse. This setup provides flexibility to work in Import mode for speed or Direct Query mode for live data — all while maintaining enterprise security through Azure AD or Personal Access Tokens. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

Delta Lake is a storage layer that brings reliability, consistency, and flexibility to big data lakes. It enables advanced features such as Time Travel, Schema Evolution, and ACID Transactions, which are crucial for modern data pipelines. Feature Benefit Time Travel Access historical data for auditing, recovery, or analysis. Schema Evolution Adapt automatically to changes in the data schema. ACID Transactions Guarantee reliable and consistent data with atomic upserts. 1. Time Travel Time Travel allows you to access historical versions of your data, making it possible to “go back in time” and query past snapshots of your dataset. Use Cases:– Recover accidentally deleted or updated data.– Audit and track changes over time.– Compare dataset versions for analytics. How it works:Delta Lake maintains a transaction log that records every change made to the table. You can query a previous version using either a timestamp or a version number. Example: 2. Schema Evolution Schema Evolution allows your Delta table to adapt automatically to changes in the data schema without breaking your pipelines. Use Cases:– Adding new columns to your dataset.– Adjusting to evolving business requirements.– Simplifying ETL pipelines when source data changes. How it works:When enabled, Delta automatically updates the table schema if the incoming data contains new columns. Example: 3. ACID Transactions (with Atomic Upsert) ACID Transactions (Atomicity, Consistency, Isolation, Durability) ensure that all data operations are reliable and consistent, even in the presence of concurrent reads and writes. Atomic Upsert guarantees that an update or insert operation happens fully or not at all. Key Benefits:– No partial updates — either all changes succeed or none.– Safe concurrent updates from multiple users or jobs.– Consistent data for reporting and analytics.– Atomic Upsert ensures data integrity during merges. Atomic Upsert Example (MERGE): Here:– whenMatchedUpdateAll() updates existing rows.– whenNotMatchedInsertAll() inserts new rows.– The operation is atomic — either all updates and inserts succeed together or none. To conclude, Delta Lake makes data pipelines modern, maintainable, and error-proof. By leveraging Time Travel, Schema Evolution, and ACID Transactions, you can build robust analytics and ETL workflows with confidence, ensuring reliability, consistency, and adaptability in your data lake operations. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com