One Data Platform, Future-Ready for AI – Power BI Report Migration to Azure Databricks Summary Every organization wants to be AI-ready. Very few are data-ready. Business data typically lives across ERP, CRM, HR, and finance systems, each handled differently and reported separately. The result is familiar to most leadership teams: reports that do not agree, logic rebuilt in every dashboard, and AI initiatives that stall because the data underneath them cannot be trusted. This article makes the case for one data platform – a single, governed foundation where all business data is brought together, processed consistently, and made ready for both reporting and AI. We share how we built this at CloudFronts on Azure and Databricks, unifying five business systems and migrating 40+ Power BI reports with zero downtime, and why that foundation is what makes AI use cases practical rather than aspirational. Table of Contents 1 The Hidden Cost of Scattered Data 2 What One Data Platform Really Means 3 Why It Matters to the Business 4 Solution Architecture Overview 5 Migration Without Disruption 6 From Reporting Platform to AI Foundation 7 AI Use Cases It Unlocks 8 Scattered vs Unified 9 Questions Every Leader Should Ask 10 Where to Start 11 FAQs 12 Conclusion The Hidden Cost of Scattered Data Most growing organizations do not set out to fragment their data. It happens one decision at a time. A new ERP for finance. A CRM for sales. An HR system for people. An accounting tool for a specific entity. Each one solves a real problem – and each one becomes another separate source of data. This project started with our own reporting. Like many growing services organizations, CloudFronts runs its business across several systems: Business Central and Finance & Operations for ERP, Dynamics 365 CRM for clients and engagements, Zoho People for HR, and Zoho Books for accounting. Leadership, finance, HR, and delivery teams rely on more than 40 Power BI reports built on these systems, from management dashboards and HR reports to the Steering Committee report we use to track quarterly meetings with client accounts. Over time, each system had been connected in its own way. Some data moved through Azure Data Factory, some through SQL Database, and some through Azure Functions. Reports connected directly to their sources, with business logic built inside each one, and they sat across separate workspaces for HR, management, and other teams. The idea behind the project was simple: instead of continuing to add pipelines and reports system by system, bring all business data into one trusted platform on Azure Databricks, migrate every existing report onto it without disrupting the people who use them, and build a foundation that is ready for AI in the process. None of the old setup looked broken on any given day. But the costs were real, and they compounded as the business grew. Numbers That Do Not Agree When the same metric is calculated in several reports, small differences appear. Leadership meetings start with reconciling numbers instead of acting on them. Effort That Keeps Growing Every new report means another connection, another copy of the logic, and another refresh to maintain. The effort grows with every report added. Governance That Is Hard to Enforce When reports sit in different workspaces with different owners, controlling who sees what becomes a manual, error-prone exercise. Data That Is Not Ready for What Is Next AI and advanced analytics need clean, connected data. Scattered data means every AI idea starts with a data consolidation project first. “The data was never the problem. The problem was that it lived in five places, and every system handled it differently.” What One Data Platform Really Means One data platform is not just a new database or a bigger data warehouse. It is a change in how the organization treats its data – from something each system and report manages on its own, to a shared business asset managed once, for everyone. Functionally, it comes down to four commitments: 1One place for all business data – Every source system feeds a single platform, so finance, sales, HR, and operations data sit side by side and can be connected. 2One way of processing it – Every source goes through the same standardized stages, so data is cleaned and shaped consistently no matter where it came from. 3One definition of every business rule – Business logic – how revenue is grouped, how a meeting is classified, how an employee is counted – is defined once and reused everywhere. 4One layer of governance – Access is controlled centrally by role, so the right people see the right data without each report owner managing permissions separately. Why It Matters to the Business For leadership, the value of one data platform shows up in outcomes rather than architecture. A Single Source of Truth Every report reads from the same curated data, so the numbers agree across finance, operations, and management reporting. Faster Answers, Less Rework New reports and new data sources are added through configuration, not rebuilt from scratch. Teams spend time on insight instead of plumbing. Stronger Governance by Design Centralized, role-based access means security is enforced by the platform, not by individual report owners. Lower Long-Term Complexity One pipeline and one framework replace a patchwork of tools, reducing maintenance and dependency on specific individuals. Scales With the Business Adding a new system or entity follows the same pattern, so growth does not multiply reporting complexity. A Foundation for AI Clean, connected, governed data is exactly what AI, machine learning, and process mining need to deliver reliable results. Solution Architecture Overview We built our one data platform on Azure and Databricks. Here is how it fits together, from five source systems to a single, governed Power BI workspace. How the Architecture Works Step 1: Connect the Source SystemsBusiness Central, Finance & Operations, Dynamics 365 CRM, Zoho People, and Zoho Books are registered once in a central ingestion list that tells the platform what data to bring in. … Continue reading From Multiple Data Sources to Azure Databricks: A Zero-Downtime Power BI Migration for Future AI Readiness →

Summary Enterprises running large, multi-country ERP systems often extract data that is technically complete but practically unusable, split across duplicate language columns and encoded with undocumented numeric values. We built a self-service data platform on Azure so that business users, not just data engineers, could define, validate, and process ERP extracts without writing a single line of code. The solution resolves two of the most common ERP data problems: a single field like “Item Description” spread across nine language-specific columns, and reference fields like “Order Status” stored only as numeric codes. A custom web portal puts business users in control of table specifications, validation rules, and processing status, while Azure Databricks and Delta Lake quietly do the heavy lifting behind the scenes. Business impact: dozens of ERP tables moved from raw, multi-language, code-heavy extracts to a single, trusted, human-readable data layer, without adding headcount to the data engineering team. Table of Contents 01  About the Customer 05  Self-Service Data Onboarding 02  The Challenge 06  Medallion Architecture 03  The Solution 07  Business Impact 08  FAQs 09  Conclusion About the Customer Customer Spotlight A Leading Digital Transformation Partner — Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Most organizations extracting data from a large ERP system successfully get the data out. The problem isn’t extraction, it’s making that data mean something the moment it lands. Business and IT teams found themselves asking the same questions on repeat: 1Why does the same field appear nine times, with a different value in each column? 2What does “Order Status= 3” actually mean, and who is the source of truth for that mapping? 3How much manual translation and lookup work happens before a single report can be trusted? 4Can business users resolve these issues themselves, without waiting weeks on an IT backlog? 5How do we scale this across dozens of tables without writing dozens of one-off scripts? Two problems came up again and again, and both are far more common across ERP implementations than most leadership teams realize. Multi-Language Columns Because the ERP system was configured for every Order Status the business operates in, a single logical field such as “Item Description” existed as up to nine separate columns, one per language: English, French, German, Spanish, and more. Reports built directly on top of the raw extract had no reliable way of knowing which column to use for which record. In practice, this meant a plant manager in France could open a report and see item names in German, while a sales report for the Spanish market silently pulled blank fields because the Spanish-language column hadn’t been populated for that record. The data was all there; it just wasn’t usable without someone manually deciding, table by table, which language column to trust. Undocumented Domain Values Reference fields like Country, Currency, and Order Status were stored as raw numeric codes rather than readable labels, for example Order Status: 1 = Completed , 2 = In Progress, 3 = Shipped. These mappings lived inside ERP configuration screens, not in the extracted data itself. That meant every downstream report, dashboard, or spreadsheet needed its own copy of the same lookup table, manually kept in sync. When a code changed or a new Order Status was added in the ERP, there was no guarantee every report using it would be updated at the same time, which meant leadership could be looking at the performance chart that was quietly wrong. The Solution Rather than writing custom transformation logic for every table (a solution that ages badly the moment a new table or region gets added), we designed a configuration-driven pipeline built on Azure Databricks, fronted by a self-service web application that puts control directly in the hands of business and functional users. Self-Service Web Portal Business users upload table specifications, review validation results, and queue tables for processing, entirely through a browser. Medallion Architecture Azure Databricks and Delta Lake refine raw extracts through Bronze, Silver, and Gold layers, without table-specific code. Automated Language Resolution Multi-language columns are detected and normalized automatically based on the specification, not hardcoded per table. Centralized Domain Mapping Numeric and coded reference values are resolved against a single, maintained lookup layer instead of scattered spreadsheets. Self-Service Data Onboarding: No Databricks Knowledge Required The centerpiece of the solution is a custom web application that lets a business or functional analyst, not a Databricks engineer, onboard a new ERP table from start to finish. Here’s what that looks like in practice: A business user uploads an Excel-based table specification defining the expected columns, data types, which fields are multi-language, and which fields are domain-coded and how to decode them. The portal validates the specification instantly, flagging missing mandatory columns or mismatches before any data is processed, so problems are caught at the source rather than three reports downstream. Once validation passes, the same user queues the table for processing with a single click. No notebook to open, no cluster to configure, no code to write or review. Behind the scenes, that specification feeds a generic, reusable Databricks framework that already knows how to apply the correct language resolution and domain-value decoding rules, so engineering effort doesn’t scale linearly with the number of tables. In effect, the portal turns “add a new ERP table to the analytics environment” from a data engineering request into a form a finance or operations analyst can complete in minutes, while still enforcing the same rigor and consistency a hand-built pipeline would require. Medallion Architecture on Databricks Once a table is queued through the portal, Azure Databricks takes over: Bronze: Raw ERP extracts are landed as-is, preserving … Continue reading How a Self-Service Data Portal Solved Multi-Language and Domain Value Chaos in ERP Data →

In production Databricks environments, data issues such as accidental overwrites, faulty MERGE conditions, or incorrect backfills are common. Delta Lake’s Time Travel is not just a feature – it is a critical recovery and governance mechanism. This blog focuses only on practical recovery strategies that are actually used in real-world production systems. Why Time Travel Is Critical in Production Common failure scenarios include: •a. INSERT OVERWRITE wiping historical data • b. Incorrect MERGE conditions deleting valid records • c. Wrong filters during backfill corrupting data Reprocessing data is expensive and risky. Time Travel enables instant rollback with minimal impact. Version vs Timestamp (What You Should Use) Always prefer version-based time travel for recovery operations. Why version-based recovery is preferred: • a. Precise and deterministic • b. No time zone dependency • c. Safest option for production recovery Use timestamp-based queries only for auditing, not recovery. Identify the Last Safe State Before performing any recovery, always inspect the table history. DESCRIBE HISTORY crm_opportunities; Key fields to review: • a. version • b. timestamp • c. operation • d. userName This history acts as the single source of truth during incidents. Recovery Patterns That Actually Work 1. Partial Data Recovery (Recommended) Recover only the affected records instead of rolling back the entire table. Advantages: • a. No downtime • b. Safe for downstream reports • c. Most production-friendly approach 2. Full Table Restore (Use Carefully) Advantages: •a. Fast and atomic Risks: •a. Impacts all downstream consumers Use this approach only when the entire table is corrupted. Safe Validation Using CLONE Before restoring data in production, validate changes using a clone. Typical use cases: • a. Validate recovered data • b. Compare versions •c. Run business checks Retention & VACUUM (Most Common Mistake) The following command causes permanent data loss: Once vacuumed aggressively, time travel breaks and rollback becomes impossible. Production-Safe Retention Recommended retention: • a. Critical tables: 30 days • b. Reporting tables: 7–14 days Auditing & Root Cause Analysis (RCA) Track who changed data and when: Compare changes between versions: Key Best Practices • a. Capture table version before running risky jobs • b. Always use version-based time travel for recovery • c. Prefer partial recovery over full restores • d. Avoid aggressive VACUUM operations • e. Extend retention for critical tables • f. Validate using CLONE before restoring To conclude, Delta Lake Time Travel is not a backup mechanism, but it is the fastest and safest recovery tool in Databricks. When used correctly, it prevents downtime, reduces reprocessing cost, and improves production reliability. For enterprise Databricks pipelines, mastering this capability is mandatory, not optional. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com

The fastest way to lose control of enterprise data? Managing governance separately across workspaces. Unity Catalog solves this with one centralized layer for security, lineage, and discovery. Data governance is crucial for any organization looking to manage and secure its data assets effectively. Databricks’ Unity Catalog is a centralized solution that provides a unified interface for managing access control, auditing, data lineage, and discovery. This blog will guide you through the process of setting up Unity Catalog in your Databricks workspace. What is Unity Catalog? Unity Catalog is Databricks’ answer to centralized data governance. It enables organizations to enforce standards-compliant security policies, apply fine-grained access controls, and visualize data lineage across multiple workspaces. It ensures compliance and promotes efficient data management. Key Features: 1] Standards-Compliant Security: ANSI SQL-based access policies that apply across all workspaces in a region. 2] Fine-Grained Access Control: Support for row- and column-level permissions. 3] Audit Logging: Tracks who accessed what data and when. 4] Data Lineage: Provides visualization of data flow and dependencies. Unity Catalog Object Hierarchy Before diving into the setup, it’s important to understand the hierarchical structure of Unity Catalog: 1] Catalogs: The top-level container (e.g., Production, Development) that represents an organizational unit or environment. 2] Schemas: Logical groupings of tables, views, and AI models within a catalog. 3] Tables and Views: These include managed tables fully governed by Unity Catalog and external tables referencing existing cloud storage. Here is the procedure to setup a Unity Catalog Metastore in association with Azure Storage, as I have done for one of our products (SmartPitch Sales & Marketing Agent) – 1] First create a storage account  with primary service being – “Azure Blob Storage or Azure Data Lake Storage Gen 2”; Performance and Redundancy can be chosen based on the requirement for which the DataBricks service is being used.Here for my Mosaic AI Agent, I have used Locally Redundant Storage & Data Lake Gen 2 2] Once the storage account is created, ensure that you have enabled “Hierarchical Namespace” When creating a Unity Catalog metastore with Azure Blob Storage, Hierarchical Namespace (HNS) is required because Unity Catalog needs: a] Folder-like structure to organize catalogs, schemas, and tables. b] Atomic operations (rename, move, delete) on directories and files. c] POSIX-style access controls for fine-grained permissions. d] Faster metadata handling for lineage and governance. HNS turns Azure Blob into ADLS Gen2, which supports these features. 3] Upload any Raw/Unclean files to your metastore folder in the blob storage, which would be required for your use in DataBricks. 4] Create a Unity Catalog Connector in Azure Portal and assign it “Storage Blob Data Contributor” Role . 5] Assign CORS (Cross-Origin Resource Sharing) settings for that storage account. Why this is necessary: In short: Without configuring CORS, Databricks cannot communicate with your storage container to read/write managed tables, schema metadata, or logs. 6] Generate SAS Token   7] Navigate to your Workspace and select “Manage Account” – this should be done from the account admin.  8] Select Catalog tab on the left and then click “Create Metastore”    9] Assign a Name, Region (Same as Workspace), The path to the storage account, and the connector id. 10] Once the Metastore is created, assign it to a workspace .  11] Once this is done, the catalogs and the schemas, and tables in within it can be created. How does Unity Catalog differ from Hive Metastore ? Feature Hive Metastore Unity Catalog Scope Workspace or cluster-specific Centralized, spans multiple workspaces and regions Architecture Single metastore tied to Spark/Hive Cloud-native service integrated with Databricks Object Hierarchy Databases → Tables → Partitions Catalogs → Schemas → Tables/Views/Models Data Assets Supported Tables, views Tables, views, files, ML models, dashboards Security Basic GRANT/DENY at database/table level Fine-grained, ANSI SQL–based (catalog, schema, table, column, row) Lineage Not available Built-in lineage and impact analysis Auditing Limited or external Integrated audit logs across workspaces Storage Management Points to storage locations; no governance Manages external and managed tables with governance Cloud Integration Primarily on cluster storage or external path Secure integration with ADLS Gen2, S3, GCS Permissions Model Spark SQL statements Attribute- and role-based access, unified policies Use Cases Basic metadata store for Spark/Hive workloads Enterprise-wide data governance, sharing, and compliance To conclude, Unity Catalog is the next-generation governance and metadata solution for Databricks, designed to give organizations a single, secure, and scalable way to manage data and AI assets. Unlike the older Hive Metastore, it centralizes control across multiple workspaces, supports fine-grained access policies, delivers built-in lineage and auditing, and integrates seamlessly with cloud storage like Azure Data Lake, S3, or GCS. When setting it up, key steps include: 1] Creating a metastore and linking it to your workspaces. 2] Enabling hierarchical namespace on Azure storage for folder-level security and operations. 3] Configuring CORS to allow Databricks domains to interact with storage. 4] Defining catalogs, schemas, and tables for structured governance. By implementing Unity Catalog, you ensure stronger security, better compliance, and faster data discovery, making your Databricks environment enterprise-ready for analytics and AI. Business Outcomes of Unity Catalog By implementing Unity Catalog, organizations can achieve: Why now? As data volumes and regulatory requirements grow, organizations can no longer rely on fragmented or legacy governance tools. Unity Catalog offers a future-proof foundation for unified data management and AI governance—essential for any modern data-driven enterprise. At CloudFronts, we help enterprises implement and optimize Unity Catalog within Databricks to ensure secure, compliant, and scalable data governance for enterprise data governance.Book a consultation with our experts to explore how Unity Catalog can simplify compliance and boost productivity for your teams.Contact us today at Transform@cloudfronts.com to get started. To learn more about functionalities of DataBricks and other Azure AI services, please refer to my other blogs from the links given below: – 1] The Hidden Cost of Bad Data:How Strong Data Management Unlocks Scalable, Accurate AI – CloudFronts 2] Automating Document Vectorization from SharePoint Using Azure Logic Apps and Azure AI Search – CloudFronts 3] Using Open AI and Logic Apps to develop a Copilot agent for … Continue reading Setting Up Unity Catalog in Databricks for Centralized Data Governance →