How We Built a Real-Time Lightweight Financial Statement Reporting Experience Directly Inside Microsoft Dynamics 365 Project Operations Summary Designed and deployed a lightweight, real-time financial statement reporting solution directly inside Microsoft Dynamics 365 Project Operations for a Texas-based Cybersecurity & AI Business Solutions firm. Eliminated dependency on heavy paginated reporting and large-scale Power BI datasets for operational financial visibility. Built an interactive HTML + JavaScript reporting framework embedded natively within Dynamics 365 CRM. Enabled dynamic filtering, instant report rendering, and printable customer-ready statements directly from the CRM interface. Introduced popup-based full-screen report rendering for detailed review and print-ready output without leaving Dynamics 365. Integrated funding balances, allocations, transactions, installment schedules, and financial snapshots into a single operational reporting experience. Reduced reporting development complexity, minimized data transformation overhead, and improved scalability compared to traditional BI-heavy architectures. Created a highly maintainable reporting model that scales efficiently as operational datasets grow without introducing significant Power BI licensing or performance constraints. Table of Contents Introduction The Business Problem The Solution Architecture Real-Time CRM-Native Reporting Lightweight Front-End Reporting Framework Popup-Based Printable Report Experience Data Model and Reporting Components Design Principles Business Impact Why This Approach Worked FAQs Conclusion 1. Introduction As organizations scale, operational reporting often becomes increasingly difficult to maintain. For a Texas-based Cybersecurity & AI Business Solutions firm operating on Microsoft Dynamics 365 Project Operations, this challenge became especially visible in financial agreement tracking and customer funding visibility. The business already had access to reporting platforms such as Power BI and paginated reports. However, these approaches introduced several operational problems: Long development cycles Heavy data-cleaning requirements Complex transformation pipelines Delayed visibility into operational data Increasing licensing costs as datasets expanded Slow report rendering for operational users Dependency on external reporting infrastructure Instead of another external BI layer, the organization wanted a lightweight operational reporting experience directly inside Dynamics 365 CRM itself. The Goal: Build a real-time, CRM-native financial reporting experience that renders instantly, supports dynamic filtering, enables printing, and scales without heavy BI infrastructure. 2. The Business Problem The organization manages multiple long-running service agreements, funding allocations, installment schedules, and customer financial balances across cybersecurity services, managed services, and AI solution engagements. Operational users needed a consolidated statement experience that could answer questions such as: What is the customer’s current available balance? Which transactions impacted the balance during a selected period? Which allocations are currently active? How much funding has been consumed vs allocated? Which installments are pending, paid, or overdue? What does the latest funding snapshot look like? Can the report be reviewed and printed directly from CRM? Paginated Reporting Limitations Increasing query complexity Performance degradation with larger datasets Heavy formatting maintenance Limited interactivity Rigid deployment cycles Power BI Challenges Significant Power Query transformations Data-cleaning pipelines Incremental refresh considerations Dataset refresh latency Licensing growth with scale Overengineering for transactional operational reporting 3. The Solution Architecture The reporting framework was designed as a native Dynamics 365 embedded reporting experience using: HTML Web Resources JavaScript Dynamics 365 Web API Native CRM navigation APIs Real-time entity retrieval Popup-based print rendering Embedded Operational Report Apply filters Select funding records Choose reporting periods Generate statements instantly Navigate operational financial data Popup Print Report Detailed review Executive presentation Customer-facing statements Printing and PDF generation 4. Real-Time CRM-Native Reporting One of the most important architectural decisions was avoiding external data replication entirely. Instead of pushing transactional data into a separate reporting warehouse, the report retrieved data directly from Dynamics 365 using the native Web API. Real-time visibility Zero synchronization lag Reduced infrastructure complexity Lower maintenance overhead Faster deployment cycles Everything rendered on demand inside the CRM session itself. 5. Lightweight Front-End Reporting Framework The reporting experience was intentionally designed to behave more like a modern application than a traditional report. Dynamic Filter Bar Users could dynamically filter reports using: This Month Last Month This Quarter Current Year Custom Date Ranges Funding Status Funding Selection The report regenerated instantly without page reloads. Responsive Report Rendering The reporting layout dynamically populated: Account Summary Transaction Details Allocation Summary Installment Details Detailed Account Summary Each section rendered independently based on live API responses. Intelligent Empty-State Handling Instead of showing blank tables or errors, the framework displayed contextual empty-state messaging such as: “No transactions during this statement period” “No active allocations” “No installment details available” This significantly improved usability for operational teams. 6. Popup-Based Printable Report Experience A major requirement was enabling users to thoroughly review and print reports directly from CRM. To solve this, the solution introduced a dedicated popup rendering architecture. Users could click: “Expand Report” This launched a fullscreen popup using Dynamics 365 navigation APIs with: Large-format rendering Print-optimized layout Full customer statement formatting Multi-page support Consistent branding Printable tables Customer reference guides The popup approach delivered several advantages: Better readability Cleaner print formatting Improved executive review experience Isolation from CRM form clutter Easier PDF generation Most importantly, the popup still worked entirely against live CRM data. 7. Data Model and Reporting Components The report consolidated multiple operational areas into a single experience. Account Summary Provided a high-level balance overview including: Balance Forward Total Credits Total Debits Closing Balance This gave immediate visibility into customer financial standing. Transaction Details Displayed detailed running balance activity including: Document date Transaction description Service type Credits Debits Running balance Transactions dynamically recalculated balances during rendering. Allocation Summary Tracked funding allocation activity including: Allocated funds Consumed funds Remaining balance Allocation status Returned allocations were handled separately with custom date logic. Installment Tracking Displayed installment lifecycle visibility including: Invoice dates Due dates Payment dates Payment terms Installment status The report intelligently handled future-dated payments and pending statuses. Detailed Funding Snapshot Displayed operational funding metrics including: Starting Balance Contracted Funds Total Budgeted Funds Collected Funds Used Funding Available Funds Allocated Funds Unallocated Funds This created a complete operational funding overview within a single screen. 8. Design Principles Several architectural principles guided the solution. Real-Time Over Batch Processing Operational reporting should reflect current business activity immediately. The solution avoided overnight refresh cycles entirely. Lightweight Over Heavy BI Not … Continue reading How We Built a Real-Time Lightweight Financial Statement Reporting Experience Directly Inside D365 PO for a Texas-Based Cybersecurity Firm →

Building a Controlled Booking-to-Time Entry Import Framework Inside Dynamics 365 Project Operations Summary Two Texas-based firms — one in Cybersecurity, another in Operational Security — required a streamlined and controlled Time Entry (TE) creation process inside Dynamics 365 Project Operations. Native D365 Project Operations limitations around Project Task visibility, booking-driven TE creation, and inconsistent resource submissions created operational inefficiencies. A fully customized solution was implemented directly inside Dynamics 365 CRM using HTML Web Resources, JavaScript, Dataverse Web API, Ribbon Enable Rules, and custom plugins. The solution centralized TE creation under Project Managers and Project Approvers, enabling controlled and secure booking-based TE management. A custom booking import framework dynamically surfaced only authorized projects and resources based on Project Approver relationships. Custom plugin logic and Resource Assignment–based task resolution automated Project Task mapping for accurate Time Entry creation. Key capabilities delivered included controlled booking imports, role-based visibility, automated task association, external comments support, and bulk TE creation. Dynamic filtering ensured Project Managers could only access resources and bookings associated with projects they were authorized to manage. The entire experience operated natively inside Dynamics 365 Project Operations without external portals, Power Apps screens, or third-party applications. The implementation reduced manual effort, improved TE submission reliability, increased operational flexibility, and enabled more accurate tracking of actual project work. Table of Contents Introduction The Business Problem & Pain Points The Solution Architecture Implementation Design Principles Business Impact Why This Approach Worked FAQs Conclusion 1 Introduction Two Texas-based firms operating in the Cybersecurity and Operational Security space relied heavily on Dynamics 365 Project Operations for project delivery tracking, resource management, and operational execution. As project operations scaled, Project Managers and Project Approvers required a faster and more controlled mechanism for creating Time Entries (TEs) directly from resource bookings. The organizations needed a solution that could simplify booking imports, improve Project Task mapping, enforce role-based visibility, and reduce the dependency on individual resources for manual TE submissions. Operationally, Project Managers were often responsible for validating and entering actual work performed, making the standard TE process inefficient and time-consuming. Key Challenges Standard Dynamics 365 Project Operations behavior did not fully support project-task-aware Time Entry creation from bookings. Project Task values were not consistently available across Resource Requirements and bookings in several PO environments. Resource-driven TE submission resulted in inconsistent and delayed operational reporting. Project Managers lacked centralized visibility and controlled access to resource bookings across approved projects. Native booking import and TE creation workflows lacked flexibility for operational governance and scalability. Goals of the Solution Centralize Time Entry creation under Project Managers and Project Approvers. Enable controlled booking imports with role-based project visibility. Automate Project Task association during TE creation. Allow bulk creation of booking-driven Time Entries directly inside CRM. Improve operational accuracy, flexibility, and governance without relying on external applications or custom portals. 2 The Business Problem & Pain Points 1. Native Booking-to-Time Entry Limitations Standard Dynamics 365 Project Operations behavior did not consistently expose Project Task information through Resource Requirements and Bookings. This created gaps in task-aware Time Entry creation and forced users to manually reconstruct operational context during the TE process. 2. Lack of Controlled Booking Visibility Default system behavior provided broader booking visibility than operationally required. The organizations needed a controlled access model where only designated Project Managers and Project Approvers could view and manage booking imports for authorized projects. 3. High Manual Effort in Time Entry Creation Project Managers and operational teams spent significant time manually entering project references, tasks, durations, and external comments for each Time Entry. This increased administrative overhead and reduced operational efficiency. 4. Inconsistent Resource-Driven Submission Process The organizations faced reliability challenges with resource-submitted Time Entries, leading to delays, missing entries, and inconsistencies in operational reporting. Project Managers required centralized ownership over TE creation to ensure accurate work tracking. 5. Fragmented User Experience Users were required to navigate across multiple Dynamics 365 screens and entities to complete routine booking import and Time Entry operations, making the process cumbersome and inefficient for daily operational usage. 6. Scalability and Maintainability Concerns The firms required a lightweight and scalable solution that could operate natively within Dynamics 365 Project Operations without introducing unnecessary Power Apps layers, external portals, or high-maintenance custom applications. 3 The Solution Architecture Architecture Diagram and Flow Figure: Complete Frontend – Backend behaviour of the TE Automation Module. Dynamics 365 Ribbon Workbench A custom “Import Resource Bookings” ribbon action was introduced to provide controlled access to the booking import process only for authorized Project Managers and Project Approvers. JavaScript + Dataverse Web API JavaScript and Dataverse Web API were used to handle dynamic project filtering, approver validation, booking retrieval, task mapping, and automated Time Entry creation directly inside CRM. HTML Web Resources Two custom HTML-based interfaces were developed: Resource Selection Interface — controlled resource visibility and selection Booking Import & TE Creation Interface — booking imports, task selection, external comments, and bulk Time Entry creation Dataverse Plugin Layer A lightweight custom C# plugin was implemented to support Project Task resolution, task validation, and booking-to-Time Entry automation scenarios not fully supported natively in Dynamics 365 Project Operations. Dataverse Entities Involved The solution leveraged multiple Project Operations entities: msdyn_project msdyn_projectteam msdyn_resourceassignment msdyn_projecttask bookableresource msdyn_resourcerequirement bookableresourcebooking msdyn_timeentry Together, these entities enabled secure, project-aware, and task-aware operational workflows directly inside Dynamics 365 CRM. Entity Relationships Figure: Relationships and associations of the involved entities. 4 Implementation 1. Role-Controlled Ribbon Visibility A custom ribbon action was implemented to ensure only authorized Project Managers and Project Approvers could access the booking import functionality. Visibility was dynamically controlled based on project approval relationships inside Dynamics 365. Figure: Case 1: When Logged in as a Project Approver/Manager. Figure: Case 2: When NOT Logged in as a Project Approver/Manager. 2. Resource Selection Experience A custom resource selection interface was developed to display only eligible resources associated with projects managed by the logged-in approver. This provided secure and simplified operational visibility. Figure: Bookable Resource Selection from a list of Active Bookable Resources, which are under any Project, where the current … Continue reading Building a Controlled Booking-to-Time Entry Import Framework Inside Dynamics 365 Project Operations for Texas-Based Operational Security & Cybersecurity Firms →

Summary Designed a two-page Power BI Income Pipeline Report for a Texas-based Cybersecurity & AI Business Solutions firm using Microsoft Dynamics 365 Project Operations. Unified visibility across Opportunity, Unbilled Income, Billed Income, and Paid Income in a single view. Introduced Average Turnaround to forecast realistic cash collection timelines based on actual payment behavior. Integrated Dynamics 365 Project Operations with QuickBooks to connect sales, delivery, invoicing, and cash collection. Enabled a 17-week rolling revenue forecast with week-by-week cash visibility. Provided dual invoice status for contractual vs realistic payment tracking. Table of Contents 1. Introduction 2. The Business Problem 3. Report Structure Overview 4. The Income Pipeline 5. Project Revenue Forecast 6. Design Principles 7. Business Impact 8. FAQs 9. Conclusion 1. Introduction Managing revenue across a professional services firm is rarely straightforward. When your business spans cybersecurity assessments, AI-driven solutions, and long-term managed services engagements, the gap between work being delivered and cash actually landing in the bank can be wide — and costly if left unmonitored. This is precisely the challenge we set out to solve for a U.S.-based Cybersecurity and AI Business Solutions firm running their operations on Microsoft Dynamics 365 Project Operations. The result was a two-page Power BI report — the Income Pipeline Report — that gives leadership a real-time, end-to-end view of every dollar moving through the business: from early-stage opportunity, through unbilled and billed income, all the way to cash collected. This post walks through how the report was built, how each data layer was modelled, and why the design decisions were made the way they were. 2. The Business Problem The firm needed clarity across four distinct but connected stages of their revenue lifecycle: Sales opportunities and pipeline value Delivered but unbilled work Outstanding invoices and expected payments Actual vs expected payment behavior This would answer as well as resolve the following questions – Where are active sales opportunities sitting, and how much pipeline value do they represent? Which project work has been delivered but not yet invoiced? Which invoices have been raised and sent to clients, and when are they realistically going to be paid? And finally, how does actual payment behaviour compare against what was expected? Each of these questions existed in isolation before. Project managers had partial visibility into their own contracts, and needed a comprehensive bird’s eye view of all of these together. Finance had QuickBooks data but lacked the context of the delivery pipeline. Leadership had no consolidated view. The Income Pipeline Report brought all of this together in a single, navigable Power BI experience. 3. Report Structure Overview The report consists of two pages: Income Pipeline Report — a high-level pipeline view across four stages: Opportunity, Unbilled Income, Billed Income, and Paid Income, each with summary cards and interactive donut charts. Project Revenue Forecast — a time-distributed breakdown of expected cash collection across a rolling 17-week horizon, organised by customer and contract. 4. The Income Pipeline The Four-Stage Pipeline Banner Across the top of the report, four chevron-style stage indicators guide the revenue journey: Opportunity → Unbilled Income → Billed Income → Paid Income Each stage includes a summary card showing record count and total value Provides immediate visibility into where revenue is sitting Highlights potential bottlenecks across the pipeline Stage 1 — Opportunity Data sourced from Dynamics 365 Sales using Business Process Flow (BPF) Uses active BPF stage (Develop, Propose, Close) instead of static fields Ensures accurate reflection of real sales progression Estimated revenue pulled directly from opportunity records Donut chart shows distribution across Develop, Propose, and Close stages Stage 2 — Unbilled Income Represents contracted or delivered work not yet invoiced Sourced from project contract lines in Dynamics 365 Project Operations Includes: Fixed Fee milestones (explicit values) Time & Material (T&M) estimates based on resource allocations T&M calculated as allocated hours × billing rate Clearly marked as estimated until billing run is executed Grouped into payment expectation buckets (30, 60, 90, 120, 180+ days) Uses Average Turnaround to forecast realistic payment timing Stage 3 — Billed Income (Confirmed Invoices) Combines Dynamics 365 Project Operations and QuickBooks data Tracks invoices that are confirmed and sent to clients Introduces Average Turnaround: Average days from invoice creation to payment Based on historical payment behaviour Each invoice has two statuses: Contractual (due date) Estimated (based on Average Turnaround) Provides realistic vs contractual payment visibility Includes: Due-date based categorisation Estimated overdue analysis Prevents misleading insights from strict payment terms alone Stage 4 — Paid Income Tracks fully collected invoices Uses QuickBooks for actual payment dates Groups payments by time bands (under 30, 60, 90 days, etc.) Enables comparison between actual vs estimated payment behaviour Continuously improves accuracy of Average Turnaround Tooltip Drill-Down Hover shows: Payment band Record count Total value Drill-through available for detailed record-level analysis 5. Project Revenue Forecast Overview Distributes expected cash collection across a rolling 17-week window Shifts view from pipeline stage to time-based forecasting Hierarchy and Structure Customer → Contract → Revenue Type Revenue types include: T&M run schedules Fixed Fee milestones Confirmed invoices Each row shows: Customer Contract Billing type Average Turnaround Value mapped to expected payment week Weeks range from Week 0 to Week 16 Top row aggregates total expected cash per week Colour Coding Amber — Unbilled income Green — Invoice within terms Red — Overdue (based on estimated payment date) Drill-Through to Detail Click any row to view detailed breakdown Includes: Billed invoices with due and estimated dates Unbilled milestones and run schedules Connects high-level forecast to transactional detail 6. Design Principles Average Turnaround over payment terms Reflects actual customer behaviour instead of contractual assumptions. Dual invoice status Provides both contractual and realistic payment visibility. Consistent time buckets Ensures comparability across Opportunity, Unbilled, Billed, and Paid stages. Weekly forecasting instead of monthly Supports short-term cash flow planning aligned with operational rhythm. 7. Business Impact Improved cash flow predictability Earlier visibility of at-risk invoices Unified cross-team visibility Improved T&M billing discipline Increased accountability 8. FAQs What is Average Turnaround and why does it … Continue reading How we designed & deployed an Income Pipeline Report for a Texas, U.S. based Cybersecurity & AI Business Solutions Firm, via MS D365 Project Operations and Power BI. →

Summary In modern data engineering environments, managing ingestion pipelines across multiple source systems becomes increasingly complex as data volume and variety grow. Hardcoded pipelines create maintenance overhead, slow down onboarding of new datasets, and introduce operational risks. This blog explains how a metadata-driven pipeline approach in Databricks can simplify ingestion by using a centralized configuration table to dynamically control pipeline behavior. It highlights how this pattern improves scalability, governance, and maintainability while enabling faster and more reliable data processing. The Real Problem: Hardcoded Pipelines Do Not Scale In many implementations, ingestion pipelines are built separately for each entity or source system. Typical issues include: As the number of entities grows, pipelines become difficult to manage and error-prone. What Is a Metadata-Driven Pipeline? A metadata-driven pipeline shifts control from code to configuration. Instead of writing separate logic for each dataset, we define ingestion behavior in a centralized configuration table. Typical metadata fields include: The pipeline reads this metadata and dynamically executes ingestion logic. Implementation Approach Step 1: Create a Configuration Table A centralized metadata table is created to define ingestion rules. Each row represents one dataset and contains all required configuration. Step 2: Dynamic Pipeline Execution The pipeline reads metadata and loops through each configuration entry. For each entity: No code changes are required when new entities are added. Step 3: Incremental Logic Control Instead of hardcoding: WHERE modifiedon > last_run The incremental field is read from metadata, allowing flexibility across different source systems. Step 4: Integration with Lakehouse Layers Metadata drives ingestion, while Lakehouse layers manage transformation. Why This Approach Works in Enterprise Environments 1. Scalability New entities can be added by inserting a new row in metadata. No pipeline duplication required. 2. Maintainability Changes in incremental logic or source structure are handled centrally. 3. Consistency All pipelines follow the same logic and standards. 4. Governance Metadata provides visibility into: Common Mistakes to Avoid Metadata-driven pipelines require discipline in design. Business Impact Metadata-driven pipelines are not just a technical optimization they are a foundational shift in how data platforms are built and managed. Organizations looking to scale their data engineering capabilities should move away from hardcoded ingestion logic and adopt configuration-driven approaches that support flexibility, governance, and long-term growth. Connect with CloudFronts to get started at transform@cloudfonts.com.

Summary Modern analytics platforms require structured data pipelines that ensure reliability, consistency, and governance across reporting systems. Traditional ETL approaches often struggle to scale as data volume and complexity increase. This blog explains how the Bronze–Silver–Gold (Medallion) architecture in Databricks provides a scalable and reliable framework for organizing data pipelines. It highlights how each layer serves a specific purpose, enabling better data quality, governance, and seamless integration with reporting tools such as Power BI. The Real Problem: Reporting Pipelines Become Fragile Over Time In many organizations: This leads to unreliable reporting and increased maintenance effort. What Is the Bronze–Silver–Gold Architecture? The Medallion architecture organizes data into three layers: Bronze Layer Raw data ingestion layer. Silver Layer Cleaned and standardized data. Gold Layer Business-ready, reporting-optimized data. Each layer has a clear responsibility. Bronze Layer: Raw Data Ingestion Purpose Key Characteristics Bronze acts as the system of record. Silver Layer: Data Standardization Purpose Key Activities Silver creates reusable datasets across reporting use cases. Gold Layer: Reporting-Ready Data Purpose Key Characteristics Gold tables are consumed directly by reporting tools. Why This Architecture Works 1. Separation of Concerns Each layer has a defined role, reducing complexity. 2. Improved Data Quality Data is progressively refined from raw to curated. 3. Better Performance Reporting queries run on optimized Gold tables. 4. Governance with Unity Catalog Access can be controlled at each layer: Common Implementation Mistakes These mistakes lead to long-term instability. Business Impact To conclude, the Bronze–Silver–Gold architecture provides a strong foundation for building scalable and reliable data pipelines in Databricks. When combined with proper governance and disciplined design, it enables organizations to deliver consistent, high-quality data for analytics and decision-making. We hope you found this article useful. If you would like to explore how AI-powered customer service can improve your support operations, please contact us at transform@cloudfronts.com.

As data platforms mature, teams often face a familiar question:Should we continue with classic ETL pipelines, or move to Delta Live Tables (DLT)? Both approaches work. Both are widely used. The real challenge is knowing which one fits your use case, not which one is newer or more popular. In this blog, I’ll break down Delta Live Tables vs classic ETL from a practical, project-driven perspective, focusing on how decisions are actually made in real data engineering work. Classic ETL in Databricks Classic ETL in Databricks refers to pipelines where engineers explicitly control each stage of data movement and transformation. The pipeline logic is written imperatively, meaning the engineer decides how data is read, processed, validated, and written. Architecturally, classic ETL pipelines usually follow the Medallion pattern: Each step is executed explicitly, often as independent jobs or notebooks. Dependency management, error handling, retries, and data quality checks are all implemented manually or through external orchestration tools. This approach gives teams maximum freedom. Complex ingestion logic, conditional transformations, API integrations, and custom performance tuning are easier to implement because nothing is abstracted away. However, this flexibility also means consistency and governance depend heavily on engineering discipline. We implemented a Classic ETL pipeline in our internal Unity Catalog project, migrating 30+ Power BI reports from Dataverse into Unity Catalog to enable AI/BI capabilities. This architecture allows data to be consumed in two ways – through an agentic AI interface for ad-hoc querying and through Power BI for governed, enterprise-grade visualizations. We chose the ETL approach because it provides strong data quality control, schema stability, and predictable performance at scale. It also allows us to apply centralized transformations, enforce governance standards, optimize storage formats, and ensure consistent semantic models across reporting and AI workloads -making it ideal for production-grade analytics and enterprise adoption. Delta Live Tables Delta Live Tables is a managed, declarative pipeline framework provided by Databricks. Instead of focusing on execution steps, DLT encourages engineers to define what tables should exist and what rules the data must satisfy. From an architectural perspective, DLT formalizes the Medallion pattern. Pipelines are defined as a graph of dependent tables rather than a sequence of jobs. Databricks automatically understands lineage, manages execution order, applies data quality rules, and provides built-in monitoring. DLT pipelines are particularly well-suited for transformation and curation layers, where data is shared across teams and downstream consumers expect consistent, validated datasets. The platform takes responsibility for orchestration, observability, and failure handling, reducing operational overhead. In my next blog, I will demonstrate how to implement Delta Live Tables (DLT) in a hands-on, technical way to help you clearly understand how it works in real-world scenarios. We will walk through the creation of pipelines, data ingestion, transformation logic, data quality expectations, and automated orchestration. The Core Architectural Difference The fundamental difference between classic ETL and Delta Live Tables is how responsibility is divided between the engineer and the platform. In classic ETL, the engineer owns the full lifecycle of the pipeline. This provides flexibility but increases maintenance cost and risk. In Delta Live Tables, responsibility is shared: the engineer defines structure and intent, while Databricks enforces execution, dependencies, and quality. This shift changes how pipelines are designed. Classic ETL is optimized for control and customization. Delta Live Tables is optimized for consistency, governance, and scalability. When Classic ETL Makes More Sense Classic ETL is a strong choice when pipelines require complex logic, conditional execution, or tight control over performance. It is well suited for ingestion layers, API-based data sources, and scenarios where transformations are highly customized or experimental. Teams with strong engineering maturity may also prefer classic ETL for its transparency and flexibility, especially when governance requirements are lighter. When Delta Live Tables Is the Better Fit Delta Live Tables excels when pipelines are repeatable, standardized, and shared across multiple consumers. It is particularly effective for silver and gold layers where data quality, lineage, and operational simplicity matter more than low-level control. DLT is a good architectural choice for enterprise analytics platforms, certified datasets, and environments where multiple teams rely on consistent data definitions. A Practical Architectural Pattern In real-world platforms, the most effective design is often hybrid. Classic ETL is used for ingestion and complex preprocessing, while Delta Live Tables is applied to transformation and curation layers. This approach preserves flexibility where it is needed and enforces governance where it adds the most value. To conclude, Delta Live Tables is not a replacement for classic ETL. It is an architectural evolution that addresses governance, data quality, and operational complexity. The right question is not which tool to use, but where to use each. Mature Databricks platforms succeed by combining both approaches thoughtfully, rather than forcing a single pattern everywhere. Choosing wisely here will save significant rework as your data platform grows. Need help deciding which approach fits your use case? Reach out to us at transform@cloudfronts.com

Working with Microsoft Dataverse Connector in Power BI is usually straightforward—until you encounter a table that simply refuses to load any rows, even though the data clearly exists in the environment. This happens especially with hidden, virtual, or system-driven tables (e.g. msdyn_businessclosure, msdyn_scheduleboardsetting) which are commonly used in Field Service and Scheduling scenarios. Before jumping to a workaround, it’s important to understand why certain Dataverse tables don’t load in Power BI, what causes this behavior, and why the standard Dataverse connector may legitimately return zero rows. Causes – 1] The Table Is a Virtual or System Table with Restricted AccessSystem-managed Dataverse tables like msdyn_businessclosure are not exposed to the Dataverse connector because they support internal scheduling and platform functions. 2] No Records Exist in the Root Business Unit Data owned by child business units is not visible to Power BI accounts associated with a different BU, resulting in zero rows returned. 3] The Table Is Not Included in the Standard Dataverse Connector Some solution-driven or non-standard tables are omitted from the Dataverse connector’s supported list, so Power BI cannot load them. Solution: Export Dataverse Data Using Power Automate + Excel Sync Since Power BI can read:-> OneDrive-hosted files-> Excel files-> SharePoint-hosted spreadsheets …a suitable workaround is to extract the restricted Dataverse table into Excel using a scheduled (When the records are few) / Dataverse triggered (When there are many records and you only want a single one, to avoid pagination) Power Automate flow. What it can do –-> Power Automate can access system-driven tables.-> Excel files in SharePoint can be refreshed by Power BI Service.-> we can bypass connector restrictions entirely.-> The method works even if entities have hidden metadata or internal platform logic. This ensures:-> Consistent refresh cycles-> Full visibility of all table rows-> No dependency on Dataverse connector limitations Use case I needed to use the Business Closures table (Dataverse entity: msdyn_businessclosure) for a few calculations and visuals in a Power BI report. However, when I imported it through the Dataverse Connector, the table consistently showed zero records, even though the data was clearly present inside Dynamics 365. There are 2 reasons possible for this –1] It is a System/Platform Tablemsdyn_businessclosure is a system-managed scheduling table, and system tables are often hidden from external connectors, causing Power BI to return no data. 2] The Table Is Not Included in “Standard Tables” Exposed to Power BIMany internal Field Service and scheduling entities are excluded from the Dataverse connector’s metadata, so Power BI cannot retrieve their rows even if they exist. So here, we would fetch the records via “Listing” in Power automate and write to an excel file to bypass the limitations that hinder the exposure of that data; without compromising on user privileges, or system roles; we can also control or filter the rows being referred directly at source before reaching PBI Report. Automation steps – 1] Select a suitable trigger to fetch the rows of that entity (Recurring or Dataverse, whichever is suitable). 2] List the rows from the entity (Sort/Filter/Select/Expand as necessary). 3] Perform a specific logic (e.g. clearing the existing rows, etc.) on the excel file where the data would be written to. 4] For each row in the Dataverse entity, select a primary key (e.g. the GUID), provide the path to the particular excel file (e.g. SharePoint -> Location -> Document Library -> File Name -> Sheet or Table in the Excel File), & assign the dynamic values of each row to the columns in the excel file. 5] Once this is done, import it to the PBI Report by using suitable Power Query Logic in the Advanced Editor as follows – -> a) Loading an Excel File from SharePoint Using Web.Contents() – Source = Excel.Workbook(Web.Contents(“https://<domain>.sharepoint.com/sites/<Location>/Business%20Closures/msdyn_businessclosures.xlsx”),null,true), What this step does: -> Uses Web.Contents() to access an Excel file stored in SharePoint Online.-> The URL points directly to the Excel file msdyn_businessclosures.xlsx inside the SharePoint site.-> Excel.Workbook() then reads the file and returns a structured object containing:All sheets, Tables, Named ranges Parameters used: null → No custom options (e.g., column detection rules)true → Indicates the file has headers (first row contains column names) -> b) Extracting a Table Named “Table1” from the Workbook – msdyn_businessclosures_Sheet = Source{[Item=”Table1″, Kind=”Table”]}[Data], This would search inside the Source object (which includes all workbook elements), and look specifically for an element where: Item = “Table1” → the name of the table in the Excel fileKind = “Table” → ensures it selects a table, not a sheet with the same name & would extract only the Data portion of that table. As a result, we get Power Query table containing the exact contents of Table1 inside the Excel workbook, to which we can further apply our logic filter, clean, etc. To conclude, when Dataverse tables refuse to load through the Power BI Dataverse Connector—especially system-driven entities like msdyn_businessclosure—the issue is usually rooted in platform-level restrictions, connector limitations, or hidden metadata. Instead of modifying these constraints, offloading the data through Power Automate → Excel → Power BI provides a controlled, reliable, and connector-independent integration path. By automating the extraction of Dataverse rows into an Excel file stored in SharePoint or OneDrive, you ensure: This method is simple to build, stable to maintain, and flexible enough to adapt to any Dataverse table -whether standard, custom, or system-managed. For scenarios where Power BI needs insights from hidden or restricted Dataverse tables, this approach remains one of the most practical and dependable solutions. I Hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com.

Why SmartPitch? – The Idea and Pain Point  The idea for SmartPitch came directly from observing the day-to-day struggles of sales and pre-sales teams. Every Marketing Qualified Lead (MQL) to Sales Qualified Lead (SQL) conversion required hours of manual work: searching through documents stored in SharePoint, combing through case studies, aligning them with solution areas, and finally packaging them into a client-ready pitch deck.  The reality was that documents across systems—SharePoint, Dynamics 365, PDFs, PPTs—remained underutilized because there was no intelligent way to bring them together.   Sales teams often relied on tribal knowledge or reused existing decks with limited personalization.  We asked: What if a sales assistant could automatically pull the right case studies, map them to solution areas, and draft an elevator pitch on demand, in minutes?  That became the SmartPitch vision: an AI-powered agent that:  As a result of this product, it has helped us reduce pitch creation time by 70%.   2. The First Prototype – Custom Copilot Studio  Our first step was to build SmartPitch using Custom Copilot Studio. It gave us a low-code way to experiment with conversational flows, integrate with Azure AI Search, and provide sales teams with a chat interface.  1. Knowledge Sources Integration  2. Data Flow  3. Conversational Flow Design  4. Integration and Security  5. Technical Stack  6. Business Process Enablement  7. Early Prototypes  With Custom Copilot, we were able to:  We successfully demoed these early prototypes in Zurich and New York. They showed that the idea worked but they also revealed serious limitations.  3. Challenges in Custom Copilot  Despite proving the concept, Custom Copilot Studio had critical shortcomings:  Lacked support for model fine-tuning or advanced RAG customization.  However, incorporating complex external APIs or custom workflows was difficult.  This limitation meant SmartPitch, in its Copilot form, couldn’t scale to meet enterprise standards.  4. Rebuilding in Azure AI Foundry – Smarter, Extensible, Connected  The next phase was Azure AI Foundry, Microsoft’s enterprise AI development platform. Unlike Copilot Studio, AI Foundry gave us:  Extending SmartPitch with Logic Apps  One of the biggest upgrades was the ability to integrate Azure Logic Apps as external tools for the agent. This allowed SmartPitch to:  This modular approach meant we could add new functionality simply by publishing a new Logic App. No redeployment of SmartPitch was required.  Automating Document Vectorization  We also solved one of the biggest bottlenecks—document ingestion and retrieval—by building a pipeline for automatic document vectorization from SharePoint:  This allowed SmartPitch to search across text, images, tables, and PDFs, providing relevant answers instead of keyword matches.  But There Were Limitations  Even with these improvements, we hit roadblocks:  At this point, we realized the true bottleneck wasn’t the agent itself, it was the quality of the data powering it.  5. Bad Data, Governance, and the Medallion Architecture  SmartPitch’s performance was only as good as the data it retrieved from. And much of the enterprise data was dirty: duplicate case studies, outdated documents, inconsistent file formats.  This led to irrelevant or misleading responses in pitches.  To address this, we turned to Databricks’ Unity Catalog and Medallion Architecture:  You can read our post on building a clean data foundation with Medallion Architecture [Link]   Now, every result SmartPitch surfaced could be trusted, audited, and tied to a governed source.  6. SmartPitch in Mosaic AI – The Final Evolution  The last stage was migrating SmartPitch into Databricks Mosaic AI, part of the Lakehouse AI platform. This was where SmartPitch matured into an enterprise-grade solution.  What We Gained in Mosaic AI:  In Mosaic AI, SmartPitch wasn’t just a chatbot it became a data-native enterprise sales assistant:  From these, we came to know the following differences between agent development in AI Foundry & DataBricks Mosaic AI –    Attribute / Aspect  Azure AI Foundry  Mosaic AI  Focus  Developer and Data Scientist  Data Engineers, Analysts, and Data Scientists  Core Use Case  Create and manage your own AI agent  Build, experiment, and deploy data-driven AI models with analytics + AI workflows  Interface  Code-first (SDKs, REST APIs, Notebooks)  No-code/low-code UI + Notebooks + APIs  Data Access  Azure Blob, Data Lake, vector DBs  Native integration with Databricks Lakehouse, Delta Lake, Unity Catalog, vector DBs  MCP Server  Only custom MCP servers supported; built-in option complex  Native MCP support with Databricks ecosystem; simpler setup  Models  90 models available  Access to open-source + foundation models (MPT, Llama, Mixtral, etc.) + partner models  Model Customization  Full model fine-tuning, prompt engineering, RAG  Fine-tuning, instruction tuning, RAG, model orchestration  Publish to Channels  Complex (Azure Bot SDK + Bot Framework + App Service)  Direct integration with Databricks workflows, APIs, dashboards, and third-party apps  Agent Update  Real-time updates in Microsoft Teams  Updates deployed via Databricks workflows; versioning and rollback supported  Key Capabilities  Prompt flow orchestration, RAG, model choice, vector search, CICD pipelines, Azure ML & responsible AI integration  Data + AI unification (native to Lakehouse), RAG with Lakehouse data, multi-model orchestration, fine-tuning, end-to-end ML pipelines, secure governance via Unity Catalog, real-time deployment  Key Components  Workspace & agent orchestration, 90+ models, OpenAI pay-as-you-go or self-hosted, security via Azure identity  Mosaic AI Agent Framework, Model Serving, Fine-Tuning, Vector Search, RAG Studio, Evaluation & Monitoring, Unity Catalog Integration  Cost / License  Vector DB: external, Model Serving: token-based pricing (GPT-3.5, GPT-4), Fine-tuning: case-by-case, Total agent cost variable (~$5k–$7k+/month)  Vector Search: $605–$760/month for 5M vectors, Model Serving: $90–$120 per million tokens, Fine-Tuning Llama 3.3: $146–$7,150, Managed Compute built into DBU usage, End-to-end AI Agent ~$5k–$7k+/month  Use Cases / Capabilities  Agents intelligent, can interact/modify responses; single AI search per agent; infrastructure setup required; custom MCP server registration  Agents intelligent, interact/modify responses; AI search via APIs (Google/Bing); in-built MCP server; complex infrastructure; slower responses as results batch sent together  Development Approach  Low-code, faster agent creation, SDK-based, easier experimentation  Manual coding using MLflow library, more customization, API integration, higher chance of errors, slower build  Models Comparison  90 models, Azure OpenAI (GPT-3.5, GPT-4), multi-modal  ~10 base models, OSS & partner models (Llama, Claude, Gemma), many models don’t support tool usage  Knowledge Source  One knowledge source of each type (adding new replaces previous)  No limitation; supports data cleaning via Medallion Architecture; SQL-only access inside agent; Spark/PySQL not supported in agent  Memory / Context Window  8K–128K tokens (up to 1M for GPT-4.1)  Moderate, not specified  Modalities  Text, code, vision, audio (some models)  Likely text-only  Special Enhancements  Turbo efficiency, reasoning, tool calling, multimodal  Varies per model (Llama, Claude, Gemma architectures)  Availability  Deployed via Azure AI Foundry  Through Databricks platform  Limitations  Only one knowledge source of each type, infrastructure complexity for MCP server  No multi-modal Spark/PySQL access, slower batch responses, limited model count, high manual development  7. Lessons Learned:  … Continue reading Inside SmartPitch: How CloudFronts Built an Enterprise-Grade AI Sales Agent Using Microsoft and Databricks Technologies  →

Everyone wants AI. Few are ready for it.  The question isn’t “When do we start?” but “Are we prepared to get it right?”  Because switching on Copilots without fixing your foundations doesn’t accelerate you. it amplifies chaos.  This article will cover how to fix your foundations for AI so that the AI tools you deploy are accurate and reliable.  Challenges of deploying AI Directly  Some of the common challenges of directly deploying AI on top of your business applications are –   And these issues just render the AI implementation as a failure immediately dismissing trust in using AI at all.  But these challenges can be overcome once the foundations of AI are in place which we’ll discuss in the next section.  Foundation of AI  At CloudFronts, we call this the 3 Pillars of AI Readiness:  Here’s how I sum up the foundation of the systems for AI –  For example, when CloudFronts helped Tinius Olsen modernize their systems, the focus wasn’t just technical uplift. It was about ensuring every business process was cloud-ready so AI models could actually trust the data.   Upgrading from legacy systems And this is the foundation that needs to be had before AI can be implemented at your organization.  Data & AI Maturity Curve by Databricks  Given the above foundations in place for your AI Adoption strategy and choosing the right framework for your implementation, the Data & AI Maturity Curve shown below can be referenced to see where your organization is on the curve and where do you want to get to –   On a high level, the foundation will get you to look back at the data and see what has happened in the past and AI tools can help you get this information accurately.  Further, once trust is established, actions like making the AI predict the future state of operations, prescribe steps and even take decisions on our behalf can be achieved – provided you really want that to happen. It might be too soon just yet.  To conclude, AI success = Foundations × Trust.  Without modern systems, connected data, and governed access, AI is just noise.  But with these in place, every AI tool you deploy whether predictive analytics or Copilots becomes an accelerator for decision-making, not a distraction.  Before you deploy AI, fix your foundations. If you’re serious about making AI a trusted accelerator not a costly experiment start with modernization, connection, and governance. At CloudFronts, we help enterprises build these foundations with confidence. Let’s connect over our email: Transfrom@cloudfronts.com  

In Part 1, we built a simple MCP server in TypeScript that exposed a “getWeather” tool. Now, let’s take the next step: connecting our MCP server to Microsoft Copilot Studio so that Copilot agents can call it directly. This section will cover: Step 1 — Publish Your MCP Server to Azure To make your MCP server accessible to Copilot Studio, you’ll need to host it online. There are multiple ways to deploy it — Azure App Service, Azure Container Apps, or even Azure Functions if you prefer serverless. For example, using Azure App Service: Test using curl to ensure it responds with MCP-compatible JSON: Step 2 — Create a New Copilot in Copilot Studio Step 3 — Add Knowledge Sources Optionally, you can enrich your Copilot by adding: This gives your Copilot a baseline knowledge to answer broader questions, while the MCP server will handle specific tasks (like fetching live weather data). Step 4 — Create a Custom Connector in Dataverse To let Copilot Studio talk to our MCP server, we need a custom connector inside Dataverse/CRM. Step 5 — Add the Custom Connector to Copilot Studio you’ll see the MCP server in your Tools section of copilot. To test the setup, let’s ask Copilot: “What’s the current weather in Mumbai?” On the first attempt, Copilot will prompt you to establish a connection. Simply open the Connection Manager, click Connect, and authorize the link to your MCP server. Once connected, Copilot will fetch the live weather details for Mumbai directly from your MCP server. and click retry on the Test window of your copilot. And just like that, your MCP server is live and fully integrated. It can now provide real-time weather updates for any city mentioned in your conversation with Copilot. You can try out different variations of questions or phrasings — Copilot will intelligently interpret your request, extract the city name, and seamlessly call the MCP server to deliver accurate weather details. Beyond Weather: Business Integrations The same process works for enterprise systems. For example, instead of getWeather, you could expose: By publishing these tools via MCP, your Copilot becomes a true enterprise assistant, capable of pulling structured business data and triggering workflows on demand. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.