Sustainability certification is one of the most operationally demanding programs a nonprofit can run. It is not just a badge on a product, it is a multi-year, multi-stakeholder process involving manufacturers, independent assessment bodies, scoring frameworks, document issuance, and public transparency requirements. When you are managing thousands of products across global industries, the cracks in a manual, spreadsheet-driven operation show up fast. This is exactly the situation a Netherlands-based nonprofit found itself in. The organization administers a globally recognized product sustainability certification program, assessing products across five dimensions: material health, product circularity, clean air and climate protection, water and soil stewardship, and social fairness. Products move through certification levels Bronze, Silver, Gold, and Platinum across a lifecycle that spans application, third-party assessment, issuance, and periodic recertification every three years. As certification volumes grew, so did the operational complexity. Disconnected tools, manual document preparation, and no single place to track everything meant the team was spending more time managing the process than running it. Rather than bolt on yet another external tool, the organization made a deliberate architectural choice: build the entire certification management platform inside Microsoft Dynamics 365, extend it with Azure Function Apps for automation, and expose public APIs for ecosystem transparency. The Goal Build a unified, scalable certification lifecycle management system inside Dynamics 365 that automates document generation, manages logo assets, and exposes public APIs for published certification data — all without introducing new platform dependencies. The Business Problem To understand what was built, you first need to understand what was broken. The organization’s operational teams were trying to answer some fairly fundamental questions every single day — What is the current certification status of a given product? Which products are approaching their recertification deadline? Which assessment body certified a product and when? Is the certificate document ready for issuance? None of these questions had a reliable, centralized answer. Certification records lived across disconnected spreadsheets and email threads, which meant any “current” view of a product’s status was only as accurate as the last person who updated a row. Certificate documents were manually composed for every issuance a slow, error-prone process that created formatting inconsistencies and delayed the experience for certified manufacturers. Logo assets were managed informally, with no version control or consistent delivery process. No Single Source of Truth Certification records scattered across spreadsheets and email threads with no reliable current view. Manual Document Creation Every certificate composed by hand slow, inconsistent, and a bottleneck manufacturers felt directly. Zero Public Transparency External stakeholders relied on manually updated static pages with no programmatic access to live data. Unscalable Operations Growing program volumes with no automation meant every new product added to the manual workload. The Solution Architecture The platform was designed around one principle: build close to where the operational data already lives, and automate at the right trigger points rather than everywhere at once. The solution runs on three deliberate layers. Critically, this architecture avoided over-engineering entirely — no separate data warehouse, no heavy ETL pipeline, no dedicated certification SaaS platform requiring its own licensing and maintenance. Everything runs inside the Microsoft ecosystem. 1 Data Layer — Custom Dynamics 365 Tables Purpose-built Dataverse tables that mirror the certification domain exactly, products, certification events, assessment bodies, category scores, and logo assets all in a single relational, auditable structure. 2 Automation Layer — Azure Function Apps, Dynamics Plugins Two event-driven Function Apps sit alongside the CRM one for certificate document generation, one for logo package delivery, both triggered by real state changes in the certification lifecycle, not a schedule. 3 Transparency Layer — Public REST APIs Public-facing APIs expose published certification data to external stakeholders, brands, retailers, regulators, and third-party platforms without any manual data exchange with the organization. Custom Dynamics 365 Data Model The data model is the foundation everything else rests on. Rather than forcing certification concepts into standard CRM entities that were never designed for this domain, the team built purpose-specific custom tables inside Dataverse that mirror how the certification program actually works. Product data Core product records, variants, and identifiers — the foundational layer that everything else references. Application handling Applications, assessments, category and requirements assessments — all managed within accounts. Assessment bodies and related workflows live here too. Public-facing entities Public tables for products, certifications, certificates, and product variants — the data layer that powers external visibility and API exposure. Together, these layers gave the organisation a complete, relational view of every certified product across its full lifecycle — all within a single operational platform. Certificate Document Generation via Azure Function App Before this system existed, every certificate document was created by hand. Someone would take a template, fill in the product details, format it, check it, and send it. For an organization issuing certificates across thousands of products, this was not just slow — it was a source of constant inconsistency and a bottleneck that manufacturers felt directly. The Azure Function App for certificate generation eliminated this entirely. Here is how it works end to end: How It Works ⚡ Trigger Certification record reaches the correct status in Dynamics 365 → 🔍 Fetch Pulls record + product data via Dynamics 365 Web API → 📄 Generate Selects correct template, populates all fields, generates document → 🔗 Store & Link Saves document and links it back to the certification record What this means in practice is that certificate issuance is now consistent, fast, and entirely hands-off for the operational team. Formatting is guaranteed every time because the template logic is defined once and applied uniformly. The function also runs independently of the CRM interface — making it resilient and reusable across multiple trigger scenarios, including bulk recertification processing. The impact: A task that previously required manual effort for every single issuance now requires none. Eliminated entirely. Logo Image Generation via Azure Function App A certified product comes with more than a document — it comes with the right to use the certification mark. For manufacturers, that logo is a commercial asset. It goes … Continue reading How a Netherlands-Based Non-Profit Transformed Certification Management with Dynamics 365 and Azure Functions →

Summary We implemented an automated cost control process for our internal AI Sales Insights solution running in Azure. Azure Budget Alerts were connected with Automation Runbooks to automatically stop resources once spending crossed a defined threshold. The setup reduced manual monitoring and helped us maintain tighter control over Azure consumption. The solution used native Azure services with Managed Identity authentication for secure automation. The same approach can be extended to Azure Virtual Machines, App Services, Databases, and other Azure resources. Azure Budget Alerts integrated with Automation Runbooks for automated cost control Table of Contents 1. Why We Needed This 2. Solution Overview 3. Azure Services Used 4. Step 1: Create the Azure Budget 5. Step 2: Create an Action Group 6. Step 3: Create the Automation Runbook 7. Step 4: Enable Managed Identity 8. Step 5: Assign Required Permissions 9. Step 6: Test the Complete Flow 10. Business Impact 11. Frequently Asked Questions 12. Conclusion Why We Needed This Our AI Sales Insights solution uses Azure services such as Function Apps and AI processing components to generate and process sales insights. Since the environment runs on a Pay As You Go subscription, costs can increase unexpectedly if resources continue running after active usage hours or during extended testing cycles. Initially, we were monitoring costs manually through Azure Cost Management dashboards and email alerts. However, this approach had a few limitations: Alerts were sometimes noticed too late Resources continued running even after thresholds were reached Development environments stayed active longer than needed Manual intervention was required every time We wanted a simple solution where Azure could automatically take action when spending crossed a defined limit. Solution Overview We implemented the following workflow: Azure Budget continuously monitors subscription spending A Budget Alert gets triggered when the threshold is reached The alert calls an Azure Action Group The Action Group triggers an Automation Runbook The Runbook authenticates using Managed Identity The Runbook automatically stops the Azure resource This gave us a lightweight and reliable automation setup for controlling costs. Azure Services Used Service Purpose Azure Cost Management Monitor Azure spending Azure Budget Alerts Detect threshold breaches Azure Action Groups Trigger automation Azure Automation Execute PowerShell Runbooks Managed Identity Secure authentication Step 1: Create the Azure Budget We started by creating a budget inside Azure Cost Management. Azure Budget configuration for automated cost monitoring Budget Configuration Setting Value Budget Amount ₹800 INR Reset Frequency Monthly Forecast Alert 90% Actual Cost Alert 100% One important thing we learned during implementation is that Azure budget calculations are not always real time. If automation starts only at 100%, additional usage may already occur before the alert executes. Because of this, setting automation thresholds slightly earlier, usually around 85% to 90%, is safer for production environments. Step 2: Create an Action Group Next, we created an Azure Action Group. The Action Group acts as the bridge between the Budget Alert and the Automation Runbook. Set Action Type as Automation Runbook Select the Automation Account Link the target Runbook Once configured, Azure automatically executes the Runbook whenever the budget alert is triggered. Step 3: Create the Automation Runbook Inside Azure Automation, we created a PowerShell Runbook using Runtime Version 5.1. The Runbook dynamically accepts Azure resource values instead of hardcoded names, making it reusable across multiple environments. Dynamic PowerShell Script # Variables $ResourceGroupName = “{Your Azure Resource Group Name}” $FunctionAppName = “{Your Azure Function App Name}” # Authenticate using System Managed Identity try { Connect-AzAccount -Identity Write-Output “Successfully authenticated with Managed Identity” } catch { Write-Error “Failed to authenticate: $_” exit 1 } # Stop Azure Function App try { Stop-AzFunctionApp ` -Name $FunctionAppName ` -ResourceGroupName $ResourceGroupName ` -Force Write-Output “Function App stopped successfully” } catch { Write-Error “Failed to stop Function App: $_” exit 1 } Step 4: Enable Managed Identity To avoid storing credentials inside the script, we enabled System Assigned Managed Identity for the Automation Account. This allowed the Runbook to authenticate securely against Azure without usernames, passwords, or secrets. This approach is cleaner, more secure, and easier to maintain. Step 5: Assign Required Permissions After enabling Managed Identity, we assigned RBAC permissions to the Automation Account on the target resource. The following roles were used: Contributor Website Contributor Function App Contributor Without proper permissions, the Runbook may authenticate successfully but still fail while stopping resources. Step 6: Test the Complete Flow Before connecting everything to the Budget Alert, we tested the Runbook manually. Verify Managed Identity authentication Confirm the Azure resource stops correctly Validate RBAC permissions Test Action Group execution Once everything worked successfully, the Runbook was connected to the Budget Alert. Business Impact Area Before After Cost Monitoring Manual Automated Resource Shutdown Manual Automatic Response Handling Manual Action Required Automatic Resource Shutdown Cost Governance Reactive Proactive This setup helped us reduce unnecessary runtime costs and eliminated the need for continuous manual monitoring of our AI Sales Insights environment. Conclusion This implementation helped us build a simple and practical cost governance process for our internal AI Sales Insights solution. Instead of relying only on dashboards and email notifications, Azure now takes automated action whenever spending crosses the configured threshold. As AI workloads continue growing, automating cloud cost control will become increasingly important for maintaining predictable and manageable Azure spending. We hope you found this blog useful. If you would like to discuss similar Azure automation and cost optimization solutions, feel free to connect with us at transform@cloudfronts.com .

What happens when a global manufacturing giant needs to modernize its operations without grinding critical business processes to a halt? The answer is not a rip-and-replace approach – it is a carefully engineered integration strategy that lets modern and legacy systems co-exist, communicate, and complement each other. Are you planning an ERP upgrade but worried about what happens to the legacy systems your operations depend on? If so, this is for you. One of North America’s leading commercial vehicle manufacturers faced exactly this dilemma. With decades of investment in legacy financial and warehouse management systems, a hard cutover to a new ERP was not an option. Yet the need for a modern, scalable platform was undeniable. Their solution? Introduce Microsoft Dynamics 365 Finance & Operations (D365 F&O) as the new operational backbone – while keeping their legacy systems in play for financial control – and build robust, bi-directional integrations to bridge both worlds. At CloudFronts, we had the privilege of architecting and implementing those integrations. This blog walks you through three core data flows: Spot Purchase Orders, Advance Shipment Notices (ASN), and Goods Receipt Notes (GRN) – and what it really takes to make a modern ERP talk to a legacy system without missing a beat. Why Replace When You Can Integrate? Legacy systems in large manufacturers are not just old software. They carry years of financial logic, vendor relationships, and compliance configurations that are too risky to discard overnight. Replacing them introduces enormous operational and compliance risk. Doing nothing, however, is not an option either. The approach our client took – and one we increasingly recommend for manufacturers, distributors, and large enterprises – is a co-existence model: This means the business gets the agility of a modern ERP on day one, without putting financial operations at risk. The three integrations do the heavy lifting. Architecture at a Glance Before diving into each integration, it helps to understand the overall data flow pattern and the Azure services involved: Component Role D365 F&O System of record for purchasing and receiving operations Legacy System Retains financial control, inventory management authority Azure Logic Apps Parent-child middleware: orchestrates, transforms, and routes data Azure Blob Storage Checkpoint management for reliable incremental processing Azure Table Storage Full execution logs for traceability, audit, and failure replay The three integrations work in concert: Integration 1: Spot Purchase Orders — D365 F&O to Legacy Business Problem A Spot Purchase Order is an ad-hoc purchase order raised outside of long-term contracts — often for urgent material procurement. Spot POs are created and managed in D365 F&O by procurement teams. However, the legacy system is the system of financial record, meaning every Spot PO created in D365 must be reflected in the legacy system for financial commitment tracking and vendor payment processing. Without integration, this would require manual re-entry – a process prone to error, delay, and duplication. How the Integration Works Parent Logic App – Spot PO Orchestrator The primary Logic App runs on a scheduled recurrence and uses a checkpoint mechanism stored in Azure Blob Storage to fetch only incremental changes – purchase orders created or modified since the last successful run. This ensures efficiency and prevents reprocessing of already-handled records. The workflow determines the operation type required for each PO: For each scenario, the Logic App fetches enriched data from multiple D365 F&O OData entities and constructs a structured JSON payload tailored for the legacy system’s API. ⚙ Tech Note: OData Entities Used PurchaseOrderHeaders, PurchaseOrderLinesV2, PurchaseLineDataEntities, WHSPurchLines, StatusCustomDatas Child Logic App – SendRequest (Reusable) Rather than embedding API communication logic directly in the orchestrator, we separated it into a reusable child Logic App. This child app receives the constructed payload, retrieves an OAuth 2.0 Bearer token, and executes the HTTP POST call to the legacy system’s API endpoint. This modular design pays dividends during maintenance: any change to authentication logic or API communication is made once in the child app and automatically applies to all parent integrations. Failed Record Handler Every enterprise integration needs robust failure recovery. When an API call fails: Sample Payload – Spot PO Create Sample JSON Payload: {   “userId”: “JSMITH”,   “order”: “456789”,   // Last 6 digits of D365 PO number   “vendor”: “VEND001”,   “receiptLoc”: “SITE01”,   “vendorOvrdCd”: “14”,   “lineItems”: [{     “orderLine”: “001”,     “item”: “ITEM001”,     “openQty”: 10,     “deliveryDate”: “061526”,  // MMddyy format for legacy compatibility     “comment”: “MPSSYS order – JSMITH” }] } } ✓ Business Impact: Zero manual re-entry of purchase orders between systems. Every Spot PO created or changed in D365 F&O is automatically reflected in the legacy system within minutes. Integration 2: Advance Shipment Notices — Legacy to D365 F&O Business Problem An Advance Shipment Notice (ASN) is a notification sent by the legacy WMS to the receiving system, informing it of an incoming shipment before it physically arrives. D365 F&O needs to receive ASNs to create Inbound Load Headers and Load Lines – enabling warehouse teams to prepare for receiving. Without this integration, receiving teams in D365 would be blind to incoming shipments until trucks arrived at the dock – eliminating any opportunity for advance dock scheduling, labor planning, or inventory pre-positioning. The Hybrid Integration Approach This integration presented an interesting technical challenge: the standard D365 F&O Inbound ASN V5 API supports a well-defined XML format, but the business required additional fields beyond what the standard API supports. The solution was a two-step Hybrid ASN Integration approach: ⚙ Tech Note: API Endpoint Pattern Insert:  POST {{BASE_URL}}/api/connector/enqueue/{{ACTIVITY_ID}}?entity=Inbound ASN V5 Enrich:  PATCH on InboundLoadHeaders and WHSASNWorkData Smart Insert vs. Update Determination To handle scenarios where an ASN might be re-sent for corrections or resynchronization, the integration includes a check before processing: This idempotent design prevents duplicate inbound loads from being created when the legacy system re-sends an ASN. One nuance worth noting: in D365’s standard ASN structure, the LoadId, ShipmentId, and LicensePlateNumber must carry the same value. The legacy system’s outbound ASN payload is configured to honour this requirement – ensuring clean data entry … Continue reading How a Top North American Commercial Vehicle Manufacturer Connected D365 F&O with Legacy Systems Without Disrupting Operations →

Summary A Houston-based cybersecurity firm eliminated report failures (~65%) by replacing SSRS with an Azure Function App pipeline. Dynamics 365 bound action ensured authentication stayed internal, bypassing Defender-related token failures. Integrated Power Pages, Power Automate, Dynamics 365, and Azure Functions for real-time PDF generation. Report generation time reduced from 3–8 minutes to under 15 seconds with zero infrastructure overhead. Table of Contents 1. About the Customer 2. The Challenge 3. The Solution 4. Technical Implementation 5. Business Impact 6. FAQs 7. Conclusion 1. About the Customer The client is a technology consulting and cybersecurity services firm based in Houston, Texas. They manage multiple concurrent client engagements using Dynamics 365 Project Operations as their core platform. Project managers and clients access live project data through a customer-facing portal built on Microsoft Power Pages. 2. The Challenge The organization needed one-click downloadable Project Status Reports from their Power Pages portal covering risks, issues, logs, and timelines. Their SSRS-based solution failed frequently due to authentication breakdowns caused by Microsoft Defender for Cloud Apps across multiple service boundaries. Key pain points: Silent authentication failures with no clear errors Retry delays of 60–90 seconds per attempt Separate SSRS infrastructure dependency Slow report customization cycle Project managers avoided generating reports during live meetings due to reliability concerns. 3. The Solution At Cloudfronts, while working on this project, I replaced the SSRS pipeline entirely with a synchronous, serverless architecture that keeps the authentication context inside the Dynamics 365 service layer. Technologies Used: Dynamics 365 Project Operations Power Pages Power Automate Plugins Azure Function Apps The solution generates fully formatted PDFs in real time using structured JSON payloads. This eliminated authentication failures while significantly improving speed and reliability. 4. Technical Implementation 1] Power Pages Button triggers Flow A “Download Report” button captures the project GUID and triggers a Power Automate flow with real-time progress feedback. 2] Dynamics 365 Plugin prepares JSON payload A bound action plugin retrieves all project data and converts it into a clean JSON payload for PDF generation. 3] Azure Function generates PDF The Azure Function processes the JSON and generates a formatted PDF, returning it as a Base64 string. 4] SharePoint Integration The generated PDF is automatically stored in the associated SharePoint document location linked to the project. This ensures centralized document management, version control, and easy access for stakeholders directly within the project workspace. 5] Portal PDF Preview The Base64 PDF is rendered directly in the portal using an iframe, allowing instant preview and download. Video: End-to-end implementation of real-time PDF report generation. 5. Business Impact 100% success rate — zero failures post deployment Under 15 seconds report generation time No infrastructure — fully serverless Zero authentication failures Faster iteration for report updates Project managers can now confidently generate reports during live client meetings. 6. FAQs Why not fix the SSRS authentication issue instead of replacing SSRS entirely? The authentication failures were a structural consequence of traversing multiple service boundaries in an environment with strict Defender for Cloud Apps session policies. Fixing them would have required either relaxing those policies — which the client’s security posture did not permit — or re-architecting the data retrieval to stay inside the platform, which is exactly what the bound action approach achieves. Replacing SSRS also removed a separate infrastructure dependency and gave the client full control over report formatting in code. Can this pattern be reused for other document types in Dynamics 365? Yes. The Azure Function App’s renderer is data-driven — it consumes a JSON payload and builds tables from whatever keys are present. The Dynamics 365 plugin can be adapted to query any entity and produce an equivalent payload. CloudFronts has applied the same pattern to inspection records, summary reports, and client-facing status documents across Professional Services and Manufacturing implementations. Does this work for environments without Microsoft Defender for Cloud Apps? Yes. The architectural benefits — synchronous generation, serverless PDF rendering, no SSRS infrastructure, and in-browser preview — apply regardless of the security layer on the environment. 7. Conclusion Replacing SSRS with an Azure Function App-based PDF renderer resolved both the reliability and authentication problems in a single architectural shift, delivering instant, professional-quality Project Status Reports from a Microsoft Power Pages portal with no legacy reporting infrastructure to maintain. The key lesson from this project is that keeping authentication within the Dynamics 365 service layer — rather than bridging to external systems — eliminates an entire category of environment-specific failures that are otherwise very difficult to diagnose and fix. By keeping authentication within Dynamics 365 and leveraging serverless architecture, the solution delivers instant, high-quality reports without infrastructure overhead. This approach demonstrates how modern cloud-native patterns can eliminate entire classes of system failures while improving user experience dramatically. Ready to modernise document generation in your Dynamics 365 environment?CloudFronts builds scalable Power Platform and Dynamics 365 solutions that replace legacy reporting infrastructure and automate document workflows. Reach out at transform@cloudfronts.com. Shashank Keny Associate Consultant · CloudFronts Shashank Keny is an Associate Consultant at CloudFronts with 1.5+ years of experience in cloud, data, and business applications. He specializes in building scalable, API-driven architectures and integrating enterprise systems across the Microsoft ecosystem. He is a Certified Databricks Data Engineer with hands-on experience in Dynamics 365 Project Operations and Dynamics 365 Sales, along with delivering business intelligence solutions using Power BI. His expertise also extends to modern AI solutions, including building custom copilots and implementing intelligent applications using Azure AI Foundry. Passionate about solving real-world business challenges through data and AI, he focuses on delivering efficient, scalable, and production-ready solutions. Experience: 1.5+ years Certification: Databricks Certified Data Engineer Specialization: Dynamics 365 Project Operations, Power BI, Azure Integrations, AI Solutions View LinkedIn Profile

The Challenge of Scaling AI in Enterprises Many organizations invest in AI initiatives but struggle to scale beyond pilot projects. Custom-built solutions are expensive, difficult to govern, and often limited to a single use case. As a result, AI investments fail to deliver sustained business value. Why Automation Alone Is Not Enough Traditional automation relies on rigid rules and predefined workflows. While effective for simple tasks, it cannot adapt to changing business conditions. Enterprises need intelligent systems that can reason, decide, and act autonomously. Understanding AI Agents in Simple Terms AI agents are intelligent software systems that understand goals, plan actions, and execute multi-step workflows with minimal human intervention. Unlike chatbots, AI agents do not just answer questions they act on insights. What Agent Bricks Bring to the Business Agent Bricks are modular, reusable AI agent components that accelerate enterprise AI adoption. They enable organizations to deploy intelligent agents quickly while maintaining security, governance, and compliance. Ask Me Anything: Execution Powered by Agent Bricks In the Ask Me Anything solution, Agent Bricks power the execution layer. They continuously evaluate enterprise data, identify project readiness gaps, and respond to leadership queries in real time. Agent Bricks Workflow Execution (Testing Screenshot) Use Case Spotlight: PMO Assistant at Scale The PMO Assistant built using Agent Bricks operates continuously, monitoring upcoming projects and flagging risks early. This reduces dependency on manual reporting and enables PMOs to focus on proactive delivery management. Business Value of an AI Workforce From a business perspective, Agent Bricks enable faster AI deployment, lower operational costs, and consistent decision-making across departments. Enterprises can scale AI solutions confidently without rebuilding logic for every new use case. Moving from Experiments to Execution To conclude, Agent Bricks help organizations move from isolated AI experiments to production-ready AI solutions. CloudFronts partners with enterprises to build scalable, governed AI workforces that deliver measurable business outcomes. I hope you found this blog useful, and if you would like to discuss anything or explore a future implementation, you can reach out to us at transform@cloudfonts.com.

The Growing Challenge of Project Readiness As organizations grow, managing project readiness becomes increasingly complex. Data related to projects, resources, and timelines is spread across CRM systems, project management tools, and booking platforms. Team Leads, CTOs, and CEOs often struggle to gain a real-time, consolidated view of whether projects are truly ready to start. This lack of visibility leads to delayed project kick-offs, inefficient resource utilization, and increased operational risk. Why Traditional Systems Fail at Scale Traditional reporting and AI systems are not designed to handle the dynamic nature of growing enterprises. They respond only to explicit prompts, operate in single-step workflows, and require significant human intervention. Leadership teams depend heavily on manual checks and follow-ups, which consume time and still fail to provide timely insights. The Shift Toward Agentic AI Organizations are now shifting from static AI responses to autonomous AI-driven decision-making. Agentic AI enables systems to understand intent, evaluate multiple data points, and decide what action to take next. This shift is critical for enterprises that want to move from reactive reporting to proactive management. What Genie Space Means for Business Leaders Genie Space is an AI-powered natural language analytics layer that allows business users to ask questions in plain English and receive immediate, governed answers. Without requiring SQL knowledge or technical expertise, Genie Space empowers leaders to access insights directly while maintaining full enterprise security and compliance through Unity Catalog. Ask Me Anything: A Unified Intelligence Layer The Ask Me Anything solution leverages Genie Space as the central intelligence layer. It connects securely to enterprise systems, preserves conversational context, and delivers consistent insights across departments. This unified approach ensures that leadership teams rely on a single source of truth for decision-making. Ask Me Anything Product Architecture Diagram Use Case Spotlight: PMO Assistant for Project Readiness In a typical PMO environment, project managers lack real-time visibility into execution readiness. Tasks may not be configured, resources may not be aligned, and risks often surface too late. The PMO Assistant powered by Genie continuously monitors projects scheduled to start within a defined window and provides instant readiness insights. Business Impact of Genie-Powered Insights By implementing Genie Space, organizations significantly reduce manual reporting effort, improve delivery confidence, and enable leadership teams to focus on strategic priorities. Faster insights lead to quicker decisions, lower operational costs, and improved customer satisfaction. To conclude, Genie Space transforms how organizations interact with their data. Instead of searching for information, leaders receive instant, trusted answers. CloudFronts helps enterprises design and deploy Genie-powered solutions that improve project visibility and decision-making across the organization. I hope you found this blog useful, and if you would like to discuss anything or explore a future implementation, you can reach out to us at transform@cloudfonts.com.

Organizations today generate massive volumes of raw data from multiple sources such as ERP systems, CRMs, APIs, logs, and IoT devices. However, raw data by itself holds little value unless it is properly processed, transformed, and optimized for analytics. In our data engineering journey, we faced challenges in building scalable and maintainable ETL pipelines that could handle growing data volumes while still delivering reliable insights. Azure Databricks helped us bridge the gap between raw data and business-ready insights. In this blog, we’ll walk through ETL best practices using Azure Databricks and how they helped us build efficient, production-grade data pipelines. Why ETL Best Practices Matter When working with large-scale data pipelines: – Raw data arrives in different formats and structures– Poorly designed ETL jobs lead to performance bottlenecks– Debugging and maintaining pipelines becomes difficult– Data quality issues propagate to downstream reports Key challenges we faced: – Tight coupling between ingestion and transformation– Reprocessing large datasets due to small logic changes– Lack of standardization across pipelines– Slow query performance on analytical layers Solution Architecture Overview Key Components: – Azure Data Lake Storage Gen2– Azure Databricks– Delta Lake– Power BI / Analytics Tools ETL Flow: – Ingest raw data from source systems into the Raw (Bronze) layer– Clean, validate, and standardize data in the Processed (Silver) layer– Apply business logic and aggregations in the Curated (Gold) layer– Expose curated datasets to reporting and analytics tools Step-by-Step ETL Best Practices with Azure Databricks Step 1: Separate Data into Layers (Bronze, Silver, Gold) – Bronze Layer: Store raw data exactly as received– Silver Layer: Apply cleansing, deduplication, and schema enforcement– Gold Layer: Create business-ready datasets and aggregations This separation ensures reusability and prevents unnecessary reprocessing. Step 2: Use Delta Lake for Reliability – Store tables in Delta format– Enable schema enforcement and schema evolution– Leverage time travel for data recovery and debugging Step 3: Build Incremental Pipelines – Process only new or changed data using watermarking– Avoid full reloads unless absolutely required– Design pipelines to safely re-run without duplications Step 4: Parameterize and Modularize Code – Use notebook parameters for environment-specific values– Create reusable functions for common transformations– Avoid hardcoding paths, table names, or business rules Step 5: Optimize Performance Early – Use partitioning based on query patterns– Apply Z-ORDER on frequently filtered columns– Cache datasets selectively for heavy transformations Step 6: Implement Data Quality Checks – Validate nulls, ranges, and duplicate records– Log rejected or invalid records separately– Fail pipelines early when critical checks fail Benefits of Following These ETL Best Practices – Scalability: Easily handle growing data volumes– Reliability: ACID-compliant pipelines with Delta Lake– Maintainability: Modular and reusable code structure– Performance: Faster queries and optimized storage– Cost Efficiency: Reduced compute usage through incremental processing Conclusion Transforming raw data into meaningful insights requires more than just moving data from one place to another. By following ETL best practices with Azure Databricks, we were able to build robust, scalable, and high-performing data pipelines that deliver reliable insights to the business. If your Databricks pipelines are becoming complex, slow, or difficult to maintain, it might be time to revisit your ETL design. Start applying these best practices today and turn your raw data into insights that truly drive decision-making. I hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.

In enterprise environments, securing data is not optional – it is foundational. As organizations scale their analytics with Microsoft Power BI, controlling who sees what data becomes critical. Instead of assigning access manually to individual users, modern security architecture leverage’s identity groups from Microsoft Entra ID (formerly Azure AD). When combined with Row-Level Security (RLS), this approach enables scalable, governed, and maintainable data access control. In this blog, we’ll explore how to design secure Power BI reports using Microsoft Entra ID group-based RLS. 1. What is Row-Level Security (RLS)? Row-Level Security (RLS) restricts data access at the row level within a dataset. For example: RLS ensures sensitive data is protected while keeping a single shared dataset. 2. What is Microsoft Entra ID? Microsoft Entra ID (formerly Azure AD) is Microsoft’s identity and access management platform. It allows organizations to: Using Entra ID groups for RLS ensures that security is managed at the identity layer rather than manually inside Power BI. 3. Why Use Group-Based RLS Instead of User-Level Assignment?  Individual User Assignment Challenges  Group-Based RLS Benefits This approach aligns with least-privilege and zero-trust security principles. Step-by-Step Guide to Sorting in the Paginated Report Step 1: Create group in Azure portal and select the require member Step 2: Once group is created, Go to Power BI service Step 3: Go to manage permission Step 4: Add group name, now available group member can access the report To conclude, designing secure Power BI reports is not just about creating dashboards — it is about implementing a governed data access strategy. By leveraging Microsoft Entra ID group-based Row-Level Security This approach transforms Power BI from a reporting tool into a secure, enterprise-grade analytics platform. Start by defining clear security requirements, create Microsoft Entra ID groups aligned with business structure, and map them to Power BI roles. For more enterprise Power BI security and architecture insights, stay connected and explore our upcoming blogs. I Hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com.

As organizations scale, full data reload pipelines quickly become inefficient and risky. Reporting refresh windows grow longer, source systems experience increased load, and data duplication issues begin to surface. In our recent Unity Catalog-based Lakehouse implementation, we modernized incremental data processing using a structured Change Data Capture (CDC) strategy. Instead of reloading entire datasets daily, we captured only incremental changes across CRM, ERP, HR, and finance systems and governed them through Unity Catalog. This blog explains how we designed and implemented CDC in a production-ready Lakehouse architecture, the decisions behind our approach, and the technical patterns that made it scalable. One of the first challenges in CDC implementations is avoiding hardcoded logic for every entity. Centralized Incremental Control Using Metadata Configuration Instead of embedding incremental rules inside notebooks, we designed a centralized configuration table that drives CDC dynamically. Each record in this control table defines: This allowed us to manage incremental extraction logic centrally without modifying pipeline code for every new table. Fig – Azure Storage Table showing IncrementalField and Timestamp columns Why This Matters This configuration-driven design enabled: Most CDC blogs discuss theory. Few show how incremental control is actually governed in production. Bronze Layer: Append-Only Incremental Capture Once incremental records are identified, they land in the bronze layer in Delta format. Key design decisions: The Bronze layer acts as the immutable change log of the system. This ensures: Bronze is not for reporting. It is for reliability. Structuring CDC Layers with Unity Catalog To ensure proper governance and separation of concerns, we structured our Lakehouse using Unity Catalog with domain-based schemas. Each environment (dev, test, prod) had its own catalog. Within each catalog: (Unity Catalog Bronze schema view) Why Unity Catalog Was Critical Unity Catalog ensured: CDC without governance can become fragile. Unity Catalog added structure and security to the incremental architecture. Silver Layer: Applying CDC with Delta MERGE The Silver layer is where CDC logic is applied. We implemented Type 1 Change Data Capture using Delta Lake MERGE operations. The logic follows: If a job runs twice, the data remains consistent. We intentionally chose Type 1 because reporting required the latest operational state rather than historical tracking. Handling Late-Arriving Data One common CDC failure point is late-arriving records. If extraction logic strictly uses: modified_timestamp > last_run_timeSome records may be missed due to clock drift or processing delays. To mitigate this, we: This ensured no silent data loss. Governance and Power BI Integration A key architectural decision was limiting Power BI access strictly to Gold tables. Through Unity Catalog: This ensured reporting teams could not accidentally query raw incremental data. The result was a clean, governed reporting layer powered by curated Delta tables. Performance Optimization Considerations To maintain optimal performance: Compared to full data reloads, incremental CDC significantly reduced cluster runtime and improved refresh stability. Common CDC Mistakes We Avoided During implementation, we intentionally avoided: These mistakes often appear only after production failures. Designing CDC carefully from the start prevented costly refactoring later. Business Impact By implementing CDC within a Unity Catalog-governed Lakehouse: The architecture is now scalable and future ready. To encapsulates, change data capture is not just an incremental filter. It is a disciplined architectural pattern. When combined with: It becomes a powerful foundation for enterprise analytics. Organizations modernizing their reporting platforms must move beyond full reload pipelines and adopt structured CDC approaches that prioritize scalability, reliability, and governance. If you found this blog useful and would like to discuss, ,Get in touch with CloudFronts at transform@cloudfronts.com.

Why Incremental Loads Matter When integrating data from external systems, whether it’s a CRM, an ERP like Business Central, or an HR platform like Zoho People, pulling all data every time is expensive, slow, and unnecessary. The smarter approach is to track what has changed since the last successful run and fetch only that delta. This is the core idea behind an incremental data pipeline: identify a timestamp or sequence field in your source system, persist the last-known watermark, and use it as a filter on your next API call. Azure Logic Apps, paired with Azure Table Storage as a lightweight checkpoint store, gives you everything you need to implement this pattern without managing any infrastructure. Architecture Overview Instead of one large workflow doing everything, we separate responsibilities. One Logic App handles scheduling and orchestration. Another handles actual data extraction. Core components: 3. Metadata Design (Azure Table) Instead of hardcoding entity names and fields inside Logic Apps, we define them in Azure Table Storage. Example structure: PartitionKey RowKey IncrementalField displayName entity businesscentral 1 systemCreatedAt Vendor Ledger Entry vendorLedgerEntries zohopeople 1 modifiedtime Leave leave Briefly, this table answers three questions: – What entity should be extracted?– Which column defines incremental logic?– What was the last successful checkpoint? When you want to onboard a new entity, you add a row. No redesign needed. 4. Logic App 1 – Scheduler Trigger: Recurrence (for example, every 15 minutes) Steps: This Logic App should not call APIs directly. Its only job is orchestration. Keep it light. 5. Logic App 2 – Incremental Processor Trigger: HTTP (called from Logic App 1) Functional steps: Example: This is where the real work happens. 6. Checkpoint Strategy Each entity must maintain: – LastSuccessfulRunTime– Status– LastRecordTimestamp After successful extraction: Checkpoint = max(modifiedOn) from extracted data. This ensures: Checkpoint management is the backbone of incremental loading. If this fails, everything fails. This pattern gives you a production-grade incremental data pipeline entirely within Azure’s managed services. By centralizing entity configuration and watermarks in Azure Table Storage, you create a data-driven pipeline where adding a new integration is as simple as inserting a row — no code deployment required. The two-Logic-App architecture cleanly separates orchestration from execution, enables parallel processing, and ensures your pipeline is resilient to failures through checkpoint-based watermark management. Whether you’re pulling from Business Central, Zoho People, or any REST API that exposes a timestamp field, this architecture scales gracefully with your data needs. Explore the case study below to learn how Logic Apps were implemented to solve key business challenges: Ready to deploy AIS to seamlessly connect systems and improve operational cost and efficiency? Get in touch with CloudFronts at transform@cloudfronts.com.