What happens when a global manufacturing giant needs to modernize its operations without grinding critical business processes to a halt? The answer is not a rip-and-replace approach – it is a carefully engineered integration strategy that lets modern and legacy systems co-exist, communicate, and complement each other. Are you planning an ERP upgrade but worried about what happens to the legacy systems your operations depend on? If so, this is for you. One of North America’s leading commercial vehicle manufacturers faced exactly this dilemma. With decades of investment in legacy financial and warehouse management systems, a hard cutover to a new ERP was not an option. Yet the need for a modern, scalable platform was undeniable. Their solution? Introduce Microsoft Dynamics 365 Finance & Operations (D365 F&O) as the new operational backbone – while keeping their legacy systems in play for financial control – and build robust, bi-directional integrations to bridge both worlds. At CloudFronts, we had the privilege of architecting and implementing those integrations. This blog walks you through three core data flows: Spot Purchase Orders, Advance Shipment Notices (ASN), and Goods Receipt Notes (GRN) – and what it really takes to make a modern ERP talk to a legacy system without missing a beat. Why Replace When You Can Integrate? Legacy systems in large manufacturers are not just old software. They carry years of financial logic, vendor relationships, and compliance configurations that are too risky to discard overnight. Replacing them introduces enormous operational and compliance risk. Doing nothing, however, is not an option either. The approach our client took – and one we increasingly recommend for manufacturers, distributors, and large enterprises – is a co-existence model: This means the business gets the agility of a modern ERP on day one, without putting financial operations at risk. The three integrations do the heavy lifting. Architecture at a Glance Before diving into each integration, it helps to understand the overall data flow pattern and the Azure services involved: Component Role D365 F&O System of record for purchasing and receiving operations Legacy System Retains financial control, inventory management authority Azure Logic Apps Parent-child middleware: orchestrates, transforms, and routes data Azure Blob Storage Checkpoint management for reliable incremental processing Azure Table Storage Full execution logs for traceability, audit, and failure replay The three integrations work in concert: Integration 1: Spot Purchase Orders — D365 F&O to Legacy Business Problem A Spot Purchase Order is an ad-hoc purchase order raised outside of long-term contracts — often for urgent material procurement. Spot POs are created and managed in D365 F&O by procurement teams. However, the legacy system is the system of financial record, meaning every Spot PO created in D365 must be reflected in the legacy system for financial commitment tracking and vendor payment processing. Without integration, this would require manual re-entry – a process prone to error, delay, and duplication. How the Integration Works Parent Logic App – Spot PO Orchestrator The primary Logic App runs on a scheduled recurrence and uses a checkpoint mechanism stored in Azure Blob Storage to fetch only incremental changes – purchase orders created or modified since the last successful run. This ensures efficiency and prevents reprocessing of already-handled records. The workflow determines the operation type required for each PO: For each scenario, the Logic App fetches enriched data from multiple D365 F&O OData entities and constructs a structured JSON payload tailored for the legacy system’s API. ⚙ Tech Note: OData Entities Used PurchaseOrderHeaders, PurchaseOrderLinesV2, PurchaseLineDataEntities, WHSPurchLines, StatusCustomDatas Child Logic App – SendRequest (Reusable) Rather than embedding API communication logic directly in the orchestrator, we separated it into a reusable child Logic App. This child app receives the constructed payload, retrieves an OAuth 2.0 Bearer token, and executes the HTTP POST call to the legacy system’s API endpoint. This modular design pays dividends during maintenance: any change to authentication logic or API communication is made once in the child app and automatically applies to all parent integrations. Failed Record Handler Every enterprise integration needs robust failure recovery. When an API call fails: Sample Payload – Spot PO Create Sample JSON Payload: {   “userId”: “JSMITH”,   “order”: “456789”,   // Last 6 digits of D365 PO number   “vendor”: “VEND001”,   “receiptLoc”: “SITE01”,   “vendorOvrdCd”: “14”,   “lineItems”: [{     “orderLine”: “001”,     “item”: “ITEM001”,     “openQty”: 10,     “deliveryDate”: “061526”,  // MMddyy format for legacy compatibility     “comment”: “MPSSYS order – JSMITH” }] } } ✓ Business Impact: Zero manual re-entry of purchase orders between systems. Every Spot PO created or changed in D365 F&O is automatically reflected in the legacy system within minutes. Integration 2: Advance Shipment Notices — Legacy to D365 F&O Business Problem An Advance Shipment Notice (ASN) is a notification sent by the legacy WMS to the receiving system, informing it of an incoming shipment before it physically arrives. D365 F&O needs to receive ASNs to create Inbound Load Headers and Load Lines – enabling warehouse teams to prepare for receiving. Without this integration, receiving teams in D365 would be blind to incoming shipments until trucks arrived at the dock – eliminating any opportunity for advance dock scheduling, labor planning, or inventory pre-positioning. The Hybrid Integration Approach This integration presented an interesting technical challenge: the standard D365 F&O Inbound ASN V5 API supports a well-defined XML format, but the business required additional fields beyond what the standard API supports. The solution was a two-step Hybrid ASN Integration approach: ⚙ Tech Note: API Endpoint Pattern Insert:  POST {{BASE_URL}}/api/connector/enqueue/{{ACTIVITY_ID}}?entity=Inbound ASN V5 Enrich:  PATCH on InboundLoadHeaders and WHSASNWorkData Smart Insert vs. Update Determination To handle scenarios where an ASN might be re-sent for corrections or resynchronization, the integration includes a check before processing: This idempotent design prevents duplicate inbound loads from being created when the legacy system re-sends an ASN. One nuance worth noting: in D365’s standard ASN structure, the LoadId, ShipmentId, and LicensePlateNumber must carry the same value. The legacy system’s outbound ASN payload is configured to honour this requirement – ensuring clean data entry … Continue reading Connecting a Modern ERP with Legacy System →

Summary A Houston-based cybersecurity firm eliminated report failures (~65%) by replacing SSRS with an Azure Function App pipeline. Dynamics 365 bound action ensured authentication stayed internal, bypassing Defender-related token failures. Integrated Power Pages, Power Automate, Dynamics 365, and Azure Functions for real-time PDF generation. Report generation time reduced from 3–8 minutes to under 15 seconds with zero infrastructure overhead. Table of Contents 1. About the Customer 2. The Challenge 3. The Solution 4. Technical Implementation 5. Business Impact 6. FAQs 7. Conclusion 1. About the Customer The client is a technology consulting and cybersecurity services firm based in Houston, Texas. They manage multiple concurrent client engagements using Dynamics 365 Project Operations as their core platform. Project managers and clients access live project data through a customer-facing portal built on Microsoft Power Pages. 2. The Challenge The organization needed one-click downloadable Project Status Reports from their Power Pages portal covering risks, issues, logs, and timelines. Their SSRS-based solution failed frequently due to authentication breakdowns caused by Microsoft Defender for Cloud Apps across multiple service boundaries. Key pain points: Silent authentication failures with no clear errors Retry delays of 60–90 seconds per attempt Separate SSRS infrastructure dependency Slow report customization cycle Project managers avoided generating reports during live meetings due to reliability concerns. 3. The Solution At Cloudfronts, while working on this project, I replaced the SSRS pipeline entirely with a synchronous, serverless architecture that keeps the authentication context inside the Dynamics 365 service layer. Technologies Used: Dynamics 365 Project Operations Power Pages Power Automate Plugins Azure Function Apps The solution generates fully formatted PDFs in real time using structured JSON payloads. This eliminated authentication failures while significantly improving speed and reliability. 4. Technical Implementation 1] Power Pages Button triggers Flow A “Download Report” button captures the project GUID and triggers a Power Automate flow with real-time progress feedback. 2] Dynamics 365 Plugin prepares JSON payload A bound action plugin retrieves all project data and converts it into a clean JSON payload for PDF generation. 3] Azure Function generates PDF The Azure Function processes the JSON and generates a formatted PDF, returning it as a Base64 string. 4] SharePoint Integration The generated PDF is automatically stored in the associated SharePoint document location linked to the project. This ensures centralized document management, version control, and easy access for stakeholders directly within the project workspace. 5] Portal PDF Preview The Base64 PDF is rendered directly in the portal using an iframe, allowing instant preview and download. Video: End-to-end implementation of real-time PDF report generation. 5. Business Impact 100% success rate — zero failures post deployment Under 15 seconds report generation time No infrastructure — fully serverless Zero authentication failures Faster iteration for report updates Project managers can now confidently generate reports during live client meetings. 6. FAQs Why not fix the SSRS authentication issue instead of replacing SSRS entirely? The authentication failures were a structural consequence of traversing multiple service boundaries in an environment with strict Defender for Cloud Apps session policies. Fixing them would have required either relaxing those policies — which the client’s security posture did not permit — or re-architecting the data retrieval to stay inside the platform, which is exactly what the bound action approach achieves. Replacing SSRS also removed a separate infrastructure dependency and gave the client full control over report formatting in code. Can this pattern be reused for other document types in Dynamics 365? Yes. The Azure Function App’s renderer is data-driven — it consumes a JSON payload and builds tables from whatever keys are present. The Dynamics 365 plugin can be adapted to query any entity and produce an equivalent payload. CloudFronts has applied the same pattern to inspection records, summary reports, and client-facing status documents across Professional Services and Manufacturing implementations. Does this work for environments without Microsoft Defender for Cloud Apps? Yes. The architectural benefits — synchronous generation, serverless PDF rendering, no SSRS infrastructure, and in-browser preview — apply regardless of the security layer on the environment. 7. Conclusion Replacing SSRS with an Azure Function App-based PDF renderer resolved both the reliability and authentication problems in a single architectural shift, delivering instant, professional-quality Project Status Reports from a Microsoft Power Pages portal with no legacy reporting infrastructure to maintain. The key lesson from this project is that keeping authentication within the Dynamics 365 service layer — rather than bridging to external systems — eliminates an entire category of environment-specific failures that are otherwise very difficult to diagnose and fix. By keeping authentication within Dynamics 365 and leveraging serverless architecture, the solution delivers instant, high-quality reports without infrastructure overhead. This approach demonstrates how modern cloud-native patterns can eliminate entire classes of system failures while improving user experience dramatically. Ready to modernise document generation in your Dynamics 365 environment?CloudFronts builds scalable Power Platform and Dynamics 365 solutions that replace legacy reporting infrastructure and automate document workflows. Reach out at transform@cloudfronts.com. Shashank Keny Associate Consultant · CloudFronts Shashank Keny is an Associate Consultant at CloudFronts with 1.5+ years of experience in cloud, data, and business applications. He specializes in building scalable, API-driven architectures and integrating enterprise systems across the Microsoft ecosystem. He is a Certified Databricks Data Engineer with hands-on experience in Dynamics 365 Project Operations and Dynamics 365 Sales, along with delivering business intelligence solutions using Power BI. His expertise also extends to modern AI solutions, including building custom copilots and implementing intelligent applications using Azure AI Foundry. Passionate about solving real-world business challenges through data and AI, he focuses on delivering efficient, scalable, and production-ready solutions. Experience: 1.5+ years Certification: Databricks Certified Data Engineer Specialization: Dynamics 365 Project Operations, Power BI, Azure Integrations, AI Solutions View LinkedIn Profile

The Challenge of Scaling AI in Enterprises Many organizations invest in AI initiatives but struggle to scale beyond pilot projects. Custom-built solutions are expensive, difficult to govern, and often limited to a single use case. As a result, AI investments fail to deliver sustained business value. Why Automation Alone Is Not Enough Traditional automation relies on rigid rules and predefined workflows. While effective for simple tasks, it cannot adapt to changing business conditions. Enterprises need intelligent systems that can reason, decide, and act autonomously. Understanding AI Agents in Simple Terms AI agents are intelligent software systems that understand goals, plan actions, and execute multi-step workflows with minimal human intervention. Unlike chatbots, AI agents do not just answer questions they act on insights. What Agent Bricks Bring to the Business Agent Bricks are modular, reusable AI agent components that accelerate enterprise AI adoption. They enable organizations to deploy intelligent agents quickly while maintaining security, governance, and compliance. Ask Me Anything: Execution Powered by Agent Bricks In the Ask Me Anything solution, Agent Bricks power the execution layer. They continuously evaluate enterprise data, identify project readiness gaps, and respond to leadership queries in real time. Agent Bricks Workflow Execution (Testing Screenshot) Use Case Spotlight: PMO Assistant at Scale The PMO Assistant built using Agent Bricks operates continuously, monitoring upcoming projects and flagging risks early. This reduces dependency on manual reporting and enables PMOs to focus on proactive delivery management. Business Value of an AI Workforce From a business perspective, Agent Bricks enable faster AI deployment, lower operational costs, and consistent decision-making across departments. Enterprises can scale AI solutions confidently without rebuilding logic for every new use case. Moving from Experiments to Execution To conclude, Agent Bricks help organizations move from isolated AI experiments to production-ready AI solutions. CloudFronts partners with enterprises to build scalable, governed AI workforces that deliver measurable business outcomes. I hope you found this blog useful, and if you would like to discuss anything or explore a future implementation, you can reach out to us at transform@cloudfonts.com.

The Growing Challenge of Project Readiness As organizations grow, managing project readiness becomes increasingly complex. Data related to projects, resources, and timelines is spread across CRM systems, project management tools, and booking platforms. Team Leads, CTOs, and CEOs often struggle to gain a real-time, consolidated view of whether projects are truly ready to start. This lack of visibility leads to delayed project kick-offs, inefficient resource utilization, and increased operational risk. Why Traditional Systems Fail at Scale Traditional reporting and AI systems are not designed to handle the dynamic nature of growing enterprises. They respond only to explicit prompts, operate in single-step workflows, and require significant human intervention. Leadership teams depend heavily on manual checks and follow-ups, which consume time and still fail to provide timely insights. The Shift Toward Agentic AI Organizations are now shifting from static AI responses to autonomous AI-driven decision-making. Agentic AI enables systems to understand intent, evaluate multiple data points, and decide what action to take next. This shift is critical for enterprises that want to move from reactive reporting to proactive management. What Genie Space Means for Business Leaders Genie Space is an AI-powered natural language analytics layer that allows business users to ask questions in plain English and receive immediate, governed answers. Without requiring SQL knowledge or technical expertise, Genie Space empowers leaders to access insights directly while maintaining full enterprise security and compliance through Unity Catalog. Ask Me Anything: A Unified Intelligence Layer The Ask Me Anything solution leverages Genie Space as the central intelligence layer. It connects securely to enterprise systems, preserves conversational context, and delivers consistent insights across departments. This unified approach ensures that leadership teams rely on a single source of truth for decision-making. Ask Me Anything Product Architecture Diagram Use Case Spotlight: PMO Assistant for Project Readiness In a typical PMO environment, project managers lack real-time visibility into execution readiness. Tasks may not be configured, resources may not be aligned, and risks often surface too late. The PMO Assistant powered by Genie continuously monitors projects scheduled to start within a defined window and provides instant readiness insights. Business Impact of Genie-Powered Insights By implementing Genie Space, organizations significantly reduce manual reporting effort, improve delivery confidence, and enable leadership teams to focus on strategic priorities. Faster insights lead to quicker decisions, lower operational costs, and improved customer satisfaction. To conclude, Genie Space transforms how organizations interact with their data. Instead of searching for information, leaders receive instant, trusted answers. CloudFronts helps enterprises design and deploy Genie-powered solutions that improve project visibility and decision-making across the organization. I hope you found this blog useful, and if you would like to discuss anything or explore a future implementation, you can reach out to us at transform@cloudfonts.com.

Organizations today generate massive volumes of raw data from multiple sources such as ERP systems, CRMs, APIs, logs, and IoT devices. However, raw data by itself holds little value unless it is properly processed, transformed, and optimized for analytics. In our data engineering journey, we faced challenges in building scalable and maintainable ETL pipelines that could handle growing data volumes while still delivering reliable insights. Azure Databricks helped us bridge the gap between raw data and business-ready insights. In this blog, we’ll walk through ETL best practices using Azure Databricks and how they helped us build efficient, production-grade data pipelines. Why ETL Best Practices Matter When working with large-scale data pipelines: – Raw data arrives in different formats and structures– Poorly designed ETL jobs lead to performance bottlenecks– Debugging and maintaining pipelines becomes difficult– Data quality issues propagate to downstream reports Key challenges we faced: – Tight coupling between ingestion and transformation– Reprocessing large datasets due to small logic changes– Lack of standardization across pipelines– Slow query performance on analytical layers Solution Architecture Overview Key Components: – Azure Data Lake Storage Gen2– Azure Databricks– Delta Lake– Power BI / Analytics Tools ETL Flow: – Ingest raw data from source systems into the Raw (Bronze) layer– Clean, validate, and standardize data in the Processed (Silver) layer– Apply business logic and aggregations in the Curated (Gold) layer– Expose curated datasets to reporting and analytics tools Step-by-Step ETL Best Practices with Azure Databricks Step 1: Separate Data into Layers (Bronze, Silver, Gold) – Bronze Layer: Store raw data exactly as received– Silver Layer: Apply cleansing, deduplication, and schema enforcement– Gold Layer: Create business-ready datasets and aggregations This separation ensures reusability and prevents unnecessary reprocessing. Step 2: Use Delta Lake for Reliability – Store tables in Delta format– Enable schema enforcement and schema evolution– Leverage time travel for data recovery and debugging Step 3: Build Incremental Pipelines – Process only new or changed data using watermarking– Avoid full reloads unless absolutely required– Design pipelines to safely re-run without duplications Step 4: Parameterize and Modularize Code – Use notebook parameters for environment-specific values– Create reusable functions for common transformations– Avoid hardcoding paths, table names, or business rules Step 5: Optimize Performance Early – Use partitioning based on query patterns– Apply Z-ORDER on frequently filtered columns– Cache datasets selectively for heavy transformations Step 6: Implement Data Quality Checks – Validate nulls, ranges, and duplicate records– Log rejected or invalid records separately– Fail pipelines early when critical checks fail Benefits of Following These ETL Best Practices – Scalability: Easily handle growing data volumes– Reliability: ACID-compliant pipelines with Delta Lake– Maintainability: Modular and reusable code structure– Performance: Faster queries and optimized storage– Cost Efficiency: Reduced compute usage through incremental processing Conclusion Transforming raw data into meaningful insights requires more than just moving data from one place to another. By following ETL best practices with Azure Databricks, we were able to build robust, scalable, and high-performing data pipelines that deliver reliable insights to the business. If your Databricks pipelines are becoming complex, slow, or difficult to maintain, it might be time to revisit your ETL design. Start applying these best practices today and turn your raw data into insights that truly drive decision-making. I hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.

In enterprise environments, securing data is not optional – it is foundational. As organizations scale their analytics with Microsoft Power BI, controlling who sees what data becomes critical. Instead of assigning access manually to individual users, modern security architecture leverage’s identity groups from Microsoft Entra ID (formerly Azure AD). When combined with Row-Level Security (RLS), this approach enables scalable, governed, and maintainable data access control. In this blog, we’ll explore how to design secure Power BI reports using Microsoft Entra ID group-based RLS. 1. What is Row-Level Security (RLS)? Row-Level Security (RLS) restricts data access at the row level within a dataset. For example: RLS ensures sensitive data is protected while keeping a single shared dataset. 2. What is Microsoft Entra ID? Microsoft Entra ID (formerly Azure AD) is Microsoft’s identity and access management platform. It allows organizations to: Using Entra ID groups for RLS ensures that security is managed at the identity layer rather than manually inside Power BI. 3. Why Use Group-Based RLS Instead of User-Level Assignment?  Individual User Assignment Challenges  Group-Based RLS Benefits This approach aligns with least-privilege and zero-trust security principles. Step-by-Step Guide to Sorting in the Paginated Report Step 1: Create group in Azure portal and select the require member Step 2: Once group is created, Go to Power BI service Step 3: Go to manage permission Step 4: Add group name, now available group member can access the report To conclude, designing secure Power BI reports is not just about creating dashboards — it is about implementing a governed data access strategy. By leveraging Microsoft Entra ID group-based Row-Level Security This approach transforms Power BI from a reporting tool into a secure, enterprise-grade analytics platform. Start by defining clear security requirements, create Microsoft Entra ID groups aligned with business structure, and map them to Power BI roles. For more enterprise Power BI security and architecture insights, stay connected and explore our upcoming blogs. I Hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com.

As organizations scale, full data reload pipelines quickly become inefficient and risky. Reporting refresh windows grow longer, source systems experience increased load, and data duplication issues begin to surface. In our recent Unity Catalog-based Lakehouse implementation, we modernized incremental data processing using a structured Change Data Capture (CDC) strategy. Instead of reloading entire datasets daily, we captured only incremental changes across CRM, ERP, HR, and finance systems and governed them through Unity Catalog. This blog explains how we designed and implemented CDC in a production-ready Lakehouse architecture, the decisions behind our approach, and the technical patterns that made it scalable. One of the first challenges in CDC implementations is avoiding hardcoded logic for every entity. Centralized Incremental Control Using Metadata Configuration Instead of embedding incremental rules inside notebooks, we designed a centralized configuration table that drives CDC dynamically. Each record in this control table defines: This allowed us to manage incremental extraction logic centrally without modifying pipeline code for every new table. Fig – Azure Storage Table showing IncrementalField and Timestamp columns Why This Matters This configuration-driven design enabled: Most CDC blogs discuss theory. Few show how incremental control is actually governed in production. Bronze Layer: Append-Only Incremental Capture Once incremental records are identified, they land in the bronze layer in Delta format. Key design decisions: The Bronze layer acts as the immutable change log of the system. This ensures: Bronze is not for reporting. It is for reliability. Structuring CDC Layers with Unity Catalog To ensure proper governance and separation of concerns, we structured our Lakehouse using Unity Catalog with domain-based schemas. Each environment (dev, test, prod) had its own catalog. Within each catalog: (Unity Catalog Bronze schema view) Why Unity Catalog Was Critical Unity Catalog ensured: CDC without governance can become fragile. Unity Catalog added structure and security to the incremental architecture. Silver Layer: Applying CDC with Delta MERGE The Silver layer is where CDC logic is applied. We implemented Type 1 Change Data Capture using Delta Lake MERGE operations. The logic follows: If a job runs twice, the data remains consistent. We intentionally chose Type 1 because reporting required the latest operational state rather than historical tracking. Handling Late-Arriving Data One common CDC failure point is late-arriving records. If extraction logic strictly uses: modified_timestamp > last_run_timeSome records may be missed due to clock drift or processing delays. To mitigate this, we: This ensured no silent data loss. Governance and Power BI Integration A key architectural decision was limiting Power BI access strictly to Gold tables. Through Unity Catalog: This ensured reporting teams could not accidentally query raw incremental data. The result was a clean, governed reporting layer powered by curated Delta tables. Performance Optimization Considerations To maintain optimal performance: Compared to full data reloads, incremental CDC significantly reduced cluster runtime and improved refresh stability. Common CDC Mistakes We Avoided During implementation, we intentionally avoided: These mistakes often appear only after production failures. Designing CDC carefully from the start prevented costly refactoring later. Business Impact By implementing CDC within a Unity Catalog-governed Lakehouse: The architecture is now scalable and future ready. To encapsulates, change data capture is not just an incremental filter. It is a disciplined architectural pattern. When combined with: It becomes a powerful foundation for enterprise analytics. Organizations modernizing their reporting platforms must move beyond full reload pipelines and adopt structured CDC approaches that prioritize scalability, reliability, and governance. If you found this blog useful and would like to discuss, ,Get in touch with CloudFronts at transform@cloudfronts.com.

Why Incremental Loads Matter When integrating data from external systems, whether it’s a CRM, an ERP like Business Central, or an HR platform like Zoho People, pulling all data every time is expensive, slow, and unnecessary. The smarter approach is to track what has changed since the last successful run and fetch only that delta. This is the core idea behind an incremental data pipeline: identify a timestamp or sequence field in your source system, persist the last-known watermark, and use it as a filter on your next API call. Azure Logic Apps, paired with Azure Table Storage as a lightweight checkpoint store, gives you everything you need to implement this pattern without managing any infrastructure. Architecture Overview Instead of one large workflow doing everything, we separate responsibilities. One Logic App handles scheduling and orchestration. Another handles actual data extraction. Core components: 3. Metadata Design (Azure Table) Instead of hardcoding entity names and fields inside Logic Apps, we define them in Azure Table Storage. Example structure: PartitionKey RowKey IncrementalField displayName entity businesscentral 1 systemCreatedAt Vendor Ledger Entry vendorLedgerEntries zohopeople 1 modifiedtime Leave leave Briefly, this table answers three questions: – What entity should be extracted?– Which column defines incremental logic?– What was the last successful checkpoint? When you want to onboard a new entity, you add a row. No redesign needed. 4. Logic App 1 – Scheduler Trigger: Recurrence (for example, every 15 minutes) Steps: This Logic App should not call APIs directly. Its only job is orchestration. Keep it light. 5. Logic App 2 – Incremental Processor Trigger: HTTP (called from Logic App 1) Functional steps: Example: This is where the real work happens. 6. Checkpoint Strategy Each entity must maintain: – LastSuccessfulRunTime– Status– LastRecordTimestamp After successful extraction: Checkpoint = max(modifiedOn) from extracted data. This ensures: Checkpoint management is the backbone of incremental loading. If this fails, everything fails. This pattern gives you a production-grade incremental data pipeline entirely within Azure’s managed services. By centralizing entity configuration and watermarks in Azure Table Storage, you create a data-driven pipeline where adding a new integration is as simple as inserting a row — no code deployment required. The two-Logic-App architecture cleanly separates orchestration from execution, enables parallel processing, and ensures your pipeline is resilient to failures through checkpoint-based watermark management. Whether you’re pulling from Business Central, Zoho People, or any REST API that exposes a timestamp field, this architecture scales gracefully with your data needs. Explore the case study below to learn how Logic Apps were implemented to solve key business challenges: Ready to deploy AIS to seamlessly connect systems and improve operational cost and efficiency? Get in touch with CloudFronts at transform@cloudfronts.com.

From a customer perspective, the hardest part of data engineering isn’t building pipelines-it’s ensuring that the data customers rely on is accurate, consistent, and trustworthy. When reports show incorrect revenue or missing customer information, confidence drops quickly. This is where Delta Live Tables in Databricks makes a real difference for customers. Instead of customers dealing with broken dashboards, manual fixes in BI tools, or delayed insights, Delta Live Tables enforces data quality at the pipeline level. Using a Bronze–Silver–Gold approach: Data validation rules are built directly into the pipeline, and customers gain visibility into data quality through built-in monitoring-without extra tools or manual checks. Quick Preview Building data pipelines is not the difficult part. The real challenge is building pipelines that are reliable, monitored, and enforce data quality automatically. That’s where Delta Live Tables in Databricks makes a difference. Instead of stitching together notebooks, writing custom validation scripts, and setting up separate monitoring jobs, Delta Live Tables lets you define your transformations once and handles the rest. Let’s look at a simple example. Imagine an e-commerce company storing raw order data in a Unity Catalog table called: cf.staging.orders_raw The problem? The data isn’t perfect. Some records have negative quantities. Some orders have zero amounts. Customer IDs may be missing. There might even be duplicate order IDs. If this raw data goes straight into reporting dashboards, revenue numbers will be wrong. And once business users lose trust in reports, it’s hard to win it back. Instead of fixing issues later in Power BI or during analysis, we fix them at the pipeline level. In Databricks, we create an ETL pipeline and define a simple three-layer structure: Bronze for raw data, Silver for cleaned data, and Gold for business-ready aggregation. The Bronze layer simply reads from Unity Catalog: Nothing complex here. We’re just loading data from Unity Catalog. No manual dependency setup required. The real value appears in the Silver layer, where we enforce data quality rules directly inside the pipeline: Here’s what’s happening behind the scenes. Invalid rows are automatically removed. Duplicate orders are eliminated. Data quality metrics are tracked and visible in the pipeline UI. There’s no need for separate validation jobs or manual checks. This is what simplifies pipeline development. You define expectations declaratively, and Delta Live Tables enforces them consistently. Finally, in the Gold layer, we create a clean reporting table: At this point, only validated and trusted data reaches reporting systems. Dashboards become reliable. Delta Live Tables doesn’t replace databases, and it doesn’t magically fix bad source systems. What it does is simplify how we build and manage reliable data pipelines. It combines transformation logic, validation rules, orchestration, monitoring, and lineage into one managed framework. Instead of reacting to data issues after reports break, we prevent them from progressing in the first place. For customers, trust in data is everything. Delta Live Tables helps organizations ensure that only validated, reliable data reaches customer-facing dashboards and analytics. Rather than reacting after customers notice incorrect numbers, Delta Live Tables prevents poor-quality data from moving forward. By unifying transformation logic, data quality enforcement, orchestration, monitoring, and lineage in one framework, it enables teams to deliver consistent, dependable insights. The result for customers is simple: accurate reports, faster decisions, and confidence that the data they see reflects reality. I Hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com.

If you’re new to Databricks, chances are someone told you “Everything starts with a Notebook.” They weren’t wrong. In Databricks, a Notebook is where your entire data engineering workflow begins from reading raw data, transforming it, visualizing trends, and even deploying jobs. It’s your coding lab, dashboard, and documentation space all in one. What Is a Databricks Notebook? A Databricks Notebook is an interactive environment that supports multiple programming languages such as Python, SQL, R, and Scala. Each Notebook is divided into cells you can write code, add text (Markdown), and visualize data directly within it. Unlike local scripts, Notebooks in Databricks run on distributed Spark clusters. That means even your 100 GB dataset is processed within seconds using parallel computation. So, Notebooks are more than just code editors they are collaborative data workspaces for building, testing, and documenting pipelines. How Databricks Notebooks Work Under the hood, every Notebook connects to a cluster a group of virtual machines managed by Databricks. When you run code in a cell, it’s sent to Spark running on the cluster, processed there, and results are sent back to your Notebook. This gives you the scalability of big data without worrying about servers or configurations. Setting Up Your First Cluster Before running a Notebook, you must create a cluster it’s like starting the engine of your car. Here’s how: Step-by-Step: Creating a Cluster in a Standard Databricks Workspace Once the cluster is active, you’ll see a green light next to its name now it’s ready to process your code. Creating Your First Notebook Now, let’s build your first Databricks Notebook: Your Notebook is now live ready to connect to data and start executing. Loading and Exploring Data Let’s say you have a sales dataset in Azure Blob Storage or Data Lake. You can easily read it into Databricks using Spark: df = spark.read.csv(“/mnt/data/sales_data.csv”, header=True, inferSchema=True)display(df.limit(5)) Databricks automatically recognizes your file’s schema and displays a tabular preview.Now, you can transform the data: from pyspark.sql.functions import col, sumsummary = df.groupBy(“Region”).agg(sum(“Revenue”).alias(“Total_Revenue”))display(summary) Or, switch to SQL instantly: %sqlSELECT Region, SUM(Revenue) AS Total_RevenueFROM sales_dataGROUP BY RegionORDER BY Total_Revenue DESC Visualizing DataDatabricks Notebooks include built-in charting tools.After running your SQL query:Click + → Visualization → choose Bar Chart.Assign Region to the X-axis and Total_Revenue to the Y-axis.Congratulations — you’ve just built your first mini-dashboard! Real-World Example: ETL Pipeline in a Notebook In many projects, Databricks Notebooks are used to build ETL pipelines: Each stage is often written in a separate cell, making debugging and testing easier.Once tested, you can schedule the Notebook as a Job running daily, weekly, or on demand. Best Practices To conclude, Databricks Notebooks are not just a beginner’s playground they’re the backbone of real data engineering in the cloud.They combine flexibility, scalability, and collaboration into a single workspace where ideas turn into production pipelines. If you’re starting your data journey, learning Notebooks is the best first step.They help you understand data movement, Spark transformations, and the Databricks workflow everything a data engineer need. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudFronts.com