Category Archives: Logic App
Forecasting with Prophet-When Does It Make Sense
Summary Prophet is a forecasting method from Metaās Core Data Science team. It works well for business numbers that follow repeating patterns, such as retail sales, website traffic, and revenue. This blog explains where Prophet helps, why it is easy to understand and use, and when it needs extra care. Table of Contents 01 Introduction 02 Retail Sales Forecasting: Planning for Demand 03 Forecasting Website Traffic and Infrastructure Needs 04 Forecasting Business Numbers: Growth and Operational Trends 05 Why Use Prophet? 06 When Prophet May Not Fit Well 07 Making Forecasts Useful for the Business 08 FAQs 09 Conclusion Introduction Every organization generates data. Sales, customer activity, website traffic, daily operations, and system usage all build up a history over time. The value of this data is not only in understanding what happened yesterday. It is also in preparing for what may happen next and making better decisions today. This is where forecasting is useful for business leaders. Prophet, developed by Metaās Core Data Science team, is a practical way to forecast business numbers that follow repeating patterns, seasonal changes, and changing growth. However, Prophet is not suitable for every forecasting problem. How well it works depends on the business problem, the past data available, and the decisions the forecast will support. Where Prophet Can Help 1 Retail Sales Plan stock, purchasing and delivery around weekly, seasonal and holiday patterns. 2 Website Traffic Forecast visits and usage to plan how much system capacity is needed. 3 Business Numbers Forecast revenue, new customers and user growth for planning. 1. Retail Sales Forecasting: Planning for Demand Retail demand changes throughout the year. Sales are affected by weekly shopping habits, seasons, promotions, and holidays. These patterns directly affect inventory, purchasing, delivery planning, and revenue expectations. A 2020 research study, Application of Facebookās Prophet Algorithm for Successful Sales Forecasting Based on Real-world Data, used Prophet on actual sales data from a large retail company in Bosnia and Herzegovina. The study showed how Prophet, together with testing on past data, can be used to forecast retail sales and check how reliable the forecast is for different products. Why Prophet Makes Sense Seasonal patterns: Retail sales often rise and fall on weekly and yearly cycles, and Prophet can include these. Holidays: Known holidays and special events can be added to the forecast. Changing trends: The forecast can allow sales trends to change instead of assuming steady growth. Many products: The same approach can be used for many products. The reliability of each productās forecast should be checked separately. The goal is not only to predict sales. It is to help the business make better decisions about stock availability, purchasing schedules, delivery planning, and cash planning. A forecast is useful when it connects directly to a business decision. 2. Forecasting Website Traffic and Infrastructure Needs Websites and online platforms produce operational data all the time. Website visits, app usage, and system load often follow patterns based on weekdays, weekends, seasons, and business events. Metaās original work on Prophet aimed to make business forecasting easier to use at a larger scale, including forecasting the numbers used to plan how much system capacity is needed. One example is using past website traffic to forecast future demand. The aim is to understand how traffic is likely to change and whether the current infrastructure can support it. Why Prophet Makes Sense More than one pattern: Daily traffic can follow both weekly and yearly patterns. Changing growth: Growth can change as products change or more people start using them. Capacity planning: Forecasts help infrastructure teams plan for future needs. Spotting differences: Forecasts can be compared with actual traffic to notice unexpected changes. For leaders, the benefit is linking the forecast to infrastructure spending decisions. Instead of relying only on past peak usage or reacting once demand rises, teams can use forecast demand as one more input when planning capacity. This is helpful when cost, system performance, and customer experience all need to be balanced. 3. Forecasting Business Numbers: Growth and Operational Trends Organizations regularly track numbers such as revenue, new customers, user activity, and operational volumes. Prophet was designed to support business forecasting at larger scale, including numbers such as revenue and user growth. These numbers often show repeating patterns, an overall growth trend, and changes caused by business events. Why Prophet Makes Sense Easy to understand: The trend and seasonal parts of the forecast make it easier to read and explain. Changing growth: It can handle growth that speeds up or slows down over time. Known business events: Events such as product launches can be included when their effect is described properly. Easy to review: Business teams can look at the trend and seasonal parts instead of receiving only a final number. This is useful when forecasts need to be reviewed and understood by finance, operations, and leadership, and not only by the data team. The forecast can support revenue planning, growth estimates, resource allocation, and operational reviews. One thing to keep in mind: a forecast of a business number does not explain why that number is changing. Understanding the effect of pricing, marketing campaigns, customer behavior, or other business factors may need additional analysis. Why Use Prophet? Prophet builds a forecast from parts of the past data: the trend, the seasonal patterns, and the effect of holidays and events. These parts reflect patterns that are common in business data, and they do not require a lot of manual setup. What Prophet Looks At The parts of past data that shape a Prophet forecast 1 Trend The general direction of the numbers over time. 2 Weekly patterns M T W T F S S Repeating changes across the days of the week. 3 Yearly patterns Jan Dec Repeating changes across the seasons of the year. 4 Holidays and events Known dates, such as holidays or product launches, that affect the numbers. From Past Data to a Business Decision 1 Past Business Data Sales, traffic, revenue or other numbers. 2 Prophet … Continue reading Forecasting with Prophet-When Does It Make Sense
Share Story :
Dynamics 365 Project Operations and Business Central Integration: Automate Data Sync Without Manual Work
Please find this CloudFronts Product at ā CloudFronts PO-BC Integration Module | Microsoft Marketplace Watch the webinar ā CloudFronts PO-BC Integration Webinar | YouTube Summary Dynamics 365 Project Operations (PO) holds customers, projects and time. Business Central (BC) holds invoices and payments. When the two are not connected, people copy data by hand, which is slow and causes mistakes. The CloudFronts PO-BC Integration Module is a Dynamics 365 Project Operations and Business Central integration. You set up the link once, and every change in PO shows up in BC automatically. The result is no double typing, fewer mistakes and faster billing. This blog shows how it works, step by step. Table of Contents 1Introduction 2Business Challenge 3Why an Integration Hub? 4Implementation Procedure 5End-to-End Working 6Why This Approach Works 7Real-World Use Cases 8Frequently Asked Questions 9Conclusion Introduction: Why Connect Dynamics 365 Project Operations with Business Central Many project-based businesses use two systems. Project Operations (PO) is where the sales and project teams keep customers, projects and time. Business Central (BC) is where the finance team keeps invoices, payments and money records. Both systems need the same details. A customer made in PO must also exist in BC. A project made in PO must also be there when it is time to send an invoice. Without a link, someone has to copy all of this by hand. Here is the full picture of how work moves from a lead to a payment. The top row is PO and the bottom row is BC. The arrows show where the two systems share data. The full flow from lead to payment, and where PO and BC share data. The Solution: We built an Integration Hub. It is a simple web app where you connect PO and BC, set the rules once, and let the data move on its own. Business Challenge: Manual Data Entry Between PO and BC Think about a normal project business. The sales team makes a customer in PO. The project team makes a project and adds tasks. The finance team then needs the same customer and project in BC to send invoices. These details need to be shared between the two systems: Customers and contacts Projects and tasks Planned work and costs Approved time and expenses Vendor invoices and purchase orders Paid status of invoices When people copy these by hand, many problems come up: Extra effort: The same details are typed in two systems. Mistakes: A wrong phone number or address is easy to type. Details that do not match: The two systems slowly stop agreeing with each other. Late invoices: If data reaches finance late, money comes in late. Hard to grow: More customers and projects mean more copying and checking. The Objective: When something changes in PO, BC should get the change by itself. No one should have to copy anything by hand. Why Use an Integration Hub for PO and BC Integration? The Integration Hub is one place where you create and manage every link between PO and BC. You fill a form, choose what to send, and set simple rules. You do not need to write a lot of code. With the Integration Hub, you can: Set up each link once Choose what data moves, from where, and to where Write simple rules for which detail goes where Test the result before it goes live See all links in one list, with their status and last runs Add new links as your business grows In simple words: Think of the Integration Hub as a control room. From here you decide what data moves, from where, and to where. Implementation Procedure: How to Integrate Project Operations with Business Central Step 1 ā Choose the Project On the first screen you pick a project. A project here is a group of links. We use the project named PO-BC, which is for Project Operations and Business Central. Choose the project. We are using PO-BC. Step 2 ā See the List of Integrations Inside PO-BC you can see all the links. Each row shows its order number, its status, and its last few runs. You can run, edit or deploy each one from here. To add a new one, click New Integration. The list of integrations. Click New Integration to add one. Step 3 ā Fill the New Integration Form A form opens on the right side. Here you tell the platform: Integration Name: A name for this link. For example, Accounts. Integration Type: Master means basic details that rarely change, like customers. Transactional means things that happen often, like invoices. Source System and Source Entity: Where the data starts, and what kind of record it is. Here the system is PO, and the record type is accounts. Target System and Target Entity: Where the data goes, and what kind of record it becomes. Here the system is BC, and the record type is customer. Sequence No.: The order in which links run. Customers must exist first, so this is 1. The new integration form. Use Sync CRM Fields to add any missing fields that BC needs on the PO side. Tip: There is a handy button called Sync CRM Fields. Sometimes BC needs a detail that PO does not have yet. This button creates the missing field in PO for you. Scroll down and add a short description. This helps everyone understand later what the link does. Then click Create Integration. Add a clear description, then click Create Integration. Step 4 ā Open the Flow After you create the link, you can open its flow. It is a simple picture with boxes from left to right: Source: the PO data (accounts) Data Mapper: the rules that change the data into the shape BC needs Target: the BC data (customer) Return to Source: an optional step to send data back to PO The flow. The Data Mapper box is where we set the rules. Step 5 ā Write the Mapping Rules PO and BC use different names for … Continue reading Dynamics 365 Project Operations and Business Central Integration: Automate Data Sync Without Manual Work
Share Story :
From Multiple Data Sources to Azure Databricks: A Zero-Downtime Power BI Migration for Future AI Readiness
One Data Platform, Future-Ready for AI – Power BI Report Migration to Azure Databricks Summary Every organization wants to be AI-ready. Very few are data-ready. Business data typically lives across ERP, CRM, HR, and finance systems, each handled differently and reported separately. The result is familiar to most leadership teams: reports that do not agree, logic rebuilt in every dashboard, and AI initiatives that stall because the data underneath them cannot be trusted. This article makes the case for one data platform – a single, governed foundation where all business data is brought together, processed consistently, and made ready for both reporting and AI. We share how we built this at CloudFronts on Azure and Databricks, unifying five business systems and migrating 40+ Power BI reports with zero downtime, and why that foundation is what makes AI use cases practical rather than aspirational. Table of Contents 1 The Hidden Cost of Scattered Data 2 What One Data Platform Really Means 3 Why It Matters to the Business 4 Solution Architecture Overview 5 Migration Without Disruption 6 From Reporting Platform to AI Foundation 7 AI Use Cases It Unlocks 8 Scattered vs Unified 9 Questions Every Leader Should Ask 10 Where to Start 11 FAQs 12 Conclusion The Hidden Cost of Scattered Data Most growing organizations do not set out to fragment their data. It happens one decision at a time. A new ERP for finance. A CRM for sales. An HR system for people. An accounting tool for a specific entity. Each one solves a real problem – and each one becomes another separate source of data. This project started with our own reporting. Like many growing services organizations, CloudFronts runs its business across several systems: Business Central and Finance & Operations for ERP, Dynamics 365 CRM for clients and engagements, Zoho People for HR, and Zoho Books for accounting. Leadership, finance, HR, and delivery teams rely on more than 40 Power BI reports built on these systems, from management dashboards and HR reports to the Steering Committee report we use to track quarterly meetings with client accounts. Over time, each system had been connected in its own way. Some data moved through Azure Data Factory, some through SQL Database, and some through Azure Functions. Reports connected directly to their sources, with business logic built inside each one, and they sat across separate workspaces for HR, management, and other teams. The idea behind the project was simple: instead of continuing to add pipelines and reports system by system, bring all business data into one trusted platform on Azure Databricks, migrate every existing report onto it without disrupting the people who use them, and build a foundation that is ready for AI in the process. None of the old setup looked broken on any given day. But the costs were real, and they compounded as the business grew. Numbers That Do Not Agree When the same metric is calculated in several reports, small differences appear. Leadership meetings start with reconciling numbers instead of acting on them. Effort That Keeps Growing Every new report means another connection, another copy of the logic, and another refresh to maintain. The effort grows with every report added. Governance That Is Hard to Enforce When reports sit in different workspaces with different owners, controlling who sees what becomes a manual, error-prone exercise. Data That Is Not Ready for What Is Next AI and advanced analytics need clean, connected data. Scattered data means every AI idea starts with a data consolidation project first. “The data was never the problem. The problem was that it lived in five places, and every system handled it differently.” What One Data Platform Really Means One data platform is not just a new database or a bigger data warehouse. It is a change in how the organization treats its data – from something each system and report manages on its own, to a shared business asset managed once, for everyone. Functionally, it comes down to four commitments: 1One place for all business data – Every source system feeds a single platform, so finance, sales, HR, and operations data sit side by side and can be connected. 2One way of processing it – Every source goes through the same standardized stages, so data is cleaned and shaped consistently no matter where it came from. 3One definition of every business rule – Business logic – how revenue is grouped, how a meeting is classified, how an employee is counted – is defined once and reused everywhere. 4One layer of governance – Access is controlled centrally by role, so the right people see the right data without each report owner managing permissions separately. Why It Matters to the Business For leadership, the value of one data platform shows up in outcomes rather than architecture. A Single Source of Truth Every report reads from the same curated data, so the numbers agree across finance, operations, and management reporting. Faster Answers, Less Rework New reports and new data sources are added through configuration, not rebuilt from scratch. Teams spend time on insight instead of plumbing. Stronger Governance by Design Centralized, role-based access means security is enforced by the platform, not by individual report owners. Lower Long-Term Complexity One pipeline and one framework replace a patchwork of tools, reducing maintenance and dependency on specific individuals. Scales With the Business Adding a new system or entity follows the same pattern, so growth does not multiply reporting complexity. A Foundation for AI Clean, connected, governed data is exactly what AI, machine learning, and process mining need to deliver reliable results. Solution Architecture Overview We built our one data platform on Azure and Databricks. Here is how it fits together, from five source systems to a single, governed Power BI workspace. How the Architecture Works Step 1: Connect the Source SystemsBusiness Central, Finance & Operations, Dynamics 365 CRM, Zoho People, and Zoho Books are registered once in a central ingestion list that tells the platform what data to bring in. … Continue reading From Multiple Data Sources to Azure Databricks: A Zero-Downtime Power BI Migration for Future AI Readiness
Share Story :
Predicting the Demand: Automating Demand Forecasting in Dynamics 365 Business Central Using Azure Logic Apps, Data Lake, and Databricks
Predicting the Demand: Automating Demand Forecasting in Dynamics 365 Business Central Using Azure Logic Apps, Data Lake, and Databricks Summary Knowing how many products to keep in warehouses is tough for manufacturers and distributors. During busy seasons, customer orders can jump 10 times higher than normal. When teams rely on manual spreadsheets, they often run out of products or buy too much and run out of storage space. This article explains a simple, automated solution built with Microsoft Dynamics 365 Business Central, Microsoft Azure, and Azure Databricks. Table of Contents The Problem: Swings in Customer Demand The 5-Step Solution Overview How Data is Collected and Cleaned (Medallion) How the Prophet Forecasting Model Works Clear Decision-Making with Forecasted Metrics Real Benefits for Businesses Frequently Asked Questions What You Will Learn Why manual spreadsheets and static inventory numbers fail when demand spikes. How Azure Logic Apps fetches data from Business Central automatically by using scheduled triggers. How raw records are organized into Bronze, Silver, and Gold layers. How the Prophet model forecasts demand using yearly trends and weekly patterns. How dynamic safety stock gives purchasing teams clear replenishment recommendations. 1. The Problem: Swings in Customer Demand Most manufacturers and distributors face a big challenge: customer demand is not steady throughout the year. Some months are quiet, while other months bring huge surges in orders. Season Months Demand Level What Happens Peak Busy Season June ā August 8x ā 10x Surge Huge spike in customer orders. Suppliers take longer to deliver, risking major stockouts. Mid-Year Rush January 3x ā 4x Normal Quick wave of replacement orders and new account setups. Spring Planning March ā May 2x Normal Customers use annual budgets to place advance orders for summer projects. Regular Season Off-Peak Months 1x Baseline Standard, steady daily orders. Why Traditional Methods Fail: Static Rules: Standard ERP rules use fixed inventory numbers all year. These are too small for busy seasons (causing stockouts) and too large for slow seasons (wasting money). Longer Supplier Delays: When everyone orders at once during peak seasons, suppliers take weeks longer to deliver parts. Full Warehouses: Storing large boxes during slow months takes up valuable warehouse space and ties up cash. Manual Spreadsheet Errors: Planning teams spend hours copying and pasting data into Excel spreadsheets without automated forecasting tools. “You don’t need to replace your ERP system. By adding automated cloud forecasting with Azure and Databricks to Dynamics 365 Business Central, past sales history turns into clear, actionable purchasing foresight.” 2. The 5-Step Solution Overview To solve this, we created an automated pipeline that connects daily ERP transactions to cloud forecasting and delivers clear inventory planning targets. How the Automated Flow Works 1 Dynamics 365 Business Central Holds daily sales, purchases, items, and warehouse records. ↓ 2 Azure Logic Apps (Scheduled Ingestion) Fetches data from Business Central automatically by using scheduled triggers without slowing down the ERP system. ↓ 3 Azure Data Lake (Cloud Storage) Stores all historical files securely in one central place. ↓ 4 Azure Databricks (Prophet Model) Cleans the data, runs Prophet forecasting models, and calculates the forecasted buffer stock needed for every item. ↓ 5 Visual Reports in Power BI Forecasted demand and recommended safety stock are displayed in Power BI reports for clear decision-making. 3. How Data is Cleaned & Organized (Bronze, Silver, Gold) In Azure Databricks, data moves through three simple stages known as the Medallion Architecture: Bronze Layer Raw Data Stores exact copies of daily files directly from Business Central (sales, purchases, items, warehouses). Keeps a complete, untouched history so nothing is ever lost. Silver Layer Cleaned Data Fixes missing dates, removes duplicates, and standardizes item numbers across all warehouses. Separates real customer orders from internal warehouse transfers. Gold Layer Forecasting Results Combines daily sales into clear trends and calculates forecasted stock targets for each product. Ready to feed interactive Power BI reports for planners and stakeholders. 4. How the Prophet Forecasting Model Works The Prophet forecasting model analyzes four key factors from past sales: The 3 Things the Model Learns: Overall Growth: Is customer demand growing year over year? Yearly Seasons: Which months have huge order spikes, and which months are quiet? Weekly Patterns: Do customers place most orders on weekdays compared to weekends? By combining these patterns, the system calculates the recommended safety stock for every item and warehouse: Forecasted Safety Stock: The recommended buffer quantity to keep on hand to protect against unexpected surges or supplier delivery delays. 5. Clear Decision-Making with Forecasted Metrics Instead of relying on guesswork in disconnected spreadsheets, supply chain planners have clear, data-driven targets calculated by Azure Databricks. These forecasted metrics give purchasing and warehouse managers actionable recommendations: Projected Demand: Forward-looking estimates of how many units customers will need in upcoming months. Early Order Timing: Clear signals on when to order from suppliers before peak seasons begin. Warehouse Stock Balancing: Guidance on how much inventory to position across regional warehouse hubs. 6. Real Benefits for Manufacturers Order 6ā8 Weeks Ahead Purchasing teams get early warnings before big busy seasons, allowing them to book orders before supplier queues fill up. Balanced Warehouses Items are placed in the right regional warehouses closest to where customers will buy them. More Warehouse Space Bulky products arrive only when needed, keeping aisles clear and reducing expensive storage costs. Data-Driven Planning No more spending days building complicated formulas in Excel. Machine learning provides reliable demand curves and inventory targets. 7. Frequently Asked Questions (FAQ) 1 Will this slow down Business Central for daily users? No. Data is copied automatically during quiet nighttime hours into Azure. All calculations happen in the cloud, so Business Central stays fast and responsive for everyday business. 2 Why use the Prophet model instead of standard ERP reorder rules? Standard ERP rules use one fixed number for the entire year. The Prophet model automatically adapts to upcoming seasons, supplier lead times, and sales trends. 3 How do planning teams use these calculated metrics? Planning and purchasing teams access these forecasted metrics directly through interactive Power … Continue reading Predicting the Demand: Automating Demand Forecasting in Dynamics 365 Business Central Using Azure Logic Apps, Data Lake, and Databricks
Share Story :
From ERP Data to Process Mining Insights: Building an Automated Pipeline for Real-Time Process Visibility
Summary Clean ERP data sitting in a data lake doesn’t answer the question every operations leader eventually asks: where exactly is our process breaking down? We built an automated pipeline that connects a client-facing web portal, Azure Table Storage, and Azure Databricks to a leading process mining platform, turning validated ERP data into a living view of how work actually flows. The pipeline is fully status-driven: every record is tracked from submission through processing to completion, with no manual exports or spreadsheet hand-offs. Purchase order data is modeled through a medallion architecture and delivered to the process mining platform, where AI-driven analysis automatically surfaces bottlenecks and deviations from the expected process. Business impact: process owners moved from static, after-the-fact reporting to a near real-time, evidence-based view of process performance. Table of Contents 01 About the Customer 05 The Six-Step Pipeline 02 The Challenge 06 Architecture Overview 03 The Solution 07 Business Impact 04 AI-Driven Process Mining 08 FAQs About the Customer Customer Spotlight A Leading Digital Transformation Partner ā Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Standardized, clean data answers “what happened.” It rarely answers “why is this taking so long” or “where exactly is this process breaking down.” The business kept running into the same limitations: 1Why do purchase orders take longer to close in some regions than others? 2Which approval step is quietly adding the most delay to the process? 3How do we get validated ERP data into a process analysis tool without manual exports every time? 4How do we know, at any point in time, what has been processed, what’s pending, and what failed? 5Can this insight be generated automatically, instead of requiring a manual investigation every quarter? The Solution We extended the existing Databricks-based data platform with an automated, status-driven delivery layer connecting a client web portal, Azure Table Storage, Azure Databricks, and a leading process mining platform, orchestrated end-to-end with minimal manual intervention. Status-Driven Orchestration Every record carries a live status, from initial submission through sync completion, tracked in Azure Table Storage. Automated Bulk Processing Azure Logic Apps trigger the pipeline through APIs, so batches of records are processed without manual intervention. Reusable Databricks Framework The same medallion pipeline used for data standardization models Purchase Order data for process mining. AI-Driven Process Analysis The process mining platform’s AI reconstructs the real, as-executed process and highlights bottlenecks automatically. The Six-Step Pipeline Here’s how a single record moves from submission to a fully synced, process-mining-ready state: ⚙ Client Web PortalEnd-to-end data pipeline · Azure + Databricks 6 steps 🌐 1) Website Input The user submits data via the client web portal, a form or API request initiates the pipeline. ↓ 🗃 2) Azure Table Sync Incoming data is written and synced into Azure Table Storage. ↓ 📁 3) Status Filter Records from Azure Table are filtered where status matches: ✓ Perfect🕑 Queue ↓ ⚡ 4) Databricks Pipeline The framework is executed through the Databricks pipeline, processing all filtered records in batch. ↓ 🔄 5) Azure Table Update Once the Databricks sync completes, status is updated in Azure Table: Queue→✓ Synced ↓ 📊 6) UI Reflection Synced data is reflected back to the client web portal UI for the end user. Architecture Overview Once records reach the “Synced” state, the same medallion architecture used for data standardization models Purchase Order Details and Purchase Order Lines and delivers them into the process mining platform: ERPExtracts Row-header files → Bronze Raw landing → Silver Cleansed & standardized → Gold Business-ready models → DeltaLake Parquet delivery → ProcessMining AI-driven analysis Because the framework is configuration-driven, the same architecture can extend to additional ERP data lake sources, SFTP feeds, or other cloud storage without a redesign. AI-Driven Process Mining Analysis With Purchase Order Details and Purchase Order Lines modeled and delivered on a reliable, automated cadence, the process mining platform’s AI reconstructs the real, as-executed purchase order process directly from the underlying event data. Instead of relying on assumptions about how the process should work, process owners see how it actually works: where orders stall, which approval paths deviate from the intended flow, and where cycle time is quietly being lost. “A purchase order may look fine on paper, but the process data tells you exactly where it got stuck, and that gap surfaces automatically.” Business Impact Before After Manual exports required to analyze process performance Fully automated, status-driven pipeline from intake to process mining No visibility into where a record stood in processing Live status tracking from submission through sync completion Process bottlenecks discovered through manual investigation AI-driven analysis surfaces deviations and delays automatically Static, after-the-fact process reporting Near real-time, evidence-based process visibility One-off integration effort per process area Reusable framework, extendable to other business processes Frequently Asked Questions Does this require a specific process mining platform? No. The pipeline delivers modeled, business-ready data through Delta Lake and Parquet, which can be connected to most modern process mining platforms. How often is data refreshed in the process mining platform? The pipeline is designed for batch processing on a defined schedule, and can be tuned toward near real-time delivery depending on business needs and source system constraints. Can this be extended beyond Purchase Order data? Yes. Because the framework is configuration-driven, the same approach can extend to other process areas such as order-to-cash or procure-to-pay. What happens if a record fails validation? Records that don’t meet the status criteria simply remain in a pending state and are not passed downstream, so failures are visible and traceable rather than silently dropped. Conclusion Clean data is the foundation, but process visibility is where the business … Continue reading From ERP Data to Process Mining Insights: Building an Automated Pipeline for Real-Time Process Visibility
Share Story :
How a Self-Service Data Portal Solved Multi-Language and Domain Value Chaos in ERP Data
Summary Enterprises running large, multi-country ERP systems often extract data that is technically complete but practically unusable, split across duplicate language columns and encoded with undocumented numeric values. We built a self-service data platform on Azure so that business users, not just data engineers, could define, validate, and process ERP extracts without writing a single line of code. The solution resolves two of the most common ERP data problems: a single field like “Item Description” spread across nine language-specific columns, and reference fields like “Order Status” stored only as numeric codes. A custom web portal puts business users in control of table specifications, validation rules, and processing status, while Azure Databricks and Delta Lake quietly do the heavy lifting behind the scenes. Business impact: dozens of ERP tables moved from raw, multi-language, code-heavy extracts to a single, trusted, human-readable data layer, without adding headcount to the data engineering team. Table of Contents 01 About the Customer 05 Self-Service Data Onboarding 02 The Challenge 06 Medallion Architecture 03 The Solution 07 Business Impact 08 FAQs 09 Conclusion About the Customer Customer Spotlight A Leading Digital Transformation Partner ā Europe Our customer is a leading enterprise headquartered in Europe, operating across diverse manufacturing and supply chain divisions. Having already standardized their ERP data through a medallion architecture on Databricks, leadership wanted to go a step further: not only manage ERP data at scale, but also connect it seamlessly into process mining tools to uncover how core processes truly perform in practice. The focus was on gaining operational clarity into workflows such as purchase order management, invoice handling, and procurement cycles. The Challenge Most organizations extracting data from a large ERP system successfully get the data out. The problem isn’t extraction, it’s making that data mean something the moment it lands. Business and IT teams found themselves asking the same questions on repeat: 1Why does the same field appear nine times, with a different value in each column? 2What does “Order Status= 3” actually mean, and who is the source of truth for that mapping? 3How much manual translation and lookup work happens before a single report can be trusted? 4Can business users resolve these issues themselves, without waiting weeks on an IT backlog? 5How do we scale this across dozens of tables without writing dozens of one-off scripts? Two problems came up again and again, and both are far more common across ERP implementations than most leadership teams realize. Multi-Language Columns Because the ERP system was configured for every Order Status the business operates in, a single logical field such as “Item Description” existed as up to nine separate columns, one per language: English, French, German, Spanish, and more. Reports built directly on top of the raw extract had no reliable way of knowing which column to use for which record. In practice, this meant a plant manager in France could open a report and see item names in German, while a sales report for the Spanish market silently pulled blank fields because the Spanish-language column hadn’t been populated for that record. The data was all there; it just wasn’t usable without someone manually deciding, table by table, which language column to trust. Undocumented Domain Values Reference fields like Country, Currency, and Order Status were stored as raw numeric codes rather than readable labels, for example Order Status: 1 = Completed , 2 = In Progress, 3 = Shipped. These mappings lived inside ERP configuration screens, not in the extracted data itself. That meant every downstream report, dashboard, or spreadsheet needed its own copy of the same lookup table, manually kept in sync. When a code changed or a new Order Status was added in the ERP, there was no guarantee every report using it would be updated at the same time, which meant leadership could be looking at the performance chart that was quietly wrong. The Solution Rather than writing custom transformation logic for every table (a solution that ages badly the moment a new table or region gets added), we designed a configuration-driven pipeline built on Azure Databricks, fronted by a self-service web application that puts control directly in the hands of business and functional users. Self-Service Web Portal Business users upload table specifications, review validation results, and queue tables for processing, entirely through a browser. Medallion Architecture Azure Databricks and Delta Lake refine raw extracts through Bronze, Silver, and Gold layers, without table-specific code. Automated Language Resolution Multi-language columns are detected and normalized automatically based on the specification, not hardcoded per table. Centralized Domain Mapping Numeric and coded reference values are resolved against a single, maintained lookup layer instead of scattered spreadsheets. Self-Service Data Onboarding: No Databricks Knowledge Required The centerpiece of the solution is a custom web application that lets a business or functional analyst, not a Databricks engineer, onboard a new ERP table from start to finish. Here’s what that looks like in practice: A business user uploads an Excel-based table specification defining the expected columns, data types, which fields are multi-language, and which fields are domain-coded and how to decode them. The portal validates the specification instantly, flagging missing mandatory columns or mismatches before any data is processed, so problems are caught at the source rather than three reports downstream. Once validation passes, the same user queues the table for processing with a single click. No notebook to open, no cluster to configure, no code to write or review. Behind the scenes, that specification feeds a generic, reusable Databricks framework that already knows how to apply the correct language resolution and domain-value decoding rules, so engineering effort doesn’t scale linearly with the number of tables. In effect, the portal turns “add a new ERP table to the analytics environment” from a data engineering request into a form a finance or operations analyst can complete in minutes, while still enforcing the same rigor and consistency a hand-built pipeline would require. Medallion Architecture on Databricks Once a table is queued through the portal, Azure Databricks takes over: Bronze: Raw ERP extracts are landed as-is, preserving … Continue reading How a Self-Service Data Portal Solved Multi-Language and Domain Value Chaos in ERP Data
Share Story :
Overcoming Zoho API Limitations in Payroll Automation for a Global Hardware Manufacturer
Summary This blog highlights how Azure Logic Apps was used to overcome a critical API limitation encountered during the integration of Zoho People with FNO for payroll management. During the implementation for a global manufacturing hardware enterprise, we discovered that Zoho’s API allows a maximum of 200 records to be fetched in a single request. While this limitation may not impact smaller organizations, it creates significant challenges for enterprises managing large employee datasets. To address this issue, a scalable Azure Logic Apps solution was developed that dynamically retrieves records in batches, consolidates the results, and returns a complete dataset for downstream processing. This blog explains: Table of Contents 1. Customer Scenario During the implementation of a payroll integration between Zoho People and FNO, employee master data needed to be synchronized automatically to support payroll processing. The organization maintained a large workforce within Zoho People, and payroll operations depended on accurate employee data being transferred to downstream systems. As the integration design progressed, a significant limitation was identified within Zoho’s API framework. The API could return a maximum of 200 records per request. For organizations with hundreds or thousands of employees, this restriction created a challenge in retrieving complete employee datasets efficiently. 2. Business Challenge The integration required access to the full employee dataset from Zoho People. However, the following challenges emerged: Limited API Response Size Zoho’s API only returns 200 records per request. Large Employee Dataset The organization maintained significantly more than 200 employee records. Manual Pagination Not Feasible Static API calls would require manual intervention or complex custom development. Scalability Concerns As employee counts continued to grow, the solution needed to support future expansion without requiring redesign. The objective was to create a scalable and automated mechanism capable of retrieving all employee records regardless of volume. 3. Integration Architecture The solution architecture follows a simple but highly scalable pattern. Process Flow 4. Configuration Steps Step 1: Add HTTP Trigger Step 2: Initialize Variables Step 3: Do Until Loop Step 4: HTTP Request Action Step 5: Output Variable Step 6: Compose Variable Step 7: Append to Array Variable Step 8: Set Variable Step 8: Increment Variable Step 9: Add Response Trigger 5. Why Azure Logic Apps? Azure Logic Apps was instrumental in creating a flexible and efficient solution. Key capabilities that made Logic Apps the ideal choice included: Dynamic Variable Management Allows runtime manipulation of counters and arrays. Scalable Workflow Execution Supports large datasets without requiring custom application development. Native API Integration Provides seamless connectivity with REST-based services. Low-Code Development Accelerates implementation and simplifies maintenance. Enterprise Reliability Offers monitoring, logging, and error-handling capabilities required for production environments. 6. Outcome The final solution successfully overcame Zoho’s API record limitation. The Logic App automatically: This approach ensured the success of the Zoho-FNO integration while maintaining scalability for future business growth. 7. Business Impact 1] Fully Automated Data Retrieval Employee data is retrieved without manual intervention. 2] Improved Scalability The solution can support organizations with thousands of employee records. 3] Reduced Development Complexity Logic Apps eliminated the need for extensive custom coding. 4] Faster Integration Processing Data retrieval occurs efficiently through automated pagination. 5] Improved Reliability Built-in monitoring and error handling improve operational stability. 6] Future-Proof Architecture The solution continues to perform effectively as employee counts grow. To conclude, Integration projects often reveal platform-specific limitations that require creative problem-solving. In this implementation, Zoho’s 200-record API limitation had the potential to impact payroll synchronization for a growing workforce. By leveraging Azure Logic Apps, we developed a scalable and automated solution capable of dynamically retrieving and consolidating employee data regardless of record volume. The solution not only resolved the immediate challenge but also established a reliable and future-ready integration framework capable of supporting continued organizational growth. For organizations facing similar API limitations, Azure Logic Apps provides a powerful platform for building scalable, low-code integration solutions that simplify complex data processing requirements.
Share Story :
Payroll Transformation for a Global Hardware Manufacturer Using Zoho People and Finance & Operations
As businesses scale, payroll complexity grows bringing challenges around employee data, attendance, compensation structures, and compliance. Manual processes not only consume valuable time but also increase the risk of costly errors. The Zoho People-FNO integration transforms payroll into a streamlined, automated process, ensuring accurate salary calculations, seamless data synchronization, and complete transparency across HR and finance operations. We recently implemented this solution for a global manufacturing hardware enterprise, enabling them to automate payroll workflows, eliminate manual data reconciliation, improve payroll accuracy, and reduce administrative overhead. The integration provided a scalable foundation for managing a growing workforce while maintaining compliance and enhancing the employee experience through faster, more transparent payroll processing. For organizations focused on operational efficiency and sustainable growth, this integration delivers measurable business value from day one. Understanding the Architecture of Zoho and FNO Integration The integration between Zoho People and FNO involves a clear, structured workflow. Below is an overview of the steps involved: This architecture ensures a smooth flow of data between Zoho and FNO, simplifying payroll management for businesses of all sizes. Key Advantages of Zoho and FNO Integration The integration between Zoho People and FNO streamlines payroll management, providing several key benefits: Effortless Payroll Management for Growing Businesses To conclude, efficient payroll management is essential for any growing business. By integrating Zoho People with FNO, businesses can automate payroll processes, ensure accurate calculations, and provide employees with easy access to their payslips. The seamless data flow, real-time updates, and reduced manual intervention significantly improve operational efficiency and transparency. If youāre ready to optimize your payroll system, now is the time to take action. Embrace the Zoho and FNO integration to simplify your processes, reduce errors, and create a transparent payroll system that benefits both your employees and your organization. Contact us today to learn how this integration can simplify your payroll management process. Reach out at transform@cloudfronts.com.
Share Story :
From Raw Data to Insights: ETL Best Practices with Azure Databricks
Organizations today generate massive volumes of raw data from multiple sources such as ERP systems, CRMs, APIs, logs, and IoT devices. However, raw data by itself holds little value unless it is properly processed, transformed, and optimized for analytics. In our data engineering journey, we faced challenges in building scalable and maintainable ETL pipelines that could handle growing data volumes while still delivering reliable insights. Azure Databricks helped us bridge the gap between raw data and business-ready insights. In this blog, weāll walk through ETL best practices using Azure Databricks and how they helped us build efficient, production-grade data pipelines. Why ETL Best Practices Matter When working with large-scale data pipelines: – Raw data arrives in different formats and structures– Poorly designed ETL jobs lead to performance bottlenecks– Debugging and maintaining pipelines becomes difficult– Data quality issues propagate to downstream reports Key challenges we faced: – Tight coupling between ingestion and transformation– Reprocessing large datasets due to small logic changes– Lack of standardization across pipelines– Slow query performance on analytical layers Solution Architecture Overview Key Components: – Azure Data Lake Storage Gen2– Azure Databricks– Delta Lake– Power BI / Analytics Tools ETL Flow: – Ingest raw data from source systems into the Raw (Bronze) layer– Clean, validate, and standardize data in the Processed (Silver) layer– Apply business logic and aggregations in the Curated (Gold) layer– Expose curated datasets to reporting and analytics tools Step-by-Step ETL Best Practices with Azure Databricks Step 1: Separate Data into Layers (Bronze, Silver, Gold) – Bronze Layer: Store raw data exactly as received– Silver Layer: Apply cleansing, deduplication, and schema enforcement– Gold Layer: Create business-ready datasets and aggregations This separation ensures reusability and prevents unnecessary reprocessing. Step 2: Use Delta Lake for Reliability – Store tables in Delta format– Enable schema enforcement and schema evolution– Leverage time travel for data recovery and debugging Step 3: Build Incremental Pipelines – Process only new or changed data using watermarking– Avoid full reloads unless absolutely required– Design pipelines to safely re-run without duplications Step 4: Parameterize and Modularize Code – Use notebook parameters for environment-specific values– Create reusable functions for common transformations– Avoid hardcoding paths, table names, or business rules Step 5: Optimize Performance Early – Use partitioning based on query patterns– Apply Z-ORDER on frequently filtered columns– Cache datasets selectively for heavy transformations Step 6: Implement Data Quality Checks – Validate nulls, ranges, and duplicate records– Log rejected or invalid records separately– Fail pipelines early when critical checks fail Benefits of Following These ETL Best Practices – Scalability: Easily handle growing data volumes– Reliability: ACID-compliant pipelines with Delta Lake– Maintainability: Modular and reusable code structure– Performance: Faster queries and optimized storage– Cost Efficiency: Reduced compute usage through incremental processing Conclusion Transforming raw data into meaningful insights requires more than just moving data from one place to another. By following ETL best practices with Azure Databricks, we were able to build robust, scalable, and high-performing data pipelines that deliver reliable insights to the business. If your Databricks pipelines are becoming complex, slow, or difficult to maintain, it might be time to revisit your ETL design. Start applying these best practices today and turn your raw data into insights that truly drive decision-making. I hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
How to Build an Incremental Data Pipeline with Azure Logic Apps
Why Incremental Loads Matter When integrating data from external systems, whether it’s a CRM, an ERP like Business Central, or an HR platform like Zoho People, pulling all data every time is expensive, slow, and unnecessary. The smarter approach is to track what has changed since the last successful run and fetch only that delta. This is the core idea behind an incremental data pipeline: identify a timestamp or sequence field in your source system, persist the last-known watermark, and use it as a filter on your next API call. Azure Logic Apps, paired with Azure Table Storage as a lightweight checkpoint store, gives you everything you need to implement this pattern without managing any infrastructure. Architecture Overview Instead of one large workflow doing everything, we separate responsibilities. One Logic App handles scheduling and orchestration. Another handles actual data extraction. Core components: 3. Metadata Design (Azure Table) Instead of hardcoding entity names and fields inside Logic Apps, we define them in Azure Table Storage. Example structure: PartitionKey RowKey IncrementalField displayName entity businesscentral 1 systemCreatedAt Vendor Ledger Entry vendorLedgerEntries zohopeople 1 modifiedtime Leave leave Briefly, this table answers three questions: – What entity should be extracted?– Which column defines incremental logic?– What was the last successful checkpoint? When you want to onboard a new entity, you add a row. No redesign needed. 4. Logic App 1 ā Scheduler Trigger: Recurrence (for example, every 15 minutes) Steps: This Logic App should not call APIs directly. Its only job is orchestration. Keep it light. 5. Logic App 2 ā Incremental Processor Trigger: HTTP (called from Logic App 1) Functional steps: Example: This is where the real work happens. 6. Checkpoint Strategy Each entity must maintain: – LastSuccessfulRunTime– Status– LastRecordTimestamp After successful extraction: Checkpoint = max(modifiedOn) from extracted data. This ensures: Checkpoint management is the backbone of incremental loading. If this fails, everything fails. This pattern gives you a production-grade incremental data pipeline entirely within Azure’s managed services. By centralizing entity configuration and watermarks in Azure Table Storage, you create a data-driven pipeline where adding a new integration is as simple as inserting a row ā no code deployment required. The two-Logic-App architecture cleanly separates orchestration from execution, enables parallel processing, and ensures your pipeline is resilient to failures through checkpoint-based watermark management. Whether you’re pulling from Business Central, Zoho People, or any REST API that exposes a timestamp field, this architecture scales gracefully with your data needs. Explore the case study below to learn how Logic Apps were implemented to solve key business challenges: Ready to deploy AIS to seamlessly connect systems and improve operational cost and efficiency? Get in touch with CloudFronts at transform@cloudfronts.com.