Category Archives: Blog
From Raw to Reliable: Cleaning Data at Scale with Azure Databricks
Are you struggling with messy spreadsheets full of duplicates, missing values, and inconsistent records? Youāre not alone. Data professionals spend nearly 80% of their time cleaning and preparing data before any real analysis begins. The truth is simple: without clean data, business reports are unreliable, AI models fail, and decision-making slows down. In this blog, weāll show you how Azure Databricks makes data cleaning easier, faster, and scalableāturning raw inputs into reliable insights with just a few lines of code. Why Clean Data Matters For business leaders, whether youāre a Team Lead, CTO, or CEO, clean data directly impacts growth: With Azure Databricks, you get a cloud-native, Spark-powered platform that handles big data at scale while integrating seamlessly with Azure Data Lake, Synapse, and Power BI. Practical Example: Cleaning a Sales Dataset in Azure Databricks Imagine you have a raw CSV file in Azure Data Lake with customer sales data: Issues in the data: Solution with PySpark in Databricks: Output after cleaning: CustomerID Name Country Sales 101 Alice USA 500 102 Bob USA 300 103 Unknown UK 450 104 David India 0 With just a few lines of Spark code, the dataset is now ready for reporting, visualization, or machine learning. To conclude, clean data is the foundation of every reliable business insight. With Azure Databricks, you can automate messy, manual processes and create repeatable, scalable pipelines that keep your data reliableāno matter how fast your business grows. ā Start small: try building a simple cleaning pipeline in Azure Databricks today.ā Save time: focus more on insights, less on manual data prep.ā Scale with confidence: as your data grows, Databricks grows with you. š Want to take the next step? Explore how Databricks integrates with Power BI for real-time dashboards or with MLflow for machine learning pipelines. Stay tuned for our next post where weāll cover these use cases in detail. ⨠With Databricks, your journey from raw to reliable data starts today. Contact us today at Transform@cloudfronts.com to get started. To learn more about functionalities of DataBricks and other Azure AI services, please refer to my other blogs from the links given below: ā 1] The Hidden Cost of Bad Data:How Strong Data Management Unlocks Scalable, Accurate AI ā CloudFronts 2] Automating Document Vectorization from SharePoint Using Azure Logic Apps and Azure AI Search ā CloudFronts 3] Using Open AI and Logic Apps to develop a Copilot agent for Elevator Pitches & Lead Qualification ā CloudFronts
Share Story :
Setting Up Unity Catalog in Databricks for Centralized Data Governance
The fastest way to lose control of enterprise data? Managing governance separately across workspaces. Unity Catalog solves this with one centralized layer for security, lineage, and discovery. Data governance is crucial for any organization looking to manage and secure its data assets effectively. Databricksā Unity Catalog is a centralized solution that provides a unified interface for managing access control, auditing, data lineage, and discovery. This blog will guide you through the process of setting up Unity Catalog in your Databricks workspace. What is Unity Catalog? Unity Catalog is Databricks’ answer to centralized data governance. It enables organizations to enforce standards-compliant security policies, apply fine-grained access controls, and visualize data lineage across multiple workspaces. It ensures compliance and promotes efficient data management. Key Features: 1] Standards-Compliant Security: ANSI SQL-based access policies that apply across all workspaces in a region. 2] Fine-Grained Access Control: Support for row- and column-level permissions. 3] Audit Logging: Tracks who accessed what data and when. 4] Data Lineage: Provides visualization of data flow and dependencies. Unity Catalog Object Hierarchy Before diving into the setup, it’s important to understand the hierarchical structure of Unity Catalog: 1] Catalogs: The top-level container (e.g., Production, Development) that represents an organizational unit or environment. 2] Schemas: Logical groupings of tables, views, and AI models within a catalog. 3] Tables and Views: These include managed tables fully governed by Unity Catalog and external tables referencing existing cloud storage. Here is the procedure to setup a Unity Catalog Metastore in association with Azure Storage, as I have done for one of our products (SmartPitch Sales & Marketing Agent) ā 1] First create a storage account with primary service being ā āAzure Blob Storage or Azure Data Lake Storage Gen 2ā; Performance and Redundancy can be chosen based on the requirement for which the DataBricks service is being used.Here for my Mosaic AI Agent, I have used Locally Redundant Storage & Data Lake Gen 2 2] Once the storage account is created, ensure that you have enabled āHierarchical Namespaceā When creating a Unity Catalog metastore with Azure Blob Storage, Hierarchical Namespace (HNS) is required because Unity Catalog needs: a] Folder-like structure to organize catalogs, schemas, and tables. b] Atomic operations (rename, move, delete) on directories and files. c] POSIX-style access controls for fine-grained permissions. d] Faster metadata handling for lineage and governance. HNS turns Azure Blob into ADLS Gen2, which supports these features. 3] Upload any Raw/Unclean files to your metastore folder in the blob storage, which would be required for your use in DataBricks. 4] Create a Unity Catalog Connector in Azure Portal and assign it āStorage Blob Data Contributorā Role . 5] Assign CORS (Cross-Origin Resource Sharing) settings for that storage account. Why this is necessary: In short: Without configuring CORS, Databricks cannot communicate with your storage container to read/write managed tables, schema metadata, or logs. 6] Generate SAS Token 7] Navigate to your Workspace and select āManage Accountā – this should be done from the account admin. 8] Select Catalog tab on the left and then click āCreate Metastoreā 9] Assign a Name, Region (Same as Workspace), The path to the storage account, and the connector id. 10] Once the Metastore is created, assign it to a workspace . 11] Once this is done, the catalogs and the schemas, and tables in within it can be created. How does Unity Catalog differ from Hive Metastore ? Feature Hive Metastore Unity Catalog Scope Workspace or cluster-specific Centralized, spans multiple workspaces and regions Architecture Single metastore tied to Spark/Hive Cloud-native service integrated with Databricks Object Hierarchy Databases ā Tables ā Partitions Catalogs ā Schemas ā Tables/Views/Models Data Assets Supported Tables, views Tables, views, files, ML models, dashboards Security Basic GRANT/DENY at database/table level Fine-grained, ANSI SQLābased (catalog, schema, table, column, row) Lineage Not available Built-in lineage and impact analysis Auditing Limited or external Integrated audit logs across workspaces Storage Management Points to storage locations; no governance Manages external and managed tables with governance Cloud Integration Primarily on cluster storage or external path Secure integration with ADLS Gen2, S3, GCS Permissions Model Spark SQL statements Attribute- and role-based access, unified policies Use Cases Basic metadata store for Spark/Hive workloads Enterprise-wide data governance, sharing, and compliance To conclude, Unity Catalog is the next-generation governance and metadata solution for Databricks, designed to give organizations a single, secure, and scalable way to manage data and AI assets. Unlike the older Hive Metastore, it centralizes control across multiple workspaces, supports fine-grained access policies, delivers built-in lineage and auditing, and integrates seamlessly with cloud storage like Azure Data Lake, S3, or GCS. When setting it up, key steps include: 1] Creating a metastore and linking it to your workspaces. 2] Enabling hierarchical namespace on Azure storage for folder-level security and operations. 3] Configuring CORS to allow Databricks domains to interact with storage. 4] Defining catalogs, schemas, and tables for structured governance. By implementing Unity Catalog, you ensure stronger security, better compliance, and faster data discovery, making your Databricks environment enterprise-ready for analytics and AI. Business Outcomes of Unity Catalog By implementing Unity Catalog, organizations can achieve: Why now? As data volumes and regulatory requirements grow, organizations can no longer rely on fragmented or legacy governance tools. Unity Catalog offers a future-proof foundation for unified data management and AI governanceāessential for any modern data-driven enterprise. At CloudFronts, we help enterprises implement and optimize Unity Catalog within Databricks to ensure secure, compliant, and scalable data governance for enterprise data governance.Book a consultation with our experts to explore how Unity Catalog can simplify compliance and boost productivity for your teams.Contact us today at Transform@cloudfronts.com to get started. To learn more about functionalities of DataBricks and other Azure AI services, please refer to my other blogs from the links given below: – 1] The Hidden Cost of Bad Data:How Strong Data Management Unlocks Scalable, Accurate AI – CloudFronts 2] Automating Document Vectorization from SharePoint Using Azure Logic Apps and Azure AI Search – CloudFronts 3] Using Open AI and Logic Apps to develop a Copilot agent for … Continue reading Setting Up Unity Catalog in Databricks for Centralized Data Governance
Share Story :
Connecting Your MCP Server to Microsoft Copilot Studio – Part 2
In Part 1, we built a simple MCP server in TypeScript that exposed a “getWeather” tool. Now, letās take the next step: connecting our MCP server to Microsoft Copilot Studio so that Copilot agents can call it directly. This section will cover: Step 1 ā Publish Your MCP Server to Azure To make your MCP server accessible to Copilot Studio, youāll need to host it online. There are multiple ways to deploy it ā Azure App Service, Azure Container Apps, or even Azure Functions if you prefer serverless. For example, using Azure App Service: Test using curl to ensure it responds with MCP-compatible JSON: Step 2 ā Create a New Copilot in Copilot Studio Step 3 ā Add Knowledge Sources Optionally, you can enrich your Copilot by adding: This gives your Copilot a baseline knowledge to answer broader questions, while the MCP server will handle specific tasks (like fetching live weather data). Step 4 ā Create a Custom Connector in Dataverse To let Copilot Studio talk to our MCP server, we need a custom connector inside Dataverse/CRM. Step 5 ā Add the Custom Connector to Copilot Studio you’ll see the MCP server in your Tools section of copilot. To test the setup, letās ask Copilot: āWhatās the current weather in Mumbai?ā On the first attempt, Copilot will prompt you to establish a connection. Simply open the Connection Manager, click Connect, and authorize the link to your MCP server. Once connected, Copilot will fetch the live weather details for Mumbai directly from your MCP server. and click retry on the Test window of your copilot. And just like that, your MCP server is live and fully integrated. It can now provide real-time weather updates for any city mentioned in your conversation with Copilot. You can try out different variations of questions or phrasings ā Copilot will intelligently interpret your request, extract the city name, and seamlessly call the MCP server to deliver accurate weather details. Beyond Weather: Business Integrations The same process works for enterprise systems. For example, instead of getWeather, you could expose: By publishing these tools via MCP, your Copilot becomes a true enterprise assistant, capable of pulling structured business data and triggering workflows on demand. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
A Hands-on Guide to Managing Inventory with Microsoft Dynamics 365 Business Central
Inventory is the core of many businesses. Whether youāre selling products, making goods, or managing a supply chain, keeping the right stock at the right time is key. Microsoft Dynamics 365 Business Central helps businesses handle inventory with ease and clarity. 1. Central Item List Item lists are the backbone of inventory management. Business Central lets you create a structured list of all your productsāwhether you buy them, sell them, or just store them. This organized list becomes the single source of truth across all departments. 2. Real-Time Inventory Levels Business Central keeps track of: This helps businesses plan better and fulfill orders faster without confusion. 3. Multi-Location Tracking If you manage inventory in multiple places (like stores, warehouses, or branches), Business Central supports that too. You can: 4. Reorder and Stock Planning With built-in reorder logic, Business Central tells you when to buy and how much to buy. It considers: This reduces guesswork and supports a smooth procurement process. 5. Purchase and Sales Integration When a purchase order is received or a sales order is shipped, inventory updates automatically. This minimizes the need for manual updates and keeps everyone on the same page. 6. Lot and Serial Number Tracking Business Central supports lot numbers and serial numbers. This helps with: 7. Inventory Valuation Methods You can choose how to value your inventory: This supports accurate financial reporting and cost control. 8. Inventory Transfers Do you need to move items from one location to another? Use transfer orders. You can record: 9. Inventory Adjustments Sometimes physical counts donāt match system data. Business Central allows easy stock corrections for: 10. Reports and Insights With built-in reports and dashboards, you can track: These insights will assist you in making well-informed decisions and planning ahead. Why It Matters Good inventory management helps you: Business Central gives you the tools to manage stock simply and efficiently. If you’re using spreadsheets or disconnected tools to manage inventory, now is a good time to explore Business Central. It gives you more control, better insights, and smoother operationsāall in one place. We hope you found this blog useful. If you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
Simplifying File-Based Integrations for Dynamics 365 with Azure Blob and Logic Apps
Integrating external systems with Dynamics 365 often involves exchanging files like CSVs or XMLs between platforms. Traditionally, these integrations require custom code, complex workflows, or manual intervention, which increases maintenance overhead and reduces reliability. Thankfully, leveraging Azure Blob Storage and Logic Apps can streamline file-based integrations, making them more efficient, scalable, and easier to maintain. Why File-Based Integrations Are Still Common While APIs are the preferred method for system integration, file-based methods remain popular in many scenarios: The challenge comes in orchestrating file movement, transforming data, and ensuring it reaches Dynamics 365 reliably. Enter Azure Blob Storage Azure Blob Storage is a cloud-based object storage solution designed for massive scalability. When used in file-based integrations, it acts as a reliable intermediary: Orchestrating with Logic Apps Azure Logic Apps is a low-code platform for building automated workflows. Itās particularly useful for integrating Dynamics 365 with file sources: Real-Time Example: Automating Sales Order Uploads Traditional Approach: Solution Using Azure Blob and Logic Apps: Outcome: Best Practices Benefits To conclude, file-based integrations no longer need to be complicated or error-prone. By leveraging Azure Blob Storage for reliable file handling and Logic Apps for automated workflows, Dynamics 365 integrations become simpler, more maintainable, and scalable. The real-time sales order example shows that businesses can save time, reduce errors, and ensure data flows seamlessly between systems allowing teams to focus on their core operations rather than manual file processing. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
How to Enable Recycle Bin in Dynamics 365 CRM
When working with Dynamics 365 CRM, one common request from users and admins is:āHow do we get a Recycle Bin to recover accidentally deleted records?ā Unlike SharePoint or Windows, Dynamics 365 doesnāt come with a native Recycle Bin. But that doesnāt mean youāre out of luck! There are a few smart ways to implement soft delete or restore capabilities depending on your organizationās needs. In this blog, weāll explore all the available options ā from built-in Power Platform features to custom approaches ā to simulate or enable Recycle Bin-like functionality in Dynamics 365 CRM. Option 1: Use the Built-in Dataverse Recycle Bin (Preview/GA in Some Regions) Microsoft is gradually rolling out a Recycle Bin feature for Dataverse environments. How to Enable: Option 2: Implement a Custom Recycle Bin (Recommended for Full Control) You can also write a bulk delete after 15-30 days to actually clear these records from Dataverse. Option 3: Restore from Environment Backups If a record is permanently deleted, your last line of defence is a full environment restore. Not ideal for frequent recovery, but lifesaving in major accidents. Tips and Tools you can use. If you also want to track who deleted what and when, Auditing might be helpful. You cannot restore deleted records using this. It is useful only for traceability and compliance, not recovery. XrmToolBox Plugins like Recycle Bin Manager simulate soft delete and allow browsing deleted records. While Dynamics 365 CRM doesn’t provide a built-in Recycle Bin like other Microsoft products, there are several reliable ways to implement soft-delete or recovery mechanisms that fit your organizationās needs. Whether you leverage Dataverse’s native capabilities, create a custom status based Recycle Bin, or track deletions through auditing and backups, it’s essential to plan ahead for data protection and user experience. By proactively enabling recovery options, you not only safeguard critical business data but also empower users with confidence and control over their CRM operations. Whatās Your Approach? Have you built your own Recycle Bin experience in Dynamics 365? Share your thoughts or tips in the comments below! We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
Mastering String Functions in Business Central: Practical Examples with AL
When working with Microsoft Dynamics 365 Business Central, string manipulation is an essential part of everyday development. From reversing names to formatting messages, AL provides multiple ways to handle text. In this blog, weāll explore different approaches to string handling through practical examples, including custom logic and built-in AL string functions. Why String Handling Matters in Business Central Strings are everywhereācustomer names, item descriptions, invoice messages, and more. Being able to manipulate these strings efficiently allows developers to: Letās dive into some real-world AL examples where we extend the Customer List page with new actions for string manipulation. Part 1: Custom String Handling Approaches š¹ a) Method 1: Using List of Text We can reverse a string by adding each character into a List of [Text] and then calling .Reverse(). action(CFS_ReverseSelectedCustomerName) { Caption = ‘Reverse Customer Name’; ApplicationArea = All; trigger OnAction() var StringList: List of [Text]; StringLetter: Text; ReversedString: Text; begin ReversedString := ”; foreach StringLetter in Rec.Name do StringList.Add(StringLetter); StringList.Reverse(); foreach StringLetter in StringList do ReversedString += StringLetter; Message(ReversedString); end; } This approach is useful when you want more control over the collection of characters. Output: Method 2: Using Index Manipulation Here, we iterate through the string from end to start and build the reversed string. action(CFS_NewIndex) { Caption = ‘New Index’; ApplicationArea = All; trigger OnAction() var ReversedString: Text; i: Integer; begin for i := StrLen(Rec.Name) downto 1 do ReversedString += CopyStr(Rec.Name, i, 1); Message(ReversedString); end; } A more direct approach, simple and efficient for reversing text. Method 3: Using CopyStr The CopyStr function is perfect for extracting characters one by one. action(CFS_NewText) { Caption = ‘New Text’; ApplicationArea = All; trigger OnAction() var ReversedString: Text; i: Integer; begin for i := StrLen(Rec.Name) downto 1 do ReversedString += CopyStr(Rec.Name, i, 1); Message(ReversedString); end; } Part 2: Built-in AL String Functions Beyond custom logic, AL offers powerful built-in functions for working with text. Letās explore a few. Evaluate Examples The Evaluate function converts text into other datatypes (integer, date, boolean, duration, etc.). action(CFS_EvaluateExamples) { Caption = ‘Evaluate Examples’; ApplicationArea = All; trigger OnAction() var MyInt: Integer; MyDate: Date; MyBool: Boolean; MyDuration: Duration; Value: Text; OkInt: Boolean; OkDate: Boolean; OkBool: Boolean; OkDur: Boolean; begin Value := ‘150’; OkInt := Evaluate(MyInt, Value); Value := ‘2025-09-01’; OkDate := Evaluate(MyDate, Value); Value := ‘TRUE’; OkBool := Evaluate(MyBool, Value); Value := ‘3d 5h 45m’; OkDur := Evaluate(MyDuration, Value); Message( ‘Integer = %1 (Ok: %2)\Date = %3 (Ok: %4)\Boolean = %5 (Ok: %6)\Duration = %7 (Ok: %8)’, MyInt, OkInt, MyDate, OkDate, MyBool, OkBool, MyDuration, OkDur); end; } Super handy when parsing user input or imported text data. String Functions Examples Now letās use some of the most common string functions in AL. action(CFS_StringFunctions) { Caption = ‘String Functions’; ApplicationArea = All; trigger OnAction() var SourceTxt: Text[100]; CopyTxt: Text[50]; DelTxt: Text[50]; Len: Integer; SubstTxt: Text[100]; UpperTxt: Text[50]; LowerTxt: Text[50]; begin SourceTxt := ‘Dynamics 365 Business Central’; CopyTxt := CopyStr(SourceTxt, 1, 8); // “Dynamics” DelTxt := DelStr(SourceTxt, 1, 9); // “365 Business Central” Len := StrLen(SourceTxt); // 29 SubstTxt := StrSubstNo(‘Hello %1, welcome to %2!’, ‘User’, ‘Business Central’); UpperTxt := UpperCase(SourceTxt); // “DYNAMICS 365 BUSINESS CENTRAL” LowerTxt := LowerCase(SourceTxt); // “dynamics 365 business central” Message( ‘Original: %1\CopyStr: %2\DelStr: %3\Length: %4\Substitute: %5\Upper: %6\Lower: %7’, SourceTxt, CopyTxt, DelTxt, Len, SubstTxt, UpperTxt, LowerTxt); end; } These built-in functions save time and make string handling straightforward. To conclude, whether youāre reversing a customerās name, converting text into other data types, or formatting user-friendly messages, ALās string manipulation capabilities are both flexible and powerful. By combining custom logic with built-in functions, you can handle almost any text-related scenario in Business Central. Try experimenting with these functions in your own extensionsāyouāll be surprised how often they come in handy! We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
How to Use Metadata Search in Visual Studio for Dynamics 365 Finance & Operations
When working on big projects, it can be hard to quickly find classes, methods, or properties spread across different libraries. Searching through files one by one takes too much time. This is where metadata search in Visual Studio helps. It lets you explore and navigate types from external libraries (like .NET assemblies or NuGet packages) even if you donāt have the source code. Visual Studio creates a readable view of the compiled metadata, so you can instantly see definitions and details, almost like built-in documentation. References Metadata search – Visual Studio Usage -In Visual Studio, go to Dynamics 365 > Metadata Search OR Ctrl + R, Ctrl + S.-Start typing the name of the element you are looking for.-Results appear as you type.-Double-click a result to go directly to that metadata or code.-You can also right-click to add items to your project. You can filter via the follow criteria: To conclude, Metadata search in Visual Studio is a fast and easy way to find code and metadata in Dynamics 365 F&O projects. Using it saves time, reduces frustration, and helps you understand and work with large projects more efficiently.If you require additional support or have specific questions about your ERP setup, please don’t hesitate to contact us for personalized guidance. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
Why Clients Need Custom Power BI Solutions for Territory-Based Reporting
Data is one of the most valuable assets for modern organizations. But without the right reporting structure, decision-makers struggle to extract meaningful insights. At CloudFronts, we specialize in tailoring Power BI to meet specific client needs. In this blog, weāll share how we customized a territory-based account analysis report for a clientās sales teamāand why such customizations deliver real business value. Problem Statement The clientās leadership team faced three challenges with their existing reports: 1. Lack of clarity: Territories on the map looked identical, creating confusion. 2. No drill-down path: Managers could not move easily from high-level territory views to account-level details. 3. Data security concerns: All managers could see all account data, raising confidentiality issues. These gaps reduced adoption of the reports and slowed decision-making. Solution Approach We delivered a tailored Power BI solution with the following enhancements: 1. High-Impact Visuals with Conditional Formatting Each territory was assigned a unique color on the map, instantly improving readability. 2. Structured Multi-Page Navigation – Page 1: Territory Map ā for leadership to view performance at a glance. – Page 2: Drill-Through ā for Territory Managers to analyze accounts in detail. – Page 3: Tabular Data ā for operations teams to validate and export account data. 3. Data Security with Row-Level Security (RLS) Each Territory Manager could only view accounts from their assigned states, ensuring sensitive client data was protected. 4. User Adoption Focus By mirroring the real workflow of Territory Managers, adoption rates significantly increased. Key Learnings – Custom visuals drive clarity: Unique formatting makes reports intuitive. – Security builds trust: Clients are reassured when their data is properly protected with RLS. – Role-based design improves efficiency: Reports that align with how teams work reduce training needs and accelerate insights. – Better adoption leads to ROI: Customized reports quickly become part of daily decision-making, maximizing the value of Power BI investments. To conclude, for clients, Power BI customizations are not just a ānice-to-haveāāthey are a business necessity. By aligning reports with organizational structures, ensuring secure access, and simplifying navigation, businesses gain faster insights, stronger adoption, and higher ROI from their BI investments. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.
Share Story :
Automating Document Vectorization from SharePoint Using Azure Logic Apps and Azure AI Search
In modern enterprises, documents stored across platforms like SharePoint often remain underutilized due to the lack of intelligent search capabilities. What if your organization could automatically extract meaning from those documentsāturning them into searchable vectors for advanced retrieval systems? Thatās exactly what weāve achieved by integrating Azure Logic Apps with Azure AI Search. Workflow Overview Whenever a user uploads a file to a designated SharePoint folder, a scheduled Azure Logic App is triggered to: Once stored, a scheduled Azure Cognitive Search Indexer kicks in. This indexer: Technologies / resources used: –-> SharePoint: A common document repository for enterprise users, ideal for collaborative uploads. -> Azure Logic Apps: Provides low-code automation to monitor SharePoint for changes and sync files to Blob Storage. It ensures a reliable, scheduled trigger mechanism with minimal overhead. -> Blob Storage: Serves as the staging ground where documents are centrally stored for indexingācheaper and more scalable than relying solely on SharePoint connectors. -> Azure AI Search (Cognitive Search): The intelligence layer that runs a skillset pipeline to extract, transform, and vectorize the content, enabling semantic search, multimodal RAG (Retrieval Augmented Generation), and other AI-enhanced scenarios. Why Not Vectorize Directly from SharePoint? Reference:-1. https://learn.microsoft.com/en-us/azure/search/search-howto-index-sharepoint-online2. https://learn.microsoft.com/en-us/azure/search/search-howto-indexing-azure-blob-storage How to achieve this? Stage 1: – Logic App to sync Sharepoint files to blob Firstly, create a designated Sharepoint directory to upload the required documents for vectorization. Then create the logic app to replicate the files along with itās format and properties to the associated blob storage – 1] Assign the site address and the directory name where the documents are uploaded in Sharepoint ā In the trigger action āWhen an item is created or modifiedā. 2] Assign a recurrence frequency, start time and time zone to check/verify for new documents and keep the blob container updated. 3] Add an action component ā āGet file content using pathā; and dynamically provide the full path (includes file extension), from the trigger 4] Finally, add an action to create blobs in the designated container that would be vectorized ā provide the storage acc. name, directory path, the name of blob (Select to dynamically get the file name with extension for the trigger), blob content (from the get file content action). 5] On successfully saving & running this logic app, either manually or on trigger, the files are replicated in itās exact form to the blob storage. Stage 2 :- Azure AI Search resource to vectorize the files in blob storage In Azure Portal (Home – Microsoft Azure), search for Azure AI Search service, and provide the necessary details, based on your requirement select a pricing tier. Once resource is successfully created, select āImport & vectorize dataā From the 2 options ā RAG and Multimodal RAG Index, select the latter one.RAG combines a retriever (to fetch relevant documents) with a generative language model (to generate answers) using text-only data. Multimodal RAG extends the RAG architecture to include multiple data types such as text, images, tables, PDFs, diagrams, audio, or video. Workflow: Now follow the steps and provide the necessary details for the index creation Enable deletion tracking, to remove the records of deleted documents from the index Provide a document intelligence resource to enable OCR, and to get location metadata for multiple document types. Select image verbalization (to verbalize text in images) or multimodal embedding to vectorize the whole image. Assign the LLM model for generating the embeddings for the text/images provide an image output location, to store images extracted from the files Assign a schedule to refresh the indexer and to keep the search index up to date with new documents. Once successfully created, search keywords in the search explorer of the index, to verify the vectorization, the results are provided based on it’s relevance and score/distance, to the userās search query. Let us test this index in Custom Copilot Agent , by importing this index as an azure ai search knowledge source. On fetching details of certain document specific information, the index is searched for the most appropriate information, and the result is rendered in readable format by generative AI. We hope you found this blog useful, and if you would like to discuss anything, you can reach out to us at transform@cloudfronts.com.