Go through the blog to understand how to integrate a CSV file stored in your ADLS to Azure SQL DB.

Read this blog to understand how one can create logic App using Azure portal to create an API for SQL DB.

Read this blog to understand about Triggers in Logic App. Learn how you can work with Simple Timer Invoke and how to trigger Logic App containing a start Block as ‘When an HTTP request is received’

The BYOD feature for the Dynamics 365 for finance and Operations was release in 2016, it’s provide feature for the customer to bring their own Azure SQL Database, but the drawback through this was Entity Store only accessible for the inbuilt data warehouse only which means it is not accessible to outside D365. The new feature for the Data Lake Gen2 makes the Entity Store get Sync with Data Lake. Following are the steps to setup the environment. Let’s create the Storage Account for the Data Lake Gen2 Click on the Create Resource and search for Storage accounts and go to the Storage Account and click on Add. Choose the Subscription and Resource group, resource group is the container that holds the resource. Here, we are creating the new resource for our Data Lake. Make sure you select the Location same as your Power BI dataset environment and set the other option as shown below. Once storage account gets created, we can access it from the resource group open the storage account Now let’s take a copy of the connection string since it will be required later, the connection string can be access by going to setting under the storage account. Let’s create the Key vault resource to store the secret and create the secret for the connection string. Secret can be created by going to the key vault by going to the Key under the Settings tab, click on the generate the key. We are creating the secret for the connection string that we have copied earlier, set the value for the connection string as the key that we have copied and click on the create. Once we have created the app secret for the connection the next step is to authorize the user and resource to the request. Here we are going to register app for the authorization of the D365FO environment. To register the app, go to the Azure Active Directory and Select the app registration and click on the new registration. Fill the application name of your choice and redirect URI is set to WEB and select the D365 environment URI as shown below. Once we register the app the next step is to grant the API permission as a part of consent process. Grant all the permission that application should require. From Azure Key Vault select the Permission as user_impersonation which provide full access to Azure Key Vault service. The next step is to create client secret and make sure to note down the value of the app generated since we are going to use that in the D365FO environment. Next is to add the D365FO in access policy list and select the Key and Secret permission to Get and List from the drop down. Once we have added D365FO in access policy list the next step is to add the Application ID and Application Secret of Azure Key Vault in Data connection in D365FO environment, which can be access by going to Module > System administration > Setup > System parameters > Data connections tab. Note:- Make sure if data connection tab is not visible to you which means you are missing some configuration setting during environment setup, to make the data connection tab available we need to check the Value of CDSAMaster in SQL Studio Management Studio. Open SSMS go to the AxDW database and check the value in SYSFLIGHTING for CDSAMaster if not present then insert the value in table. SELECT * FROM SYSFLIGHTING  /*To check the Flightname values*/ INSERT INTO SYSFLIGHTING VALUES (‘CDSAMaster’,1,12719367,5637144576,5637144589,1) /*     CDSAMaster FlightName     1 Enabled     12719367 FlightserviceID     5637144576 Partition     5637144589 RecID     1 Recversion */ Before Update After Updating Once Data connections is added fill the Application ID, Application Secret, DNS name and connection secret name as shown below and make sure to enable the Data Lake Integration. Test the Azure Key Vault and Azure Storage. Note:- If you are getting error 401 which means Azure user in which data lake is hosted has no access to D365FO environment in that case you need to import the user in environment and assign role as Administrator. Create Reports using Azure Data Lake Gen2 Once done with above steps the next step is to configure the storage account, following are the requirement for the Power BI. The storage account must be created in the same AAD tenant as your Power BI tenant. The storage account must be created in the same region as your Power BI tenant. The storage account must have the Hierarchical Name Space feature enabled (Make sure to enable this at time of storage account creation) Power BI service must be granted a Reader role on the storage account. You must have a Global Administrator account, this account is required to connect and configure Power BI to store the dataflow definition, and data, in your Azure Data Lake Storage Gen2 account  As we have created storage account previously lets, grant the Reader role to the storage account, in Azure portal Go to the Storage account > Access control > Role Assignments then click on Add role. Once we assign role to the Storage Account the next step is to Assign the directory level permission to the Storage that we have created. Here we are granting Power BI permission to the file system. We need to get the collection of IDs of the Power BI. Which can get by navigating to the AAD > Enterprise Application > All Application, copy the Object IDs of the Power BI Premium, Power BI Service and Power Query Online. For each Power BI Object that we have collected in previous steps grant the below access for each of the object. Once we granted the access to the storage the next step is to connect your Azure Data Lake Storage Gen 2 to Power BI Go to your Power BI service > Click on Admin portal navigate to Dataflow settings then Select the Connect your Azure Data Lake Storage Gen2 button. The following … Continue reading Develop D365 Finance, SCM and Retail Reports using Azure Data Lake Gen2 →

Introduction: In order to move your Pipeline from one environment to other you require your Pipeline to be saved on Github. This Blog will guide you through how to connect your Azure Data Factory Pipeline with your Github account. Step 1: Create a new Azure Data Factory and Tick the Enable GIT checkbox.     Step 2: Create a new Repository in your Github Account.     Step 3: Copy the URL from of the newly created repository from the address bar.     Step 4: Enter the GIT URL, the repository name which you recently created, branch name (usually master) and your root folder (usually same as your repository name) and click on Create.     Step 5: Now you must authorize your GIT Account on ADF. You will be promoted to Login with your Github Account. Do the necessary and click on Authorize Azure Data Factory.     Step 6: Now you must select the Working Branch. We are using the default master branch. Click Save and then you can proceed to create your Pipeline.     Conclusion: Each Time you publish the changes in ADF they are simultaneously stored in the GIT repository’s branch.

In order to copy data from Dynamics AX to On-Premises SQL Server, we need certain prerequisites – Login into https://portal.azure.com/#home Create a Data Factory and Click on Author and Monitor. Go to Author and create the following- Connections (Linked Services) Datasets Pipelines CONNECTIONS         SOURCE Click on New Linked service and create a source connection. Give it a name, Description (optional) and select AutoResolveIntegrationRuntime. We select AutoResolveIntegrationRuntime as our source is cloud-based (Dynamics AX), If it is on-premises, then only we need to install and select Self-hosted Integration runtime. URL: OData Endpoint AAD resource: Same as URL without /data Service Principal ID:  Application Id from App Registrations. Service Principal Key: Value from Certificates and Secrets in App registrations.        SINK In our case, Sink is SQL Server so we enter the SSMS credentials to connect to our on-premises SQL Server. Install the Integration runtime to provide and establish connection of on-premise SQL Server with the cloud. Test the connection. INTERGRATION RUNTIME We can choose between Express-setup and Manual Setup to install the integration runtime. Then go to Dynamics AX, and grant this service principal proper permission to access your Dynamics AX. In order to grant permission, go to Dynamics AX -> System Administration and add your Client ID there. DATASETS Create Dataset for Source and Sink. For the source dataset, select the source connection from the drop down for Linked Service field. In the path field, Select the desired entity. For the sink dataset, In the table field, you can either select a specific table to copy your data or check the auto-create option to make a new table in the sink. Check the Edit option in the Table field and give the desired name of the table to auto create it.  

Introduction We are performing Integration of Accounts from CRM to SQL using ADF Copy activity pipeline. We want to upsert the accounts instead of inserting duplicate records again. Step 1: Auto create the Table named “accounts” in SQL Server during the first Integration run by selecting the Auto create table option.   Step 2: Create a custom data type named “AccountType” using following query.   CREATE TYPE AccountType AS TABLE( accountid uniqueidentifier, transactioncurrencyid uniqueidentifier, address1_city nvarchar(MAX), createdon datetime2(7), accountnumber nvarchar(MAX), name nvarchar(MAX), address1_country nvarchar(MAX), address1_composite nvarchar(MAX), telephone1 nvarchar(MAX), emailaddress1 nvarchar(MAX), websiteurl nvarchar(MAX), primarycontactid uniqueidentifier ) Step 3: Create a Stored Procedure named “spUpsertAccounts”.   CREATE PROCEDURE spUpsertAccounts @account AccountType READONLY AS BEGIN   MERGE dbo.accounts AS target_sqldb USING @account AS source_tblstg ON (target_sqldb.accountid = source_tblstg.accountid) WHEN MATCHED THEN UPDATE SET accountid = source_tblstg.accountid, transactioncurrencyid = source_tblstg.transactioncurrencyid, address1_city = source_tblstg.address1_city, createdon = source_tblstg.createdon, accountnumber = source_tblstg.accountnumber, name = source_tblstg.name, address1_country = source_tblstg.address1_country, address1_composite = source_tblstg.address1_composite, telephone1 = source_tblstg.telephone1, emailaddress1 = source_tblstg.emailaddress1, websiteurl = source_tblstg.websiteurl, primarycontactid = source_tblstg.primarycontactid     WHEN NOT MATCHED THEN INSERT ( accountid, transactioncurrencyid, address1_city, createdon, accountnumber, name, address1_country, address1_composite, telephone1, emailaddress1, websiteurl, primarycontactid ) VALUES ( source_tblstg.accountid, source_tblstg.transactioncurrencyid, source_tblstg.address1_city, source_tblstg.createdon, source_tblstg.accountnumber, source_tblstg.name, source_tblstg.address1_country, source_tblstg.address1_composite, source_tblstg.telephone1, source_tblstg.emailaddress1, source_tblstg.websiteurl, source_tblstg.primarycontactid ); END Step 4: Enter the Stored Procedure Name, Table Type and Table type parameter as shown in the image below:     Step 5: Publish all the changes and debug your Pipeline.  You can Verify the results in SQL Server “accounts” table.  

REST API is an Application Program Interface that uses HTTP request to GET, PUT, POST and DELETE data, it’s an architecture style approach to communicate with third party application. In order to integrate our module with third party applications or in order to pull data from third party application to our database for analysis purpose the REST APIs are useful. For the analysis purpose we can consume REST API with SSIS using third party connector, but the problem with this approach is that, it will be required On Premise server for package deployment and job schedule. The alternate approach is to use serverless CRON expression. Since, mid 2018 the serverless development methodology is ditching the traditional development. In this blog we are going to see how to consume REST API with Serverless CRON expression. Here, we are using Zoho People API for integrating HR modules data to Azure DB. With Zoho People API, you can extract employee’s data and form data in XML or JSON format to develop new applications or integrate with your existing business applications. Zoho People API is independent of programming languages which helps you to develop applications in any programming languages (reference). Implementation Authentication token generation In order to access Zoho People API, Zoho People authentication token is required. The token can be generated using Browser mode and API mode. For the API mode each request has Username or Email and Password needs to include in the POST body Another approach is to register your app with Zoho by going to zoho.com/developerconsole and Add Client ID, Once added it can be used to generate new access_token. We can check the response by use using POST request, Open any REST Client and send request by using any REST API method, here we are send request to get leave data and we are using VS Code REST Client. If the request is valid then the we can get response in following format containing the data requested. 3. As we are getting proper response, the next step is to create CRON expression (Serverless approach) to pull the data from REST API, here we are going to use App services on Azure to create Timer Trigger Function App. The CRON expression is a time-based job schedular. which has six field to define time precision in following format. {second} {minute} {hour} {day} {month} {day-of-week} Each field can have one of the following types of values: Type Example When triggered A specific value “0 5 * * * *” at hh:05:00 where hh is every hour (once an hour) All values (*) “0 * 5 * * *” at 5:mm:00 every day, where mm is every minute of the hour (60 times a day) A range (- operator) “5-7 * * * * *” at hh:mm:05, hh:mm:06, and hh:mm:07 where hh:mm is every minute of every hour (3 times a minute) A set of values (, operator) “5,8,10 * * * * *” at hh:mm:05, hh:mm:08, and hh:mm:10 where hh:mm is every minute of every hour (3 times a minute) An interval value (/ operator) “0 */5 * * * *” at hh:05:00, hh:10:00, hh:15:00, and so on through hh:55:00 where hh is every hour (12 times an hour) 4. The next step is to create function app, for development purpose we are going to use Visual Studio Community 2019/2017. In Visual Studio create a project by selecting File > New > Project Select Visual C# (here we are going to use C# for development you can choose Php, Python or F#) any of your choice. Under the Visual C# select the Azure Functions and click on Next The next step is select a new Azure Function Application, Select the Timer trigger function choose your Azure Function version (v1, v2 or v3). Select Authorization level as Anonymous as we don’t want to include API Key for the function and kept the other settings as it is. After clicking on Create button, it will create the Azure Function App Solution for us, the directory structure is as follow. The next step is to publish the function on Azure, it will deploy your function on IIS and Azure. That we will see in next part of this blog.    

In this blog we’ll see how t publish the and deploy the Azure function on App service. With Continuing with the previous part of this blog Right Click on the solution and click on Publish, it will ask to select the proper subscription, resource group. Select the option relevant to you. Following are the Publishing option available for us. Publish on Azure App Service Azure Virtual Machines File system Local webserver (IIS) Note:- Make sure you have sign in with proper credential and cloud explorer is connected. Once we publish the function, we can check Azure function by going to the https://portal.azure.com/ then search for the Function App. We can see our function app in read only mode, since we have deployed it from visual studio environment. The next step is to create the development logic for the data pull from the Zoho People. Make sure to add all reference libraries required for the development, we can manage the NuGet package manager by going Right clicking on the solution and select manage NuGet package. Make sure to add all required reference to your script file. using System; using Microsoft.Azure.WebJobs; using Microsoft.Extensions.Logging; using Newtonsoft.Json; using System.Data.SqlClient; using System.Net.Http; using System.Net.Http.Headers; using System.Collections.Generic; using System.Linq; As our destination is Azure DB so, we need to create the environment variable to store ADO.NET connection string in settings.json file or by clicking on Azure App Service Settings Then click on Add Setting and then add the setting name like in following diagram we are creating setting for the Azure DB connection. Click on ok and provide the value for the setting. We can get azure connection string by going to the Azure Portal search the DB name and then click on the connection string and copy the ADO.NET connection string, replace the {your_password} with your server password. After adding the environment variable, we can check this in settings.json file The Next step is to implement the code logic and debug the code on local. That we’ll see in final part of this blog.

In this part we’ll see how to Debug the code on local. Once we done with the configuration settings the next step is to call the REST API in our code, here we are using C# for the development. Once we done with the code logic, we can run the code on local by adding the breakpoint The Azure Function app will launch in a local Functions host, and your trigger will be available locally on http://localhost:7071/api/Function1 default listening port is 7071, we can also change the port number by going to the project setting and setting Application arguments to the project settings. host start –pause-on-error –port 7079 We can also launch multiple apps at the same time, we can do this by adding multiple breakpoint at the same time and then start the function without debugging and start another function through debugger and attached the debugger to function app by, Select Debug > Attach to process. Search for fun The greyed-out function is in running state and other is running through debugger. Select the other and click Attach and then we can open the browser and run the both function app different port gets assigned to each function. Publish the App on the Azure and make sure to add the all the parameters on the configuration by going to the Configuration on the Function App.   Note:- For every changes on app we need to build and publish the app.