In this blog, we will learn how to ingest the Salesforce table data into a dedicated SQL pool using Notebook activity. In part1 we created an Azure synapse analytics workspace, dedicated SQL pool in this we have seen how to create dedicate SQL pool, and in Salesforce data we have written the python script to get data. In this blog, we will learn how to connect a dedicated SQL pool and ingest data into a table step by step. Step 1: Sign to the Azure portal. Open Azure Synapse Analytics and click on Open Synapse Studio to open your existing Notebook. Step 2: Once the Synapse Studio opens click on “Develop” and open your existing Notebook. Step 3: Add the following code to connect your dedicated SQL pool using the “pyodbc” library and write the SQL insert query to load the data into a table. Step 4: Once the script is ready, click on “Add to pipeline” as per the below screenshot. Step 5: Once you click on “New pipeline”, it will automatically create Notebook activity, give the proper pipeline name. Step 6: Debug the pipeline, here is the output of the pipeline. Hope this will help.

In this blog, we will learn how to resolve ModuleNotFoundError: No module named ‘simple salesforce’ in Notebook. Step 1: To upload to your cluster you simply navigate to “Manage”, then choose “Apache Spark Pools”, click the three dots on your Spark cluster that you want to add the package to. Step 2: Once you clicked on Packages, you can see the requirement files option. In this, you have to select the upload option to upload the files. Step 3: There are two options to create resource requirement files(.txt or .yml). Here we will use yml file. A requirement is essentially a file that you upload to the Spark cluster and runs the equivalent of a “Pip install” when the cluster starts for all the packages listed in the file. You add your extra packages here and restart the cluster (or force apply). Upload your requirement file as per the below screenshot. Step 4: Once you select your requirement file, check the “Immediately apply settings change and cancel all active applications” option to force changes to apply. Once the package installs complete, you can re-run your Notebook, it will execute successfully. Hope this will help.

In this blog, we will learn how to get Salesforce data using Notebook via a python script. In part1 we created an Azure synapse analytics workspace. In this, we will create a Notebook and write a python script to get Salesforce data step by step. Step 1: Sign to the Azure portal. Open Azure Synapse Analytics and click on Open Synapse Studio to create a Notebook. Step 2: Once the Synapse Studio opens click on “Develop” and create a new Notebook. Step 3: Provide the suitable name for your Notebook, select a language as python, and attached the apache-spark pool that you have created. Step 4: Before you write a python script to get data from Salesforce. You have to first create a new “Connected App” in your Salesforce portal(prod or sandbox). Go in “Setup”, open the “App Manager”. Then, create a “New Connected App”. Name your application. Tick the box “Enable OAuth Settings”. In “Selected OAuth Scopes”, make all scopes available. Type “http://localhost/” in “Callback URL”. Save. In the end, you should get and note down the “Consumer Key” and the “Consumer Secret”. Using user id, password, consumer key, and secret we can get the Salesforce access token. Step 5: Once you have the above information, write the following python script to get the Salesforce data. To read the data from Salesforce, here I have used the “Simple_Salesforce” python library. Step 6: Here is the output of the script. Hope this will help.