How to copy data from rest API to Azure data lake via Azure Data Factory

Posted On August 11, 2020 by Sandip Patel Posted in 

In this blog I am going to explain you how to copy data from rest API to Azure data lake using Azure Data Factory.

Step 1: Create an azure storage account via azure portal using your credentials.

A screenshot of a cell phone

Description automatically generated

Step 2: Once your storage creation done, create a container called “restapidata” that will going to use to store rest API data.

A screenshot of a cell phone

Description automatically generated

Step 3: If you have rest API endpoint then its fine otherwise search dummy rest api example in google and copy one of the URL. Here I have taken employee endpoint (http://dummy.restapiexample.com/api/v1/employees)

Step 4: Create an Azure Data Factory service in azure portal and create a pipeline.

A screenshot of a cell phone

Description automatically generated

Step 5: Create a link service for rest API.

A screenshot of a cell phone

Description automatically generated

Step 6: Create a link service for Azure data storage.

A screenshot of a social media post

Description automatically generated

Step 7: Create a dataset for rest API.

A screenshot of a cell phone

Description automatically generated

Step 8: Create a dataset for Azure data lake storage.

Step 9: Drag a copy activity in the pipeline and set a general property.

A screenshot of a cell phone

Description automatically generated

Step 10: Set a source property of copy activity. You can also click on preview data to check whether data is populated or not.

A screenshot of a social media post

Description automatically generated

Step 11: Set a sink property of copy activity.

A screenshot of a social media post

Description automatically generated

Step 11: Set a mapping property of copy activity.

A screenshot of a cell phone

Description automatically generated

Step 12: Debug the pipeline and check the output.

A screenshot of a cell phone

Description automatically generated

Also click on details to check record proceed.

A screenshot of a cell phone

Description automatically generated

I hope this will help you.


Share Story :