Filtering duplicate records in SSIS

Introduction:

In this blog, I will demonstrate how to remove or filter duplicate records in SSIS using Sort Tranformation.

Steps:

First, configure you OLE DB as your Source using the OLE DB Source as shown below.

Bring the Sort Transformation and configure it as shown below. Check the box at the bottom of the Window that says ‘Remove rows with duplicate sort values’.

Now you can connect to your destination by dragging the required block and connecting the Sort Transformation to the Destination block.

For blog purposes, I have connected it to the Multicast block.

Your Data Flow should look like:

Click on start to execute your Data Flow.

Benefits:

Removal of duplicate rows can be useful when trying to avoid Data Redundancy in your System as well as to reduce the memory required.