Filtering duplicate records in SSIS | CloudFronts

Filtering duplicate records in SSIS

Posted On December 29, 2017 by Simran Monteiro Posted in  Tagged in ,


In this blog, I will demonstrate how to remove or filter duplicate records in SSIS using Sort Tranformation.


First, configure you OLE DB as your Source using the OLE DB Source as shown below.

Bring the Sort Transformation and configure it as shown below. Check the box at the bottom of the Window that says ‘Remove rows with duplicate sort values’.

Now you can connect to your destination by dragging the required block and connecting the Sort Transformation to the Destination block.

For blog purposes, I have connected it to the Multicast block.

Your Data Flow should look like:

Click on start to execute your Data Flow.


Removal of duplicate rows can be useful when trying to avoid Data Redundancy in your System as well as to reduce the memory required.


Share Story :

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.