› BI › Filtering duplicate records in SSIS

Filtering duplicate records in SSIS

Simran Monteiro
Trainee Software Engineer
Published on December 29, 2017
Category: BI

Introduction:

In this blog, I will demonstrate how to remove or filter duplicate records in SSIS using Sort Tranformation.

Steps:

First, configure you OLE DB as your Source using the OLE DB Source as shown below.

Bring the Sort Transformation and configure it as shown below. Check the box at the bottom of the Window that says ‘Remove rows with duplicate sort values’.

Now you can connect to your destination by dragging the required block and connecting the Sort Transformation to the Destination block.

For blog purposes, I have connected it to the Multicast block.

Your Data Flow should look like:

Click on start to execute your Data Flow.

Benefits:

Removal of duplicate rows can be useful when trying to avoid Data Redundancy in your System as well as to reduce the memory required.

 



Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.

Looking for a demo?

We will get back to you very soon!
We promise.

  • This field is for validation purposes and should be left unchanged.