29 Dec’17

BIFiltering duplicate records in SSIS


In this blog, I will demonstrate how to remove or filter duplicate records in SSIS using Sort Tranformation.


First, configure you OLE DB as your Source using the OLE DB Source as shown below.

Bring the Sort Transformation and configure it as shown below. Check the box at the bottom of the Window that says ‘Remove rows with duplicate sort values’.

Now you can connect to your destination by dragging the required block and connecting the Sort Transformation to the Destination block.

For blog purposes, I have connected it to the Multicast block.

Your Data Flow should look like:

Click on start to execute your Data Flow.


Removal of duplicate rows can be useful when trying to avoid Data Redundancy in your System as well as to reduce the memory required.


Written by

Simran Monteiro

Trainee Software Engineer

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.

Want to streamline your business processes?

  • This field is for validation purposes and should be left unchanged.

Recent Articles

  • Commands to Import .bacpac file to D3FOE SQL Server

    14 May’ 2018

    Introduction: This blog article will explain how to import a .bacpac file to Microsoft SQL Server which is created from ...

    Read more
  • [Solved] – Backspace not working in HTML Web Resources

    11 May’ 2018

    Issue: When we use a HTML web resource in Dynamics 365 CRM forms or even as a separate navigation, we sometimes face is...

    Read more
  • Dynamics 365 Tip: Sending Emails using Unified Interface

    10 May’ 2018

    Introduction: The unified client which was released D365 V9 had a very refreshing UI and lots of features. One of the b...

    Read more
  • Filtering Records on BPF unlike on Form – D365

    10 May’ 2018

    Introduction: Working with D365 is best when you’re trying to use as much OOB stuff as possible. Simplest configurati...

    Read more
  • Filter records in gallery control based on value selected in other gallery control in PowerApps.

    10 May’ 2018

    Introduction: This blog explains how to Filter records in gallery control based on value selected in other gallery cont...

    Read more