High-quality data is the cornerstone of any successful CRM implementation. Accurate, duplicate-free records ensure smooth automation, reliable reporting, and consistent customer experiences across systems. However, duplicate data is a persistent challenge in CRM environments - Salesforce included. Duplicates can arise from various sources, such as:
- Manual entry errors: Different users may unknowingly enter the same customer more than once.
- Customer-submitted data: A customer might fill out a web form even though their information already exists in the system.
- System integrations: Data synced from external platforms like marketing or e-commerce systems often introduces duplicates.
- Lead conversion: When converting leads to contacts, new records may be created if an existing contact isn’t properly matched.
Salesforce provides helpful built-in tools such as Duplicate Rules to prevent duplicates during record creation. However, these tools have limitations. For instance, if a rule is configured only to warn rather than block, users can still save duplicate records. Moreover, Duplicate Rules don’t retroactively clean up existing duplicates - leaving you with a potential backlog of redundant data.
For a more comprehensive solution, it’s often best to run periodic cleanup jobs that identify, merge, and remove duplicates based on your own criteria. Fortunately, Salesforce’s SOAP API includes a Merge operation (see Salesforce documentation), and the KingswaySoft Salesforce Destination component fully supports this functionality as part of the SSIS Integration Toolkit for Salesforce product offering.
Combined with the Duplicate Detector from the SSIS Productivity Pack, you can easily build an automated, no-code required deduplication flow. In this article, we’ll walk through how to identify and merge duplicate Contact records in Salesforce using SSIS and KingswaySoft components.
Note: Salesforce’s REST API currently does not support the merge() operation. While the REST API allows for Create, Read, Update, Delete, and Upsert actions, Merge capabilities remain exclusive to the SOAP API.
We’ve also provided a downloadable sample SSIS package that contains the designs illustrated below, so that it can get you started quickly. You can customize the design by changing the duplicate detection strategy in order to be fully tailored for your specific integration needs.
Configuring the Duplicate Detection Flow
The Duplicate Detector component is an SSIS transformation component that can be used to scan and find potential duplicates from the incoming records from the upstream, based on a custom criterion. Let's consider the Salesforce Source component (likely using the same connection manager that will be used later in the destination component discussed below), which reads all Salesforce contact records. The primary goal is to find out the duplicate records that have the same First Name and Last Name, and rank them based on the Last Modified Date to determine which record to keep. In the Duplicate Detector component, we can set the filters in the General page as shown below.
In the Comparison Settings page, we define the record ranking strategy. Since we want to keep the most recently updated record, we’ll rank by LastModifiedDate in descending order.
The Duplicate Detector component has a few additional columns for gauging data quality, and out of those, the following two are used in our data flow.
- Group Code: Identifies which records belong to the same duplicate group.
- Record Rank: Assigns a rank within each group based on the chosen sorting logic (rank 1 represents the record to keep).
Once we have the duplicate records identified, we can move on to the step of merging them. This is achieved using the Salesforce destination component using the Merge action. Salesforce Merge action requires two input columns: One is a _masterRecordId, which is the Salesforce ID of the master record to which the other duplicates are being merged into; and the second field is _recordToMergeIds, which contains the IDs of the records that need to be merged. Now, to identify and split the Source duplicates into master and merge records, we can use a Conditional Split component as shown below, and set a condition based on the Record Rank. The records with Record Rank as 1 from each group of duplicates will flow into an output called MasterRecords. And the ones that are not ranked as 1 will be the records that need to be merged, and will flow into the second output called RecordsToMerge.
Next, we use Sort and Merge Join components to match up records by their Group Code and attach the corresponding MasterRecordId to each duplicate.
The output of the Merge Join component will have two fields: The MasterRecordID that contains the Salesforce ID for the records that are to be preserved, repeating for each of the IDs in the RecordsToMergeIds.
Now let's see how the Salesforce Destination handles the Merge action.
Configuring the Salesforce Merge Process
In the Salesforce Destination component, choose the Merge action and select the Contact object. Ensure that your connection manager points to the SOAP endpoint.
On the Columns page, map the fields accordingly from the upstream Merge Join output. You can also map additional Contact fields if you wish to update master records during the merge process.
Once configured, your overall data flow will look like this:
With the full data flow design ready, you can schedule this SSIS package to run periodically, ensuring your Salesforce data remains clean and consistent without manual intervention.
Conclusion
This solution shows how easily Salesforce deduplication can be automated with KingswaySoft. By combining the Duplicate Detector and Salesforce Destination components, you can eliminate redundant data, preserve key records, and maintain a reliable single source of truth - all without writing a single line of code. Customize the ranking and merging rules as needed to fit your business logic and maintain top-quality data across your CRM system.
We hope this guide helps you keep your Salesforce data cleaner and smarter with KingswaySoft!