New CRM SDK Feature - Entity Change Tracking

In CRM integration projects, your business requirements often dictate that you should only retrieve those transaction records that have been recently created or modified in your source system since last integration. When your CRM system is the source, you would typically use CRM FetchXML query to pull data from the CRM system by comparing the record's modifiedon field, which is not a solution that is reliable enough and may not perform well when the source entity has a significant number of records.

To address this particular challenge, CRM Online 2015 Update 1 release introduced the Change Tracking feature, which offers a reliable and efficient way to track transactional data changes for CRM entities.

Turn on Change Tracking

In order to track data changes of a particular entity, you have to first turn on "Change Tracking" option for the entity. You would do so by going to the entity's customization page, and tick "Change Tracking" option under "Data Services" section.

Retrieve Changes

After Change Tracking has been enabled for the entity, you can use CRM SDK RetrieveEntityChangesRequest message to pull changes from CRM server. The following is a snippet that demonstrates how to pull changes from the CRM server.

using (var service = new OrganizationService(crmConnection))
{
    var request = new RetrieveEntityChangesRequest();

    // Set which entity to get changes for.
    request.EntityName = "account";
    request.Columns = new ColumnSet("accountnumber", "name", "creditlimit");

    // Set paging preferences.
    request.PageInfo = new PagingInfo() { Count = 5000, PageNumber = 1 };

    // Set change token returned from last pull. 
    request.DataVersion = changeToken; // set to null or remove this line to do an initial pull

    // Get the changes. 
    var response = (RetrieveEntityChangesResponse)service.Execute(request);

    // TODO: Process all the changed records (see the code snippet below)

    // Save the token somewhere for future use
    changeToken = response.EntityChanges.DataToken;
    Console.WriteLine(changeToken);
}

Walk through Changes

After retrieving the changes, you will get two types of results. The first type of result is the newly added or updated records, and the second one is for the records that have been deleted since the last pull.

// Replace the TODO line in the above snippet with the following code
foreach (var change in response.EntityChanges.Changes)
{
    if (change.Type == ChangeType.NewOrUpdated)
    {
        var changedItem = (NewOrUpdatedItem)change;
        Entity newOrChangedEntity = changedItem.NewOrUpdatedEntity;
        // TODO: Process new or updated entity record
    }
    else if (change.Type == ChangeType.RemoveOrDeleted)
    {
        var deleteditem = (RemovedOrDeletedItem)change;
        EntityReference deletedEntityReference = deleteditem.RemovedItem;
        // TODO: Process deleted entity records
    }
}

Note that the returned NewOrChangedEntity is an instance of Entity class, while the RemovedItem (the deleted record) is an instance of EntityReference class.

Some Closing Notes

There are a few things that you should be aware when using this feature:

• The "Change Tracking" option is an entity-level setting. It has to be enabled for the entity before you can use RetrieveEntityChangesRequest to pull data. If the option is not enabled, and you try to use this feature, you will get an error message telling you "Entity: account isn't enabled for change tracking" where account can be any entity name that you are working with.
• Due to the fact that the "Change Tracking" option works at entity level, you would need to keep track of the change token for each entity individually. There is no organization level tracking token.
• There is one special situation that you should watch out for. Suppose there is a newly added record after last pull and it was deleted before the new pull, you will get the record in the Delete result set, which you may not have knowledge about it.
• In the case that you have selected to return a lookup field, the field's Name property is not returned.
• When change collection has more records than the page size that you have specified, you would have to page through the change collection.
• Based on my preliminary testing, it appears that CRM keeps multiple token versions, which is really nice. However, I imagine keeping too many versions on the server side would not make sense as each version would consume a certain amount of database space. There must be a limit in terms of how many versions are kept on the CRM server side. It could be either a time-based limit (say 90 days probably, which could still be a lot of change versions if I keep pulling from the server every 30 seconds) or a number-based limit. This is something that has yet to be confirmed.
• When you retrieve changes using this feature, you will get the records in their current status, the old values before the change are not returned.
• The user account needs to have organization level read (or so-called "Root Read") privileges for the concerned entity in order to uses RetrieveEntityChangesRequest to retrieve changes.
• More details about this feature can be found at the CRM SDK documentation page: Use change tracking to synchronize data with external systems.

Hope this helps.

Read the full New SDK Capabilities Blog Series:

• Part 1 - Alternate Keys
• Part 2 - Upsert
• Part 3 - No more special messages for special fields 
• Part 4 - Plugin Trace Logging 
• Part 5 - Entity Change Tracking 
• Part 6 - Transactional Batching 
• Part 7 - Optimistic Concurrency 
• Part 8 - New Query Operators
• Part 9 - Detect SDK Capabilities by Checking CRM Server Version