Announcing Migration Starter Pack for CRM Online and CRM On-Premise

16 September 2016
Daniel Cai

CRM migration can be a daunting task as there are many things that can go wrong during the migration development process. For this reason, we are very excited to announce the general availability of Migration Starter Pack for CRM Online and CRM On-Premise, a sample CRM migration package that is built on top of our SSIS Integration Toolkit software, which you can use, or customize to fit your specific migration needs.

In almost every CRM migration project, you have to have a starting point, and you have to deal with CRM entity dependency in almost every single project. Our hope is, using the sample package, you can save a significant amount of time when developing the migration tasks for those commonly used system entities that you have to deal with again and again, as we have sorted out their dependencies including handling many other details. Among them, there are a few other things that I would like to highlight which you might benefit from using the sample package. 

  • We have demonstrated some design patterns that you can utilize if you need to work with your own custom entities, including how to work with the dependencies, how to retain createdby, createdon field values, etc.
  • We have showed the techniques that you can utilize migrating CRM activity records without losing the display sequence in the activity wall (or social wall). More details about the approach can be found in our other blog post.
  • We have demonstrated the approaches that can be utilized to migrate CRM product and product family records in the sample package.There are three main challenges:
    • There are more than a few entities involves, and the dependencies among those entities are not obvious.
    • When creating a product or product family record within a product family tree, you really only get one chance to get it in the right spot in the tree, as CRM does not offer a way to move a product around with the CRM product hierarchy.
    • Products have to be in certain states in order for them to be used by associated entities.
  • We have demonstrated the way of migrating CRM quote records, which we have covered in our other blog post.
  • Migrating recurring appointment is a tricky business, which is covered by the sample package and was further discussed in our other blog post as well.

Furthermore, we have learned some interesting lessons while creating the package.

  • The ownerid field of opportunityproduct entity does not have a corresponding owneridname field, so we are trying to perform two joins in the source component to bring the owning user name or team name, which is used for the text lookup setting in the destination component. The same applies to quotedetail, salesorderdetail and activitymimeattachment entity.
  • There are a number of entities in CRM that have sort of an entity-level circular references. When this is the case, it requires multi-pass in order to migrate the involved entities properly.
  • For some entities that have a parent-child relationship, the parent has to remain open until the migration of the child entity is done. So what we have to do is to first migrate the parent entity without having the statecode/statuscode fields mapped, essentially having the parent entity records migrated in an open or active status, then migrate the child entity records, finally we use another data flow task to specifically update the parent entity's status using an Update or Upsert action. Such examples include opportunity / opportunityproduct, salesorder / salesorderdetails, etc.

The migration package can be used for different migration scenarios, such as:

  • Migrating from CRM On-Premise to CRM On Online.
  • Migrating from CRM Online to CRM On-Premise.
  • Migrating from one CRM instance to another one of the same deployment type.
  • Migrating from an old version of CRM installation to a newer one.

There might be other scenarios that you could possibly use the packages as well, we leave that to your imagination and decision.

The migration package is available at our product download page. The package is created using SSIS 2008 R2, and it should automatically upgrade if you are using a newer version of SSIS. The whole package looks like something below.

CRM Migration Starter Pack

To use the package, you would first need to:

  • Update the connection manager to point to your CRM instances, including changing the login credentials, discovery service URL, authentication type, etc.
  • For any entities in this package that you might have new metadata, you need to click the "Refresh CRM Metadata" button in the CRM source component and CRM destination component UI.
  • The package is written for CRM 2016. If you use an older version of CRM server, you need to click the "Refresh CRM Metadata" button for each data flow component just in case there are some CRM 2016 specific fields (You can do that for only the data flow tasks that fail). In case you have to work with CRM 4 or 3, you would have to manually change the service endpoint in CRM connection managers, then click the "Refresh CRM Metadata" button for each data flow component.

While developing the package, we have made the following assumptions:

  • We assume that the base currency is the same across the two CRM instances (we use text lookup feature for any lookup fields that refer to transactioncurrency entity)
  • We assume that you want to retain the createdby values.
  • To retain createdby values, we assume that all system users have been created in the target CRM system using the same name with the same name sequence format. (we use text lookup feature for any lookup fields that refer to systemuser entity. Those fields include createdby, modifiedby, ownerid, etc.)
  • We assume that you want to retain the createdon values, so we have mapped createdon field from the source to the overriddencreatedon field in the destination component in all data flow tasks.

Some additional notes about the starter package:

  • We have tried to organize the data flow tasks based on the underlying relationship dependencies. Adding new data flow tasks to the package should take such dependencies into consideration.
  • All data flow tasks are using a single-destination component. You can however use BDD component to write to CRM simultaneously which would help improve the performance.
  • In addition to the BDD component mentioned above, you can use the RowIndex derived column functionality offered in our SSIS Productivity Pack, and implement the simultaneous write in a similar fashion, please refer to our blog post for more details.
  • We have mapped createdon field from the source to the overriddencreatedon field in the destination component in all data flow tasks in order to retain the createdon value from the source system.
  • Note that the migration of some entities involves multiple data flow tasks due to the dependencies of certain fields.
  • We use Upsert action in most of our data flow tasks, so technically you can re-run the package multiple times without any harm. This can be helpful in the case that your source system is still in use while the migration is in progress in parallel. You can do multiple runs before the final push, which can minimize the time window required for the final cut over.
  • Note that we have used "Ignore Unchanged Fields" option in the data flow tasks whenever possible, as this option can help improve performance in the case you have to do multiple runs of the migration package.

Please feel free to give the sample package a try and let us know if you have any further comments. We would like to receive your feedback to make it better, and we certainly hope that you find the package useful.

comments powered by Disqus