Blog Post: Performing Bulk Operations Using Scribe Insight Adapter for Dynamics CRM 2011

For the latest updates to this post please visit the original posting here: Performing Bulk Operations Using Scribe Insight Adapter for Dynamics CRM 2011

You can speed up your data migration using the new Scribe Insight Adapter for Dynamics CRM, Version 5.4.0. How? By doing a bulk operation! The new adapter comes with Scribe Insight 7.6, which now allows you to perform bulk insert, update, and delete operations.

A bulk operation processes a block of rows in one transaction, minimizing the number of calls between the Scribe source and the CRM destination. The block of rows (as specified in the screenshot below) is sent in a bundle once to CRM for commit.

With row-based processing, each row needs to be committed before it sends another. This does not mean that if you set number of rows per operation to 1000 that it will be 1000 times faster than row-based operations. Scribe does not work that way yet! There are a number of factors that determine how many records are pushed in one transaction, including type of connection, cross reference keys, update source, or rejected rows.

Here is how you enable bulk operation in a DTS package step:

1. Go to Configure Steps.
2. Select an insert, update or delete step and then click on the Operation tab.
3. You should see a check box labeled Use Bulk mode. Check it!
4. Close this window and save the package.

You are all set!

There are certain things you’ll want to consider before you use bulk operations in your Scribe package. Some of the very handy features of Scribe becomes unavailable when we use bulk operation. Be aware of the following:

• If you have a step in the DTS file that depends on the step before it, bulk operation cannot be used. This is because blocks are committed independently. You might get into a situation where a block in step 2 gets committed before a block in step 1, which will make it out of sync. A common use of this dependency is when you want to update your source table with the GUID of the record that you created or updated in CRM—bulk operation does not work in this situation.
• If you have variables that are populated from your steps, they will not be populated because processing of records happens in block.
• If you have lookup functions in DTS that depend on a value from a step that is set to operate on bulk mode, then it will not work.

The speed advantage

Speed is the primary reason why we look for bulk operation. If the DTS steps are simple and need to process a large number rows (say in billions), then it is definitely a good idea to use bulk mode.

Below are performance comparison statistics compiled by PowerObjects using Microsoft Dynamics CRM Online:

Note: These are not Scribe’s official numbers. These are the numbers that PowerObjects got when we tested this new feature.

The focus here is not on how big these numbers are—the actual numbers can vary based on number of processes running in parallel, number of fields updated, and so on. Instead, look at the difference between the rows per minute for these two types of operations. Our test shows that the bulk insert operation is at least 5.3 times faster than row-based inserts. Similarly, bulk update is 6 times faster than row-based updates. In either case, the results clearly show that bulk operation outperforms row-based operations by multiples of 5 or more.

In conclusion, you can get more performance using bulk operation than normal row-based operation. At the same time, bulk mode has some limitations. So, to utilize Scribe Insight’s bulk mode feature of the new Adapter for Dynamics CRM 2011, consider simplifying the Scribe package steps. You can mix and match row-based or bulk mode steps in a single package.

Bulk operation is a very powerful feature and can be best utilized in migrating/integrating to CRM online where the number of rows processed is large.

Happy migrating and integrating to Dynamics CRM!

The post Performing Bulk Operations Using Scribe Insight Adapter for Dynamics CRM 2011 appeared first on PowerObjects.