While working with Salesforce integrations, if you have a large number of records that you need to read or write and the process is taking longer than anticipated, then you may benefit from using the Bulk API feature provided by Salesforce. The Salesforce Bulk API is extremely useful to speed up the process, thereby getting maximum performance for larger volumes. Not just for write actions, but Bulk API can also be used for large-scale read or data retrieval processes. The Salesforce Bulk API will read/write those records from/to your Salesforce organization, which, when set up properly, improves performance by using parallel processing and fewer network round trips. Our KingswaySoft Salesforce Source and Destination components both support Bulk API. Today we will look at the feature and do a walk-through using a simple demonstration of how to write bulk data to a Salesforce object using this feature.
In this blog post, the KingswaySoft components we will use to perform the Bulk API task are listed below:
- Salesforce Connection Manager to establish a connection to your Salesforce environment.
- Salesforce Destination component for loading data.
- Premium Derived Column component to include a component connected to the default output.
- Data Spawner component as a dummy Source component to read variable values and as a data generator for our test.
Design and configuration
The design for the case study is shown below and should be a quick setup. The first data flow (Bulk API Load) loads data into Salesforce, and the second data flow (Retrieve Bulk Job Id Variable) retrieves the bulk job id. Additionally, we have added a part to save the Job Id to a database table; however, it can be used per your requirement in the downstream data flow.
Bulk API Load data flow
The Data Spawner component will create some sample data: 100,000 records for our test today, which will get loaded into the Salesforce Destination component. The last Dummy Destination component, which acts as a data flow destination component here, will be included in the scenario where the Salesforce Destination component needs a default output connected to any Destination component (which we will see why in later sections), and in the other cases, the Destination component will be removed.
When using KingswaySoft’s Salesforce Connection Manager and choosing SOAP as the service endpoint, you will have a few options available that you can set, as shown below. If you would like to read more on each of these options, visit the help manual for the Salesforce Destination component provided at the beginning of this blog post.
When using KingswaySoft’s Salesforce Connection Manager and choosing REST as the service endpoint, the only option under the Bulk API section you can change is whether to Use Bulk API or not. The grayed-out options are handled by the service, so you will not be able to change them. The available options will be auto-selected by the service to get the best performance. And lastly, the batch size cannot be selected, as it uses the buffer memory.
Retrieve Bulk Job Id Variable data flow
You would see that even in this data flow, a Data Spawner component is being used. The Data Spawner component here is used as a dummy source (creating a null record), while the Premium Derived Column component is a transformation component and requires a primary input. The Premium Derived Column will just store the bulk id variable in a column.
Then, further downstream, we have added a Premium ADO.NET Destination component to store the sfbulkJobId column in a table.
Test and results
There are four tests we ran, each one processing 100,000 records:
- Case 1: REST connection with Bulk API enabled and Salesforce Destination default output connected to a Dummy Destination component. This is typically used if you wish to get feedback when the action is complete and requires the SalesforceRecordId in the downstream data flow process.
- Case 2: REST connection with Bulk API enabled and Salesforce Destination default output not connected.
- Case3: SOAP connection with Bulk API enabled and Salesforce Destination default output connected to a Dummy Destination component. This is typically used if you wish to get feedback when the action is complete and requires the SalesforceRecordId in the downstream data flow process.
- Case 4: SOAP connection with Bulk API enabled and Salesforce Destination default output not connected.
Now, let's look at the difference between REST and SOAP Connection Managers in relation to the Bulk API, as well as the impact of a default output connected to a component, or having no downstream component connected to the default output when the data flow is executed.
When you test this yourself, you will notice that:
- When Bulk API is enabled, and there is no default output connected to the Salesforce Destination, we received the full benefits, a significant reduction in processing time, and the process was asynchronous.
- On the other hand, when Bulk API is enabled, and the Salesforce Destination component has a default output, the benefits are lost to a degree but still work faster than a regular batch process.
- Both REST and SOAP are affected when adding a default output.
- Lastly, in our simple tests, the REST and SOAP connections had a similar reduction in processing time, so in more complex scenarios you would need more testing to see which one meets your requirements.
Conclusion
To sum up what we have observed above, having no component connected to the Salesforce's Destination component default output means we get the full benefit of Bulk API speed (the process is asynchronous). However, any server data errors will not be displayed and would need to be checked from the Salesforce side using the bulk Job ID, which, as we have seen in the second data flow task, can easily be logged. Lastly, the default output connected to another downstream Destination component as the SalesforceRecordId needs to be retrieved. Thus, using KingswaySoft Salesforce components, data processing can be simplified and performed at a high-performance rate with the Bulk API.
We hope this has helped!