Using the Kafka Destination Component
The Kafka Destination Component is an SSIS data flow pipeline component that can be used to send messages to a Kafka topic. There are three pages of configuration:
- General
- Columns
- Error Handling
General Page
- Connection Manager
-
The destination component requires an active connection to Kafka. The Connection Manager drop-down will show a list of all Kafka Connection Managers that have been created in the current SSIS package.
- Action
-
There are two actions: Commit and Create
- Topic
-
The list of topics will be presented from those available in Kafka. Messages will be written on this topic.
- Message Column Type
-
The Message Column Type allows you to specify whether you want to write the binary content of the message or the text content of the message to Kafka. There are two different modes available for Message column types: Binary and Text. The default setting is Text.
- Encoding
-
The encoding will be used to convert the message body to a byte array when sending the message. This option is only available when the Message Column Type is Text.
- Compression Type
-
The compression type shows the list of compression types that can pick one of them.
- Batch Size
-
The Batch Size to use when sending messages.
- Refresh Component Button
-
Clicking the Refresh Component button causes the component to retrieve the latest metadata and update each field to its most recent metadata. It will remove any custom fields that have been added to the columns page.
- Map Unmapped Fields Button
-
By clicking this button, the component will try to map any unmapped attributes by matching their names with the input columns from upstream components. This is useful when your source component has recently added more columns, in which case you can use this button to automatically establish the association between input columns and unmapped destination attributes.
- Clear All Mappings Button
-
By clicking this button, the component will reset all your mappings in the destination component.
- Expression fx Icon
-
Click the blue fx icon to launch SSIS Expression Editor to enable dynamic updates of the property at run time.
- Generate Documentation Icon
-
Click the Generate Documentation icon to generate a Word document that describes the component's metadata including relevant mapping, and so on.
Columns Page
The Columns page allows you to map the columns from upstream components to destination attributes.
On the Columns page, you will see a grid as shown below.
- Input Column: You can select an input column from an upstream component here.
- Kafka Field: The attribute/field that you want to write data to.
- Data Type: This column indicates the type of value for the current attribute.
- Unmap: This button provides a convenient way to unset the mapping for the selected attribute/field.
- Add Button
Clicking the Add button will bring up a dialog that will allow columns to be added that correspond to custom headers on outgoing messages.
Error Handling Page
The Error Handling page allows you to specify how errors should be handled when they happen.
There are three options available:
1. Fail on error
2. Redirect rows to error output
3. Ignore error
When the Redirect rows to the error output option is selected, rows that failed to write to Kafka will be redirected to the 'Error Output' of the Destination Component. As indicated in the screenshot below, the green output connection represents rows that were successfully written, and the red 'Error Output' connection represents rows that were erroneous. The 'Kafka ErrorMessage' output column found in the 'Error Output' may contain the error message that was reported by Kafka or the component itself.
Note: Use extra caution when selecting Ignore error option, since the component will remain silent for any errors that have occurred.