Using the Kafka Destination Component

The Kafka Destination Component is an SSIS data flow pipeline component that can be used to send messages to a Kafka topic. There are three pages of configuration:

  • General
  • Columns
  • Error Handling

General Page

Azure Queue Storage- Connection manager

Connection Manager

The destination component requires an active connection to Kafka. The Connection Manager drop-down will show a list of all Kafka Connection Managers that have been created in the current SSIS package.

Topic

The list of topics will be presented from those available in Kafka. Messages will be written on this topic.

Message Column Type

The Message Column Type allows you to specify whether you want to write the binary content of the message or the text content of the message to Kafka. There are two different modes available for Message column types: Binary and Text. The default setting is Text.

Encoding

The encoding will be used to convert the message body to a byte array when sending the message. This option is only available when the Message Column Type is Text.

Compression Type

The compression type shows the list of compression types that can pick one of them.

Batch Size

The Batch Size to use when sending messages.

Refresh Component Button

Clicking the Refresh Component button causes the component to retrieve the latest metadata and update each field to its most recent metadata. It will remove any custom fields that have been added to the columns page.

Map Unmapped Fields Button

By clicking this button, the component will try to map any unmapped attributes by matching their names with the input columns from upstream components. This is useful when your source component has recently added more columns, in which case you can use this button to automatically establish the association between input columns and unmapped destination attributes.

Clear All Mappings Button

By clicking this button, the component will reset all your mappings in the destination component.

Expression fx Icon

Click the blue fx icon to launch SSIS Expression Editor to enable dynamic updates of the property at run time.

Generate Documentation Icon

Click the Generate Documentation icon to generate a Word document that describes the component's metadata including relevant mapping, and so on.

Columns Page

The Columns page allows you to map the columns from upstream components to destination attributes.

On the Columns page, you will see a grid as shown below.

Azure Queue Storage- Connection manager

  • Input Column: You can select an input column from an upstream component here.
  • Kafka Field: The attribute/field that you want to write data to.
  • Data Type: This column indicates the type of value for the current attribute.
  • Unmap: This button provides a convenient way to unset the mapping for the selected attribute/field.
Add Button

Clicking the Add button will bring up a dialog that will allow columns to be added that correspond to custom headers on outgoing messages.

Error Handling Page

The Error Handling page allows you to specify how errors should be handled when they happen.

Kafka Destination Error Handling

There are three options available:

1. Fail on error

2. Redirect rows to error output

3. Ignore error

When the Redirect rows to the error output option is selected, rows that failed to write to Kafka will be redirected to the 'Error Output' of the Destination Component. As indicated in the screenshot below, the green output connection represents rows that were successfully written, and the red 'Error Output' connection represents rows that were erroneous. The 'Kafka ErrorMessage' output column found in the 'Error Output' may contain the error message that was reported by Kafka or the component itself.

/Kafka Destination-Error Handling Error output

Note: Use extra caution when selecting Ignore error option, since the component will remain silent for any errors that have occurred.