Using the HDFS Source Component

The HDFS Source Component is an SSIS data flow pipeline component that can be used to read / retrieve data from HDFS.

The component includes the following two pages to configure how you want to read data from HDFS.

  • General
  • Columns

General page

The General page of the HDFS Source Component allows you to specify the general settings of the component. 

HDFS Source Editor

Connection Manager

The HDFS Source Component requires a Hadoop connection in order to connect with HDFS. The Connection Manager drop-down will show a list of all Hadoop connection managers that are available to your current SSIS package.

Source Item Path

The Source Item Path specifies the base item to point to for retrieval. Click the eclipse button ('...') to open up a HDFS browser dialog to select an item.

Item Selection Mode

The Item Selection Mode specifies what HDFS item(s) will get selected. The Files Only modifier will still go through folders to retrieve items but will not return any records for the folders.

  • Selected Item: Retrieves only the item specified by Source Item Path.
  • Selected Level: Retrieves the selected item and all immediate files and folders under the path specified by the Source Item Path option.
  • Selected Level (Files only): Retrieves the selected item and all immediate files under the folder as specified by the Source Item Path option.
  • Recursive: Retrieves the selected item (specified by the Source Item Path option) and all sub items recursively.
  • Recursive (Files only): Retrieves items the same as the Recursive mode but only returns files.
Refresh Component Button

Clicking the Refresh Component button causes the component to retrieve the latest metadata and update each field to its most recent metadata.

Expression fx Button 

Clicking the fx button to launch SSIS Expression Editor to enable dynamic update of the property at run time.

Generate Documentation Button 

Clicking the Generate Documentation button to generate a Word document which describes the component's metadata including relevant mapping, and so on.

Columns page

The Columns page of the HDFS Source Component shows you all available attributes from the object that you specified on the General page. 

HDFS Source Editor

On the top left of the grid, you can see a checkbox, which can be used to toggle the selection of all available HDFS fields. This is a productive way to check or uncheck all available fields. 

The Columns Page grid consists of:

  • HDFS Field: Column that will be retrieved from the current item (file or folder).
  • Data Type: The data type of this field.

Note: As a general best practice, you should only select the HDFS fields that are needed for the downstream pipeline components.