Using the Google BigQuery Connection Manager

The Google BigQuery Connection Manager is an SSIS connection manager that can be used to establish connections with Google BigQuery.

To add a new connection, right-click the Connection Manager area in your Visual Studio project, and choose "New Connection..." from the context menu. You will be prompted the "Add SSIS Connection Manager" window. Select the "Google BigQuery (KingswaySoft)" item to add a new Google BigQuery Connection Manager.

New SSIS Connection

Add SSIS Google BigQuery Connection Manager

The Google BigQuery Connection Manager contains the following 2 pages of configuration.

  • General
  • Proxy Settings

General page

The General page allows you to specify connection properties and login credentials to Google BigQuery.

SSIS Google BigQuery Connection Manager

Token Location

Google BigQuery retrieves tokens from the Google app once you give authorization. The tokens are stored in Windows Registry so that once they expire, our component will automatically request new tokens from Google and replace the existing tokens with the new ones in Registry to minimize user effort and avoid service interruption for scheduled jobs. You have two options to choose where in Registry you would like to store these tokens:

  • Local Machine registry hive – HKEY_LOCAL_MACHINE
  • Current User hive – HKEY_CURRENT_USER
Update Token

The Update Token button completes the entire OAuth authentication process inside of the toolkit. All you need to do is login to Google and authorize our app to generate your Google BigQuery tokens. Note: you must first fill out proxy settings in the Proxy Settings tab if proxy exists.

Account Id

The Account Id is an integer number that uniquely represents your account. You would need to make sure you have Tokens created. To create your tokens, select Update Token button. All existing Account IDs will be listed in the drop-down.

Account Name

The Account Name shows the name of your Google account.

Google BigQuery API Authorization Information

This option allows you to specify your application when working with Google BigQuery API.

  • Provided by KingswaySoft: use the Client Id and Secret that is provided by KingswaySoft for your convenience when setting up the connection, it should NEVER be used for production purpose. 
  • My Own: specify your own application registration with Google 
    • Client Id 
    • Client Secret

Warning: By default you are using the Google BigQuery API authorization information (Client Id and Client Secret) provided by KingswaySoft, which is provided for your convenience when setting up the connection, and it should NEVER be used for production purpose. To obtain your own Google BigQuery API authorization credentials for product use, please follow the instructions at the link below to create your own application registration with Google: https://cloud.google.com/bigquery/quickstart-web-ui

Advanced Settings

SSIS Google BigQuery Connection Manager - advanced settings

Proxy Server Settings

The Proxy Server Settings allows you to specify how you want to configure the proxy server.

Proxy Mode

Proxy Mode option allows to specify how you want to configure the proxy server setting. There are three options available.

  • No Proxy
  • Auto-detect (Using system configured proxy)
  • Manual
Proxy Server

Using the Proxy Server option, you can provide a proxy server to connect to Google BigQuery.

Port

The Port option allows you to specify port number of the proxy server for the connection.

Username

The Username option allows you to specify the proxy user account.

Password

The Password option allows you to specify the proxy user’s password.

Note: Proxy Password is not included in the Google BigQuery connection manager's ConnectionString property by default. This is done by design for security reasons. However you can include it in your ConnectionString if you want to parameterize your connection manager. The format would be ProxyPassword=myProxyPassword; (make sure you have a semicolon as the last character). It can be anywhere in the ConnectionString.

Timeout (secs)

The Timeout (secs) option allow you to specify a timeout value in seconds for the connection. The default value is 120 seconds.

API Throttling Rate

The API Throttling Rate option allows you to restrict how many requests you want to send to Google BigQuery per second. This rate is set to 10 and you can adjust upward from there to optimize your throughput without exceeding the rate limit.

Retry on Intermittent Errors

This is an option designed to help recover from possible intermittent outages or disruption of service. It prevents the integration process from stopping due to temporary issues. Enabling this option will allow service calls to be retried upon certain types of failure. A service call may be retried up to 3 times before an exception is fired. Retries occur after 0 seconds, 15 seconds, and 60 seconds.

Warning: We have designed our retry feature carefully such that the retry should only occur when it is deemed safe to do so; however, in some occasions such retry service calls could result in the creation of duplicate data.