Help Manual - Google BigQuery Connection Manager

Documentation on SSIS Google BigQuery Connection Manager component

Using the Google BigQuery Connection Manager

The Google BigQuery Connection Manager is an SSIS connection manager that can be used to establish connections with Google BigQuery.

To add a new connection, right-click the Connection Manager area in your Visual Studio project and choose "New Connection..." from the context menu. You will be prompted with the "Add SSIS Connection Manager" window. Select the "Google BigQuery (KingswaySoft)" item to add a new Google BigQuery Connection Manager.

New SSIS Connection

Add SSIS Google BigQuery Connection Manager

The Google BigQuery Connection Manager contains the following configuration pages:

  • General
  • Advanced Settings

General Page

The General page allows you to specify connection properties and login credentials for Google BigQuery.

SSIS Google BigQuery Connection Manager

Authentication Mode

The Authentication Mode option allows you to specify how you want to authenticate to your Google service. Available options are:

  • Authorization Code
  • Service Account (server-to-server authentication)

These options have been deprecated and are only available in v10.0 or earlier. If you are using v20.1 or newer, these options will not be available.

Token Location (Available only when Authorization Code is selected)

Google BigQuery retrieves tokens from the Google app once you give authorization. The tokens are stored in the Windows Registry so that once they expire, the component can automatically request new tokens from Google and replace the existing tokens in the Registry to minimize user effort and avoid service interruption for scheduled jobs. You have two options for where in the Registry to store these tokens:

  • Local Machine registry hive – HKEY_LOCAL_MACHINE
  • Current User hive – HKEY_CURRENT_USER
App Type (Available only when Authorization Code is selected)

The selected App Type that was chosen when creating the Google token.

Account (Available only when Authorization Code is selected)

The selected available Google Account Number and Name.

Authentication (Available only when Authorization Code is selected)
Generate Token File

This button opens a dialog that allows you to log in to the service endpoint and authorize your app to generate a new token.

SSIS Google BigQuery Connection

App Type

Specify the app that will be used to connect to Google BigQuery. Available options are:

  • KingswaySoft: The Connection Manager will use the Client ID and Client Secret provided by KingswaySoft for your convenience when setting up the connection. It should NEVER be used for production purposes.
  • My Own App: Select this option to use your own Client ID and Client Secret to connect to Google BigQuery.
App Info
Client ID

The Client ID option allows you to specify the unique ID that identifies the application making the request.

Client Secret

The Client Secret option allows you to specify the secret associated with your Google application.

Redirect Url

The Redirect Url option allows you to specify the redirect URL configured for the Google application in order to complete the authentication process.

Scope

The Scope option allows you to specify the OAuth scope of the token being generated.

PKCE

The PKCE (Proof Key for Code Exchange) option can be enabled to use the PKCE OAuth authorization code flow.

Sign In & Authorize
Use Default Browser to Sign In

When this option is checked, the Sign In & Authorize button will open your default web browser to complete the OAuth authentication. When this option is unchecked, the Sign In & Authorize button will complete the OAuth authentication process inside the toolkit.

Sign In & Authorize Button

This button allows you to log in to the service endpoint and authorize your app to generate a token.

Warning: By default, you are using the Google BigQuery API authorization credentials provided by KingswaySoft for your convenience when setting up the connection. These credentials should NEVER be used for production purposes. To obtain your own Google BigQuery API authorization credentials for production use, follow the instructions in the link below to learn how to create OAuth credentials: https://developers.google.com/identity/protocols/oauth2

Path To Token File

This option allows you to specify the path to the token file on the file system.

Token File Password

This option allows you to specify the password for the selected token file.

Key File Type (Available only when Service Account is selected) (since v21.1)

Specify the private key file format that will be used for authentication. Available options are:

  • JSON
  • P12

In order to use the Service Account authentication mode, a service account must be created in Google Cloud Console and a private key generated for it. The following steps can be used:

  • Log in to the Google Cloud Platform Console.

  • Navigate to IAM & Admin.

  • Choose "Service Accounts" from the list, and then click "Create Service Account".

  • Provide the "Service Account Name". The "Service Account ID" will be automatically populated based on the name.

  • Assign permissions if required.

  • Choose the "Keys" tab, click "Add Key", and then select "Create New Key".

  • From the window that appears, select the required key type and download the private key file.

JSON Key File Path (Available when JSON is selected as the Key File Type)

Specify the path to the JSON private key file used for authentication.

Service Account

When JSON is selected, this read-only field displays the Service Account from the specified JSON key file. When P12 is selected, specify the Service Account associated with the P12 certificate.

Client Id (Available when JSON is selected as the Key File Type)

This read-only field displays the Client Id from the specified JSON key file.

Certificate Location (Available when P12 is selected as the Key File Type)

Specify where the P12 certificate used for authentication is located. Available options are:

  • Store (Recommended)
  • File System
Certificate Thumbprint (Available when Store is selected as the Certificate Location)

Specify the thumbprint of the certificate in the Windows Certificate Store that will be used for authentication.

Certificate File Path (Available when File System is selected as the Certificate Location)

Specify the path to the P12 certificate file used for authentication. Click the ellipsis button to browse the file system.

Note: The component supports an Azure Blob Shared Access Signature (SAS) URL in the certificate path.

Certificate File Password (Available when File System is selected as the Certificate Location)

Specify the password for the P12 certificate file.

Test Connection

After all the connection information has been provided, click the Test Connection button to test whether the connection settings entered are valid.

Advanced Settings Page

The Advanced Settings page of the Google BigQuery Connection Manager allows you to specify advanced settings for the connection.

SSIS Google BigQuery Connection Manager - advanced settings

Proxy Server Settings

The Proxy Server Settings section allows you to specify how you want to configure the proxy server.

Proxy Mode

The Proxy Mode option allows you to specify how you want to configure the proxy server setting. There are three options available:

  • No Proxy
  • Auto-detect (Using system-configured proxy)
  • Manual
Proxy Server

The Proxy Server option allows you to specify a proxy server to connect to Google BigQuery.

Port

The Port option allows you to specify the port number of the proxy server for the connection.

Username

The Username option allows you to specify the proxy user account.

Password

The Password option allows you to specify the proxy user's password.

Note: Proxy Password is not included in the Google BigQuery Connection Manager's ConnectionString property by default. This is done by design for security reasons. However, you can include it in your ConnectionString if you want to parameterize your connection manager. The format would be ProxyPassword=myProxyPassword; (make sure you have a semicolon as the last character). It can appear anywhere in the ConnectionString.

Misc
Timeout (secs)

The Timeout (secs) option allows you to specify a timeout value in seconds for the connection. The default value is 120 seconds.

API Throttling Rate

The API Throttling Rate option allows you to limit the number of requests sent to Google BigQuery per second. The default value is 10 requests per second. You can increase this value to optimize throughput while staying within the applicable API rate limits.

Retry on Intermittent Errors

This option is designed to help recover from possible intermittent outages or service disruptions. It prevents the integration process from stopping due to temporary issues. Enabling this option allows service calls to be retried upon certain types of failure. A service call may be retried up to 3 times before an exception is fired. Retries occur after 0 seconds, 15 seconds, and 60 seconds.

Warning: We have designed our retry feature carefully so that a retry should only occur when it is deemed safe to do so; however, in some cases, retrying service calls could result in the creation of duplicate data.