Unloading Data to Google Storage

These instructions assume that you already have a Google Storage account, know the storage location of the files you want to load, and have the appropriate credentials ready at hand. If not, follow the instructions on the Google Storage portal before proceeding with the information here. Specific prerequisites include:

• A Google Storage account with valid credentials and appropriate permissions:

  • Credentials: You need to enable interoperability access in your Google Storage bucket and provide HMAC keys as --object-storage-credential and --object-storage-identity.
  • Permissions: The Google Storage account used for unloading to the bucket should have write permissions to the bucket.

Assuming that these prerequisites are in place, you can unload data from a table or a query to files stored in a Google Storage bucket.

To unload a table or the results of a query to a Google Storage bucket:

1. Use HMAC keys for authentication.

   You can only access one object store per ybunload operation. For example, you cannot specify connection details for both a Google Storage client and an S3 client in the same command, and you cannot unload data from multiple Google Storage endpoints or storage accounts.

2. Identify the bucket location in Google Storage where you want the unloaded files to be stored.

   At the end of the ybunload command line, you will specify the destination with the -o option and a URI, which is an abbreviated Google Storage path. For example:

   -o gs://premdb/

3. Run the ybunload command in the usual way, providing the correct bucket location.

   For example, the following command would unload the match table from the premdb database and store the unloaded files in parquet format in a Google Storage bucket:

   $ ybunload -d premdb --username bobr -W -t match --format parquet
   --object-store-credential '****************************************'
   --object-store-identity 'ybbobr'
   --object-store-endpoint "https://storage.googleapis.com"
   -o gs://premdb/

   Alternatively, you can save the identity and credential values to a Java properties file and name the file in the ybunload command. See Object Storage Options.

   If you specify a log file, it must be on the local file system. Specifying a log file in an object storage location, such as an GS bucket, is not supported.