Product Documentation

Appearance

Loading from Amazon S3 ​

These instructions assume that you already have an Amazon Simple Storage Service (Amazon S3) account, know the S3 location of the files you want to load, and have your S3 credentials ready at hand. If not, follow the instructions on the AWS site before proceeding.

To bulk load a table from files in an Amazon S3 bucket:

  1. Obtain your S3 security credentials (in this example, an access key ID, a secret access key, and the region). See S3 Authentication Methods for information about alternative ways to authenticate.

  2. Run aws configure or use set commands to set the environment variables for the S3 access keys and the region.

For example, on Linux:

$ aws configure
AWS Access Key ID [****************SWUA]: XXXXXXXXXXXXXXXXXXXX
AWS Secret Access Key [****************n2L+]: ****************************************
Default region name [None]: us-east-2
Default output format [None]:

For example, on Windows:

set AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
set AWS_SECRET_ACCESS_KEY=****************************************
set AWS_REGION=us-east-2

You can also specify ybload command-line options to authenticate with access keys.

  1. Record the S3 location of the file you want to load. For example:
s3://ybbob/premdb/match01.csv

where ybbob is the S3 bucket name and premdb is a folder in that bucket.

You can only access one object store per ybload operation. For example, you cannot specify connection details for both an S3 client and an Azure client in the same command, and you cannot load data from multiple S3 endpoints or storage accounts.

For more details, see S3 Object Storage URIs.

  1. Run the ybload command in the usual way, providing the name of a bad row file on the local client system and the S3 path to the source file. For example:
$ ybload -d premdb --username bobr -W -t match --format csv --delimiter ',' 
--bad-row-file /home/brumsby/s3bad s3://ybbob/premdb/match01.csv

To load multiple source files from the premdb folder with the prefix match, you could use:

s3://ybbob/premdb/match

Alternatively, list each S3 file separately. For example:

s3://ybbob/premdb/matchst01.csv s3://ybbob/premdb/match02.csv s3://ybbob/premdb/matchst03.csv

If you specify a log file, it must also be on the local file system (like the bad row file).

In This Section

Parent topic:Loading Data from Object Storage