Product Documentation

Appearance

All platforms

S3 Object Storage Examples ​

The following examples show different ways to load a table from an S3 bucket:

Load a table from an AWS S3 bucket, using aws configure for authentication: ​

This example uses the aws configure command to set the access key, identity, and region:

$ aws configure
AWS Access Key ID [****************SWUA]: XXXXXXXXXXXXXXXXXXXX
AWS Secret Access Key [****************n2L+]: ****************************************
Default region name [None]: us-east-2
Default output format [None]:
$ ybload -d premdb --username bobr -W -t match --format csv --delimiter ',' --bad-row-file /home/brumsby/s3bad 
s3://ybbob/premdb/match.csv
Password for user bobr: 
14:01:43.158 [ INFO] ABOUT CLIENT:
...
14:01:43.342 [ INFO] Assuming source encoding matches database server encoding: LATIN9
14:01:44.923 [ INFO] Starting source s3://ybbob/premdb/match.csv
14:01:45.053 [ INFO] Using database locale: C
14:01:45.056 [ INFO] Auto-detected line separator = '\n'
14:01:45.189 [ INFO] 
...
14:01:47.665 [ INFO] SUCCESSFUL BULK LOAD: Loaded 8606 good rows in   0:00:04 (READ: 69.12KB/s WRITE: 64.59KB/s)

Load the table again, using ybload --object-store options for authentication: ​

This example loads the same table again but uses ybload options to set the identity, credential, and region values:

$ ybload -d premdb --username bobr -W -t match --format csv --delimiter ',' --bad-row-file /home/brumsby/s3bad 
--object-store-identity 'XXXXXXXXXXXXXXXXXXXX' 
--object-store-credential '****************************************' 
--object-store-region 'us-east-2' 
s3://ybbob/premdb/match.csv
Password for user bobr: 
14:23:20.736 [ INFO] ABOUT CLIENT:
...
14:23:20.807 [ INFO] Gathering metadata on input files
14:23:20.818 [ INFO] Loaded file handler for s3://
14:23:20.857 [ INFO] 
Configuration (S3 client):
   endpoint           : null
   region             : us-east-2
   identity/credential: ********
14:23:20.924 [ INFO] Assuming source encoding matches database server encoding: LATIN9
14:23:22.744 [ INFO] Starting source s3://ybbob/premdb/match.csv
...

Load the table again, using a properties file for authentication: ​

This example loads the same table again but uses a Java properties file to set the identity, credential, and region values. First create the properties file, then name the file with the --object-store-provider-config option:

$ more ybbobrS3.properties

yb.file.region = us-east-2
yb.file.identity = XXXXXXXXXXXXXXXXXXXX
yb.file.credential = ****************************************

$ ybload -d premdb --username bobr -W -t match --format csv --delimiter ',' --bad-row-file /home/brumsby/s3bad 
--object-store-provider-config ybbobrS3.properties 
s3://ybbob/premdb/match.csv
Password for user bobr: 
16:58:02.579 [ INFO] ABOUT CLIENT:
...
16:58:02.658 [ INFO] Gathering metadata on input files
16:58:02.673 [ INFO] Loaded file handler for s3://
16:58:02.713 [ INFO] 
Configuration (S3 client):
   endpoint           : null
   region             : us-east-2
   identity/credential: ********
...