Product Documentation

Appearance

ybload Object Storage Options ​

The following options are used to specify connectivity, credentials, and configuration settings for object storage systems, including Azure Blob storage and S3 object stores.

Note: The usage and requirements for these options depends on the object store provider.

--object-store-endpoint VALUE

The endpoint URI to use when connecting to an object store.

For AWS S3, if you specify the --object-store-region option, you can omit the --object-store-endpoint option. An example of an AWS S3 endpoint is:

https://s3.us-east-2.amazonaws.com

An example of an on-premises S3-compatible endpoint is:

http://ybload1.yellowbrick.io:9000

where the host name is a system where the S3 provider is installed. (The port number varies by S3 provider.)

For Azure Blob storage, if you specify the --object-store-identity option, you can omit the --object-store-endpoint option. An example of an Azure endpoint is:

https://ybbobr.blob.core.windows.net

where ybbobr is the storage account name.

You can only access one object store per ybload operation. For example, you cannot specify connection details for both an Azure client and an S3 client in the same command, and you cannot load data from multiple Azure endpoints or storage accounts.

--object-store-region STRING

The region to use when connecting to an object store.

For AWS S3, an example of a region is us-east-2.

For on-premises, S3-compatible object storage providers, ybload requires a region value to be specified even if the provider does not support or require regions. You can specify any string for the region, such as noregion or dummy. Alternatively, you can set the region in an AWS configuration file:

~/.aws/config

[default]
region = dummy

The region option is not supported for Azure Blob storage.

--object-store-profile STRING

The credential profile to use when connecting to an object store. This option overrides the settings for --object-store-identity and --object-store-credential.

This option is not supported for Azure Blob storage.

--object-store-identity STRING

The identity to use when connecting to an object store.

For AWS S3, this identity corresponds to your AWS Access Key ID. For example:

ABCD1EF2GHIJKLMN3OPQ

For Azure Blob storage, this identity is a storage account name. If you specify this option, you can omit the --object-store-endpoint option because the storage account name is always part of the standard endpoint pattern (https://<storage account>.blob.core.windows.net). For example:

https://ybbobr.blob.core.windows.net

where ybbobr is the storage account.

--object-store-credential STRING

The credential to use when connecting to an object store.

For AWS S3, this credential corresponds to your AWS Secret Access Key.

For Azure Blob storage, this credential is a storage account access key or a generated SAS token.

--object-store-provider-config JAVA_PROPERTIES_FILE

The name of a Java properties file that contains provider-specific configuration settings for an object store. For example: ybbobr_az.properties or ybbobr_s3.properties. Default: {}.

Java Properties File ​

If you choose to specify object storage options in a properties file, drop the --object-store- prefix from the ybload option names. Optionally, you can add a yb.file. prefix to the entries in the file.

AWS S3

For example:

$ more ybbobrS3.properties

yb.file.region = us-east-2
yb.file.identity = ABCD1EF2GHIJKLMN3OPQ
yb.file.credential = ****************************************

This properties file is distinct from the ~/.aws/* configuration files that the S3 SDK consults.

Azure Blob storage

endpoint = https://mystorageaccount.blob.core.windows.net identity = mystorageaccount credential = myStorageAccountKey

For example:

$ more ybbobrAz.properties

yb.file.endpoint = https://ybbobr.blob.core.windows.net yb.file.identity = ybbobr yb.file.credential = ****************************************

## S3 Pass-Through Options {#section_h5t_25k_llb .section}

The following pass-through options are supported for S3 clients. You can specify them either in a Java properties file \(referenced by `--object-store-provider-config`\) or as URI parameters in the *first* S3 URI passed to the `ybload` command. \(The defaults listed here are as defined by the AWS Java SDK 2.7.31.\)

accelerateModeEnabled
:   Defaults to `false`.

checksumValidationEnabled
:   Defaults to `true`.

chunkedEncodingEnabled
:   Defaults to `true`.

dualStackEnabled
:   Defaults to `false`.

**Parent topic:**[Loading Tables from Object Storage](../bulk_loading/ybload_from_object_storage.md)