Product Documentation

Appearance

EE: All cloud platforms
CE

CREATE EXTERNAL STORAGE ​

Create an external storage object, which can be referenced in the definition of specific external storage locations. An external storage object contains metadata for connecting to a specific object store, including the type, endpoint, and access credentials.

CREATE EXTERNAL STORAGE [ IF NOT EXISTS ] <name>
TYPE <storage_type>
[ ENDPOINT 'endpoint_uri']
[ REGION 'region_string']
[ IDENTITY 'identity_string']
[ CREDENTIAL 'credential_string']
[ WITH ADVANCED (<name> <value> [,<name> <value>]...) ]
IF NOT EXISTS

Create the object if it does not already exist. If it does exist, do not create it and do not return an error.

TYPE

Storage type: S3, AZURE, AZDL2, or MINIO

ENDPOINT

S3 endpoint URIs must be of the following form, using https:// or http://:

https://s3.us-east-1.amazonaws.com

or:

http://s3.us-west-1.amazonaws.com

Note that these endpoints are standard region-based S3 endpoints that begin with https://s3. or http://s3.

These endpoint URIs are not prefaced with a bucket name. See the AWS Documentation for a complete list of regional endpoints.

Azure and AZDL2 endpoint URIs must be of the following form:

https://<storage_account_name>.blob.core.windows.net/
REGION

For example, in S3: us-east-2 or us-west-1

For Azure object storage, do not specify a region.

IDENTITY

Access key ID for S3 endpoints, or a storage account name for Azure.

If you are going to load from a public bucket or container, you can connect anonymously and do not need to specify this option.

CREDENTIAL

Secret access key for S3 or Azure endpoints. If you are going to load from a public bucket or container, you can connect anonymously and do not need to specify this option.

WITH ADVANCED

Advanced configuration options that vary by object store provider. For AWS S3, you can specify the following Boolean options:

  • Default to false:

    dualStackEnabled
accelerateModeEnabled
pathStyleAccessEnabled

  • Default to true:

    checksumValidationEnabled
chunkedEncodingEnabled

For example:

with advanced(dualStackEnabled true, chunkedEncodingEnabled true)

You must have the correct privileges to run this command. See ON EXTERNAL object.

Note: If the type of external storage is AZURE or AZDL2, then the SAS token should be generated at the storage account level, not at the container level.

Examples ​

Create an Azure external storage object with login credentials:

create external storage "premdbAzure" type azure
endpoint 'https://trebor.blob.core.windows.net/'
identity 'trebor'
credential 'MPuQsUdSiyxBSrJxV354dGSUukNgyYwTwrOiXYZ6nSiUWsJMQNMS+HB/LK44dpqI+mgv7vRbAimWuajKIuHI6T==';

Create an S3 external storage object with login credentials:

create external storage premdbs3
type S3
endpoint 'https://s3.us-east-1.amazonaws.com'
region 'us-east-1'
identity 'ABIA4CDEFG6CUDXUY7NM'
credential '2jr0tnuNRo5Xi0Kh8WxapWDB2S+ojzVc4Jd6AMe4';

Create a similar external storage object but for anonymous use:

create external storage my_public_s3_bucket
type S3 
endpoint 'https://s3.us-west-2.amazonaws.com'
region 'us-west-2';