S3 Authentication Methods

There are several different ways to authenticate when you are using ybunload to unload data from AWS S3 or S3-compatible object storage. You can authenticate implicitly by using supported S3-specific mechanisms, or explicitly by using ybunload command-line options. Your organization's S3 administrator should provide instructions for the approach you should use. See Best Practices for Managing AWS Access Keys for further recommendations.

S3 credentials must be provided in a manner supported by ybunload and the AWS Java SDK:

• Secure methods (integrated into your organization's login/identity mechanism):
  • EC2 roles (when running on Amazon EC2 instances)
  • SAML 2.0-compatible identity provider
  • Custom identity provider bridge to AWS
• Other methods:
  • Object Storage Options
  • Environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Other AWS environment variables are also supported, including AWS_SESSION_TOKEN, AWS_PROFILE, AWS_REGION, and AWS_CREDENTIAL_PROFILES_FILE.
  • URI query parameters: aws_access_key_id and aws_secret_access_key
  • A credential file, typically located at ~/.aws/credentials (location may vary by platform)
• Required Permissions
  • Note that ybunload requires access to write the object.

An installation of the AWS CLI is not required, but it does provide the aws configure command, which is useful for setting credentials. For details, see the AWS Command Line Interface documentation.

Order of Precedence for Authentication Methods

You may have multiple credential settings available to you, based on your AWS account setup and how you script the ybunload command. Therefore, it is important to know which authentication mechanism takes precedence when the ybunload command is run. Note that credentials set explicitly with ybunload object storage options, if set (either on the command line or in a properties file), always take precedence over implicit credentials set via aws configure or other methods.

The order of precedence is as follows:

1. Access key and access key ID specified with the --object-store-identity and --object-store-credential command-line options
2. Access key and access key ID specified with the yb.file. prefix in one of the following:
   1. URI parameter
   2. Properties file named with the --object-store-provider-config command-line option
3. Access key and access key ID without the yb.file. prefix in one of the following:
   1. URI parameter
   2. Properties file named with the --object-store-provider-config command-line option
4. Implicit authentication via environment variables and supported AWS configuration files: ~/.aws/* files: (~/.aws/credentials or ~/.aws/config)