Data Warehouse Instances

A Yellowbrick instance is a deployed and provisioned data warehouse that runs in a SaaS cloud environment. After logging into the Yellowbrick Manager for the first time, you will be prompted to create your first instance.

You can create multiple instances in order to set different hardware limits for different users and workloads. Each instance defines the region (for AWS installations) and the supported hardware instance types (optional limits for creating hardware resources that will be allocated to and shared among virtual compute clusters).

You can reserve some number of nodes when you create instances.

There is also a per-instance database storage option. If you accept the default behavior for Create internal external storage, object storage for the data stored in your databases and tables will be created automatically. If you deselect this option, you have to create your own "primary storage" by using the CREATE EXTERNAL STORAGE and CREATE EXTERNAL LOCATION commands.

For example:

Local Storage refers to the size of the SSDs (that is, a limit on the data cache, not physical storage).

For example:

Suspending and Resuming Instances

To manage cost for cloud-based hardware resources, it is a best practice to suspend a data warehouse instance when it is not in use for some period of time. Suspending an instance implies the suspension of all of its compute clusters, regardless of the suspension and timeout policy set for those clusters and their current state. How long you want to allow an unused instance to remain up and running is your choice, as long as you monitor the cost. In addition to considering the cost, keep in mind that it takes a little while to resume an instance once it has been suspended.

On the Yellowbrick Manager Dashboard, you can easily see how long each instance has been up and running:

You can click the pause button next to the run time to suspend an instance. The run time is not cumulative; it restarts from 0 when you suspend and resume an instance.

The Idle Timeout value is subject to an additional grace period. For example, if you set Idle Timeout to 30 minutes and Idle Grace Period to 1 hour, the instance will not automatically suspend until an idle time of 90 minutes has passed. The idle timeout is not enforced until the grace period has elapsed.

An instance is considered idle when no user-submitted operations (create and alter cluster operations, backend queries, bulk loads, backups, and so on) are running on clusters that belong to the instance. Front-end database queries and system work that runs in the background do not apply to the idle timeout thresholds.

A cluster that is running but idle does not prevent an instance from auto-suspending. The instance suspends its clusters when it auto-suspends.

Note that you cannot change the auto-suspend behavior of an instance if the instance is currently suspended.

Changing Hardware Instance Limits

After creating an instance, you can change the number of reserved nodes and/or the maximum number of nodes available to the clusters in the instance. This option provides a way to scale an instance up or down.

Reserved Node Capacity

The Reserved field in the Create Data Warehouse Instance screen is a means of reserving and spinning up the specified number of compute nodes when an instance is created, as opposed to waiting for nodes to spin up when compute clusters are created and started. Reserved nodes are shared among all the compute clusters that are created for a single Yellowbrick instance, and they are immediately available for use when those clusters start running and doing work. (Reserved nodes that are not part of a running cluster remain available but are not billed.)

The Limit field sets the maximum node capacity for the instance, while the Reserved field sets the reserved capacity within that limit. The limit per instance must be greater than or equal to the number of reserved nodes.

This approach to reserving nodes is driven by K8S auto-scaling, which causes AWS to spin up nodes during instance creation, before any clusters have been created for the instance. The reserved nodes feature is independent from, but complementary to, the creation of on-demand capacity reservations (ODCRs) in AWS.

There is no direct integration between these two features, but you can use Yellowbrick reserved nodes or ODCRs (or both) to expedite the availability of compute resources. However, note that ODCRs belong to an AWS account and have a global application. This means that they are available to all Yellowbrick data warehouse instances as well as associated Yellowbrick services that run in their own EC2 instances (shared services, bulk load servers, and so on). Also, ODCRs are available (in theory) to non-Yellowbrick software that may be driven from the same AWS account. The presence of an ODCR in AWS facilitates the provisioning of reserved nodes via K8S auto-scaling. Nodes reserved explicitly for a Yellowbrick instance in Yellowbrick Manager apply only to that instance and only to the nodes created for compute clusters.