Metering ​

Yellowbrick cloud platforms are licensed and billed by quantity of compute virtual CPU (vCPU) consumed by elastic compute clusters. vCPU is consumed when a cluster is resumed, live and able to execute queries. The metering utilities in Yellowbrick allow you to write queries to evaluate who resumed and paused compute clusters, how long clusters have been running, the capacity and vCPU consumption of clusters, and even estimations of vCPU usage by query to provide transparency and help understand consumption.

Metering Overview ​

Metering is the process that takes transactional data (for example, node count, start and end time of cluster, core count, log queries) from sys schema and loads the data into respective views by performing certain operations (for example vcpu_seconds calculations) according to business rules. Hourly reports are generated, incorporating the utilization of vcpu_seconds based on the views. Metering can be thought of as a traditional ETL process which populates a local data mart with a dimensinoal model for reporting.

Metering data is stored for a continuous year, encompassing the entirety of the present year and the preceeding 12 months (from 1st of January of the prior year). This allows users to predict usage patterns and compare the current year with previous year.

• The data to be analysed from the source (sys schema) is extracted into temporary data storage called staging.
• The data modelling (denormalising tables, cleaning up bad data, standardising data layouts and so on) is performed on staging to populate the dimension and fact tables.
• In metering schema, multiple views are created based on several tables from fact and dim. These views are created by aggregating several values from the various fact and dim tables which provide quick information about the consumption and usage of cluster vCPU and so on. This is used for billing the customers.

Terminology ​

Instance ​

• An instance is a container of databases, users/roles, all other related data and metadata. It has an IP address, hostname and is the “service” you connect to when you connect to Yellowbrick. Each instance has a special database called yb_metering which contains all relevant billing data.

Cluster ​

• Instances contain compute clusters. The users with the right privileges can create, drop, resume and suspend clusters. Compute clusters are the “work horses” that execute queries on behalf of users.
• All queries executed against a Yellowbrick instance are executed on a compute cluster.

Node ​

• A cluster contains at least one node, which runs on a hardware instance.

Hardware Instance Type ​

• The type of machine that a cluster runs on is called a hardware instance. The cluster has a name that's recognised by the cloud (cloud_provider) and it also has a name that's recognised by Yellowbrick (provider_name). The hardware instance type is recognised by combination of these two fields.

The hardware_instance_type is a sys schema view that returns a list of available hardware instance types.

vcpu_cores ​

• A hardware instance type contains vcpu_cores. Depending on cloud provider, the number of vcpu_cores may vary.
  For example,
  • In Azure,
    • A small v1 cluster with one node has 16 vcpu_cores.
    • A large v1 cluster with one node has 80 vcpu_cores.
  • In AWS,
    • A small v1 cluster with one node has 16 vcpu_cores.
    • A large v1 cluster with one node has 96 vcpu_cores.

vcpu_seconds ​

• A vCPU second is one CPU core online, for one second. Fractional seconds consumed are rounded up (for example, consuming 1.5 seconds of 1 CPU core online counts as 2 vcpu_seconds consumed). The vCPU usage is unaffected by the type of vCPU you deploy in the cluster.
• vcpu_seconds calculation is based on the number of cores in the hardware instance. For example, A query running for 5 seconds on the instance with a small cluster (1 node with 16 cores) should calculate vcpu_seconds as 5x1x16 = 80 sec.

vcpu_estimate ​

• Running queries does not directly consume vCPU. The customers are only billed for the cluster being online, not by the queries that run on that cluster. However, it can be useful to understand what queries are consuming the most CPU so you can properly scale the running clusters and optimise vcpu_seconds.

Metering Views ​

• The metering schema views allow customers to query how they will be billed. It also measures which users and clusters consume the most vCPU and when that vCPU was consumed. The views are located in the yb_metering sys database.

• metering.vcpu_consumption - contains the vcpu_seconds used for billing.

• metering.vcpu_estimate_per_query - contains the vcpu_estimate for the queries using the most vCPU on a running cluster.

The metering scehma views can be queried directly with SQL or data can be extracted to reporting tools.