Skip to content

5.4.12 Yellowbrick Release Notes

Release Version: 5.4.12-41817.d65309c1
Release Date: 11/22/2023

Upgrade Requirements 

Read the following information before beginning an upgrade to Version 5.4. Contact Customer Support to review the process and schedule the upgrade.

Upgrades from Version 5.x

  • Upgrades from 5.3.x versions to Version 5.4 are supported; no special steps are required.
  • Upgrades from 5.2.0, 5.2.1, and 5.2.2 to Version 5.4 require an additional step to update view definitions. Yellowbrick Customer Support will walk you through the process.

Upgrades from Version 4.x Not Supported

  • Version 5.4.3 does not support upgrades from Version 4.x.

Changes in Behavior in This Version

  • 5.4.9

    Behavior of the CHAR Data Type

    The CHAR data type implementation in Yellowbrick 5.x has resulted in some behavioral inconsistencies.

    For details about changes introduced in Version 5.4.9 to address these problems, see Upcoming Behavioral Changes to the CHAR Data Type.

  • 5.4.0

    Default Behavior of WLM Restart Rules

    By default, a subset of recoverable error codes triggers an attempted restart, and an attempted restart occurs only once per query. For details, see "Restarting Queries" in the updated Version 5.4 documentation.

  • 5.4.0

    Deprecation of Legacy ybtools

    The legacy version of ybbackup and ybrestore tools ("BAR1") will no longer be supported in major releases greater than Version 5.x. Only the new version of these tools ("BAR2"), which became available with Version 4.x and 5.x, will continue to be supported. The legacy tools continue to be shipped with Version 5.4.x.

  • 5.4.0

    Deprecated Feature: Restartable Replication

    The restartable replication feature that was introduced as a tech preview in Version 5.2.0 is deprecated and disabled in Version 5.4. This feature is not supported under any circumstances for use on production systems; if you would like to use it purely for testing purposes, contact Customer Support.

  • 5.4.0

    Removing Trailing Whitespace Characters with ybload

    In Version 5.4, the following new ybload options are supported:

    • --trim-trailing-white
    • --no-trim-trailing-white

    For example, if --trim-trailing-white is used, the string 'abc   ' (which contains trailing spaces), is trimmed to 'abc'.

    If you do not specify either of these options, the default behavior is to preserve trailing whitespace characters.

    Note: When you use --format CSV, whitespace characters inside quoted fields are not trimmed.

  • 5.4.0

    Operational Change for ybtools on Windows

    In Version 5.4, ybtools on Windows platforms no longer includes .bat files, such as ybload.bat, ybbackup.bat, ybunload.bat, and so on.

Changes in Compatibility in This Version

  • 5.4.0

    ybtools Compatibility 

    Yellowbrick recommends that you always upgrade ybtools to match the Yellowbrick server version you are running (for example, Version 5.4). Using Version 5.4 of ybtools with an older server version (such as Version 5.2) may result in error messages for some commands. Using previous versions of ybtools with a Version 5.4 database is likely to work but is deprecated and not supported. See also the following section: Backward Compatibility for Backup and Restore.

    Important: On CentOS and Red Hat client platforms, you must first remove any existing 3.x version of ybtools. Then you can proceed with the installation of the Version 5.4 ybtools. You cannot upgrade directly from an earlier 3.x version of ybtools to Version 5.4.

  • 5.4.0

    Replication Compatibility and Upgrade Process

    Important: Both systems used for database replication (source and target) must be running the same major version of the database software. For example, you cannot replicate from a **4.x **source to a **5.x **target or vice versa.

    You can replicate a Version 5.x database to a later 5.x database (such as 5.2 to 5.4). You cannot replicate from a later 5.x source to an earlier 5.x target (such as 5.4 to 5.2).

    If you are upgrading from Version 5.2.x, for example, and you want to reverse replication in 5.4 (using PROMOTE), follow these steps:

    1. Pause the replica on the Version 5.2.x source system.
    2. Upgrade the target system to Version 5.4.
    3. Verify that replication works from 5.2.x to 5.4.
    4. Attempt to promote on the 5.4 target. This operation will fail with an expected error because you cannot replicate to a system that is running an earlier version of the software. The replica will remain paused on the 5.2.x source system.
    5. Upgrade the source system to 5.4.
    6. Attempt to promote again on the 5.4 target system. This operation should succeed.
    7. Attempt a new replication cycle from the reversed source to the new target (both 5.4 systems). The replication cycle should succeed on the new target.
  • 5.4.0

    Backward Compatibility for Backup and Restore

    A backup taken in Version 3.2.x, 3.3.x, 4.x, or 5.x prior to a 5.4 upgrade can be restored in Version 5.4.

    After a 5.4 upgrade or installation, the previous version of the backup and restore tools is compatible with Version 5.4. The Version 5.4 ybtools package installs two sets of backup and restore tools: legacy tools and new 5.4 tools.

    *The legacy tools are deprecated and will be removed in a future release, along with the ability to restore backups taken in previous releases. *There are significant differences in the functionality and client tool options between the legacy tools and the new tools. The tool sets are distinguished by their names:

    ybtools Version Backup Restore List/Manage Backups
    New in Version 4.x and later ybbackup ybrestore ybbackupctl
    Legacy in Version 4.x and later ybbackup-old ybrestore-old ybbackup-list-old
    Version 3.x ybbackup ybrestore ybbackup-list

    To use the legacy tools in the 5.4 package, be sure to use the correct tool name in the command. For example, to return help for the legacy version of ybbackup:

    $ ybbackup-old --help
    Yellowbrick Backup and Restore version 5.4.0-33445
    ...

    Important:

    To use backup and restore in Version 5.4, you must follow these steps:

    1. Use the legacy version of ybtools (-old) to restore any 3.x backups into 5.4 databases. Use the new 4.x or 5.x version of ybtools to restore any 4.x or 5.x backups into 5.4 databases.
    2. Take new full backups of your 5.4 databases with the new 5.4 ybtools.
    3. Stop using the legacy ybtools for backup and restore purposes. Use only the new tools.

What's New in This Version

  • 5.4.12

    Parquet support to INT96 timestamp values

    Interpret INT96 data (without a logical type) as a timestamp to provide compatibility with third-party systems, such as Impala, Hive and Spark.

    New option to ybload:

    console
    --int96-as-timestamp (default)
    --no-int96-as-timestamp
  • 5.4.12

    Improved cluster reconfiguration time

    Cluster reconfiguration time on restart, particularly for appliances with a large number of chassis has been improved.

  • 5.4.12

    DLE_DST and LE_DST Functions

    Return the difference between two character strings in terms of the number of changes required to convert one to the other. The DLE_DST function uses the Damerau-Levenshtein edit distance algorithm and the LE_DST function uses the Levenshtein edit distance algorithm to compute the "distance" between the strings.

  • 5.4.10

    JSON Functions

    Two new SQL functions for JSON manipulation, JSON_INGEST and JSON_LOOKUP, are generally available in Version 5.4.10. See the updated Version 5.4.10 documentation for details and examples.

  • 5.4.10

    Behavior of the CHAR Data Type

    The CHAR data type implementation in Yellowbrick 5.x has resulted in some behavioral inconsistencies. For details about changes introduced in Version 5.4.9 to address these problems, see Upcoming Behavioral Changes to the CHAR Data Type.

  • 5.4.10

    GSSAPI Support in ybtools

    The Yellowbrick client tools (ybtools) now have GSSAPI support, which enables support for Kerberos authentication. To enable and configure Kerberos on a Yellowbrick appliance, contact Yellowbrick Customer Support.

  • 5.4.10

    Maximum Number of Tables Increased to 100,000

    Yellowbrick appliances now support a maximum of 100,000 persistent tables per appliance. The previous limit was 70,000.

  • 5.4.0

    Parquet Format Loads and Unloads 

    The ybload and ybunload client tools support loading and unloading files in Apache Parquet format.

  • 5.4.0

    GROUP BY Extensions

    The GROUP BY clause now supports the GROUPING SETS, CUBE, and ROLLUP extensions. These extensions make it possible to return aggregate results for multiple GROUP BY expressions within one query.

    The GROUPING() function is also supported. You can use this function to identify and filter rows returned by queries that use grouping sets.

  • 5.4.0

    Single-Table Restore Operations

    You can now restore a single database table from a backup, as opposed to restoring all the objects in a database. You can restore a table to its source database or a separate target database or system.

  • 5.4.0

    Table Aliases for SET Columns in UPDATE Statements

    When the target table for an UPDATE is given an alias, that alias can be used in the SET clause as well.

  • 5.4.0

    Number of Table Columns Increased to 2000

    The maximum number of user-defined columns in a table has been increased to 2000.

  • 5.4.0

    New SQL Functions

    INV_NORM: returns an inverse normal distribution x-value, given a known probability (or percentile) and optional mean and standard deviation values.

    ROUND_VAR: a version of the ROUND function that accepts a column name or an expression for its second argument.

    TO_NUMBER: Convert a number string to an actual numeric value, using a specified format.

    yb_get_view_table_usage(): return information about views and the tables they depend on.

  • 5.4.0

    System View Changes

    The sys.restore and sys.log_restore views now have a "table_name" column to log the name of a table that was restored in a single-table operation.

  • 5.4.0

    New ybcli Commands

    The following ybcli commands are new in this release:

    • config cert get and config cert create-selfsigned are new commands that return the status of the current Yellowbrick Data on the host system. If the certificate is self-signed, you can regenerate it.
    • log clang is a new command that returns the log for compiler-related events.
  • 5.4.0

    TLS Support for ybrelay and Spark

    TLS support for ybrelay and Spark is now generally available; this feature is no longer in Tech Preview mode. Note: ybrelay is not supported on Windows clients.

  • 5.4.0

    Setting Configuration Parameters Within Queries

    A new SETTING clause may be specified before a query so that it runs with specific values applied for one or more configuration parameters.

Known Issues in This Version

You may encounter the following issues in Version 5.4.12. Use the workarounds provided in the description and contact Customer Support for additional information.

Issue #Description
27992

Database restore operations sometimes fail and report the following error:

YCATALOGRESTORE: Cannot find mapping for foreign key

If you run into this problem, contact Customer Support for a workaround.

24660

In Versions 5.2.18 and earlier, a rare race condition may occur when using joins that involve spilling, resulting in the query failing and ultimately causing a worker crash. In Version 5.4.x, such a query no longer causes a worker crash when it fails, but it does return an exception.

21423

Databases with names containing UTF-8 characters cannot be dropped via the SMC. If you run into this problem, contact Customer Support for a workaround.

20786

Running EXPLAIN ANALYZE on any write statement (INSERT, UPDATE, DELETE) modifies the underlying table, without warning. To work around this problem, run the EXPLAIN ANALYZE statement inside a transaction so you can roll it back if needed

19886

After changing the spill space percentage on your system (Temporary Space Reservation setting in the SMC), you must stop and restart the database, using ybcli. Otherwise the change will not take effect.

16101 

The SMC does not display row counts in the row store (for example, in the Table Details screen).

14447

If you use a full restore to seed a replica, you must start replication by specifying the chain name in the ALTER DATABASE command. For example:

yellowbrick=# alter database sourcedb alter replica sourcedb_replica start with chain 'February2020';
START REPLICA`

If you do not specify the chain, the first replication cycle will duplicate the seeded table rows. For details about seeding a replica, see the "Database Replication" section in the product documentation.

Issues Fixed in This Version

The following issues are addressed in this version:

ReleaseIssue #Description
5.4.1234145

An issue in the formula for common precision/scale of DECIMALS would make rendering of DECIMALs coming out of UNION ALL nodes use the wrong precision/scale.

5.4.1233374

Fix an issue where a semi join is converted into an unique inner join causing a distribution on the wrong columns, resulting in incorrect results.

5.4.1234174

Some backups could error out with: Abandoned due to client inactivity. This issue is rare and caused by a great number of backup chains in a single database that could delay the backup initialization phase.

5.4.1234031

Fix SMC issue that could cause it to stop working completely. Limit maximum number of rows returned for some queries in order to reduce memory requirements.

5.4.1233236

Improve replication cycle latency when transmitting large number of small tables.

5.4.1233234

CVE-2023-38408: The PKCS 11 feature in ssh-agent in OpenSSH before 9.3p2 has an insufficiently trustworthy search path, leading to remote code execution if an agent is forwarded to an attacker-controlled system. (Code in /usr/lib is not necessarily safe for loading into ssh-agent.) NOTE this issue exists because of an incomplete fix for CVE-2016-10009.

5.4.1232530

datediff did not correctly implement the datepart boundary feature for the following dateparts: hour, minute, second, epoch, millisecond.

5.4.1230776

Allow pg_backend_pid() to be used in more contexts such as in queries containing:

current_database
current_schema
current_user
session_user
version

Previously, trying to use pg_backend_pid() with those functions would return:

ERROR: Function PG_BACKEND_PID is not supported

5.4.1227777

When the ybrelay client requested a commit, the ybrelay server did not wait to receive all of the data to be written to the database, especially when decompression of the data was time-consuming. The shutdown process has been improved so that all the data is committed.

5.4.1228851

Sometimes on a busy system, a query that produces a P0003: too many rows would error out with KE037: Something went wrong try again later.

5.4.1227388

Fix an issue where some complex queries wouldn’t fully utilise the query compilation cache requiring a round of query compilation per invocation.

5.4.1217425

Uniformize handling of negative exponents for non-integer types in the pow function. From this release, an error message such as a negative number raised to a non-integer power yields a complex result is expected as opposed to ‘NaN’.

5.4.1216535

TO_CHAR function call with a string casted to timestamp with time zone as argument would incorrectly return nothing.

5.4.1220824

In some rare instances, the sent_rows field in sys.log_unload would contain more rows than the actual number transmitted.

5.4.1233722

Outage caused by trickle inserts on wide table. The large number of NULL columns (decimal type) led to failures flushing the data to the column store.

5.4.1232535

Backup over SSL failed with "An unexpected exception occurred during ssl handshake on channel" due to connection timeout. New version of ybbackup tool, when used against this version of the YBD server, will retry more types of connection failure including SSL connection timeouts. This behaviour is enabled by default. Up to 3 retries per table are attempted (configurable).

  • --retry-failed-backups: Enable retrying if a table fails to backup during the session. This option is only applicable to YDB instances that also support retrying.

  • --retry-per-table-max: The maximum number of retry attempts for a single table before the program terminates with an error. Default: 3

5.4.1232529

Some types of correlated sub-queries could lead to DB front-end error. In some rare circumstances, it could cause a DB restart. Queries of this type will now error out with: "This form of correlated subquery is not supported"

5.4.1232695

System diagnostics is now faster on multi chassis systems.

5.4.1232924

DESCRIBE and EXPLAIN output of the CURRENT_DATE function changed in 5.4. It has been reverted to its prior behavior.

5.4.1233184, 33092, 33470

Implement limits for the evaluation of complex expressions in a single statement. Exceeding these limits will now result in a query error at compile time.

The Yellowbrick system doesn't enforce the limits by default, they can be set using the following GUC:

  1. expression_complexity_limit
  2. regex_count_limit - Total number of regular expressions that need to be compiled in a single query

The expression complexity limit is the depth of a single expression. The value is expressed in terms of the maximum number of evaluations that may be done at query execution time for each row for a single expression.

select <target list> from <table> where colN = 1 or colN = 2 or colN = 3;

The evaluation of the predicates for the query above has a complexity of 3.

select colA = 1 OR (colB = 2 AND colC = 3), colD = 2 OR (colE = 3 AND colF = 4) from <table> ...;

The evaluation of the expression for the query above has a complexity of 3.

Setting a expression complexity limit of 3 for the query:

select colA = 1 OR (colB = 2 AND colC = 3 AND colE = 4), colD = 2 OR (colE = 3 AND colF = 4) from <table> ...;

yields

ERROR: Expression too complex (expression complexity is 4). Reduce the number of columns or simplify the query

Evaluation of the IN operator for large lists are not affected by the expression_complexity_list, if and only if all elements of the list have the same type of are trivially converted.

For example:

select <target list> from <table> where colA IN (1, 2, /* 1000 elements */...);

succeeds even if the expression complexity limit is set to a low value like 3.

Setting the limits too low may result in failing to optimise queries that would have worked prior to this change.

Recommended values:

  • regex_count_limit: 1000
  • expression_complexity_limit: 5000

Please contact customer support for more information.

5.4.1233381

Cluster manager reconfiguration performance improvements following blade events.

5.4.1233434

Support for patch release rollback.

5.4.1233577

Use of LEFT(f() , N) or RIGHT(f(), N), N < 30 in a CTAS can cause system instability and possible outage.

5.4.1232533

The JSON_LOOKUP function returns a compile error when a NOT NULL constraint is defined on the column used for the first argument.

5.4.1232559

JSON_LOOKUP function doesn't handle key values of an empty string.

5.4.1132420

SMC "Configure > Users" generates the Error: PreparedStatementCallback; uncategorized SQLException...

5.4.1132092

The GETBIT function returned wrong results when used with BIGINT columns.

5.4.1131747

ybbackup backup bundle documentation update.

5.4.1130805

SELECT FROM sys.table_info could return ERROR: OID out of range if a TEMPORARY TABLE had rows in the user rowstore.

5.4.1129699

A CPU or Network saturated manager node could result in LDAP connections failing with Can't contact LDAP server. Connections will now be automatically retried.

5.4.1129683

The alert message for the catalog row store becoming full has been updated to help distinguish it from the user rowstore.

5.4.1126078

A lime restart could result in:

  • some lime queries could fail with the error: KE032 Attempt to use spill space above limit
  • the SMC "Worker CPU Utilization" and "Worker Memory Usage" dashboards not displaying current statistics the sys.log_query view not populated
5.4.1031151

A regression in Versions 5.4.8 and 5.4.9 caused some queries with Protegrity functions to return the following error:

ERROR: Assemble failed
5.4.1030371

A system restart caused by a race condition could occur for nested subqueries.

5.4.1029495

Use of the --log-level and --logfilelog-level ybrelay options were not reflected in the ybrelay logs.

5.4.1029270

Joining on strings where both string columns are larger (>30) but of differing lengths would return wrong results.

5.4.1028973

In stored procedures performing small inserts to the rowstore, all INSERT statements within BEGIN...EXCEPTION...END blocks were ignored until a new block was started or the end of the transaction was reached, causing data to be missing from the results of the full transaction.

5.4.1028137

Under certain circumstances, parity rebuild after a drive replacement would take longer than expected.

5.4.1027810

Running a query with multiple COALESCE columns would sometimes cause out of memory errors and a Postgres crash.

5.4.928655

When output CHAR columns were not referenced consistently in the GROUP BY clause (with or without table qualifiers), the query returned the following error:

ERROR: variable not found in subplan target list
5.4.927628

In some instances, a cursor fetch query in “client wait” state was not canceled during quiesce.

5.4.828053

The number of distinct values was poorly estimated for scans with filters of the form: s

<column> = <constant>

These estimates have been improved in Version 5.4.8.

5.4.827793

Prepared statements with typemod arguments that were executed with a generic plan produced wrong results.

5.4.826268

A hung query in idle state was not present in the sys.query view and would not respond to yb_terminate_session or pg_termination_backend.

In Version 5.4.8, query cancellation has been improved for certain code paths in the planner.

5.4.826155

Under certain conditions, running ybcli commands from the secondary manager node on the Andromeda platform would return the following error:

Can not retrieve this platform's hardware generation
5.4.727956

Joining on strings where one string column is small (<=30) and the second string column is larger (>30) would return wrong results.

5.4.727470

Default WLM restart rules were applied three times and for a large number of error codes. In Version 5.4.7, a smaller subset of error codes triggers a restart, and an attempted restart occurs only once per query.

5.4.727155

A problem with the rewrite order involving CHAR data types would sometimes result in an internal error.

5.4.727147

A query with multiple subqueries containing the concat(*) function caused a system crash when the query was run during planning.

5.4.726365

Executing a stored procedure containing loops and an explicit exception statement would cause Postgres to crash if exception handling was involved.

5.4.726113

A large number of SHOW statements referencing ybd_analyze_after_writes and ybd_analyze_after_loads were logged in sys.log_query, causing OOM and spill space errors. In Version 5.4.7, these statements are not logged.

5.4.726082

In some instances, a NOT restriction of correlated subqueries was incorrectly applied, causing wrong results.

5.4.722873

Cluster reconfiguration during a system restart could, in some rare cases, result in a deadlock because the self-healing mechanism could block indefinitely. In Version 5.4.7, the self-healing mechanism is more deterministic.

5.4.79485

System view queries could return query text strings in GZIP-compressed format. In Version 5.4.7, these compressed strings are correctly decoded.

5.4.626770

Joins on CHAR columns performed poorly because the planner added a redundant TRIM function for variable CHAR strings, which prevented bloom filters from being applied.

5.4.626700

The third-party syslog-ng package was upgraded to fix a slow file descriptor leak. This leak occurred when the internal log files kept on each CMP were rotated, and triggered a DCS alert.

5.4.626680

After an upgrade to Version 5.4.5, some queries required a lot more memory to run (or required more spill space and still ran out of memory).

5.4.626594

A UNION ALL query that selected and constrained sequence values failed with the following error:

Generic UnexpectedException - Can't dlopen
5.4.626487

The documentation was updated to clarify that SUBSTRING and SUBSTR are not equivalent functions. SUBSTRING returns a VARCHAR data type, but SUBSTR returns the data type of its input string.

5.4.626486, 26477

After an upgrade to Version 5.4.5, the rewrite of LEFT joins with a NULL clause into an ANTI join caused a Postgres crash when the planner tried to access an uninitialized field from an unplanned subquery.

5.4.626302

The Yellowbrick installer proceeded as far as the firmware upgrade step, then stopped progressing and failed to time out.

5.4.626248

A distribution logic issue for SEMI LEFT joins caused redundant redistribution.

5.4.625960

After setting a configuration parameter, using SET SESSION ROLE and SET LOCAL ROLE commands in succession within the same transaction caused a Postgres crash.

5.4.625834

The IN-list rewrite optimization now works with all comparison operators. In previous releases, only the '=' and '<>' operators were supported for this optimization.

5.4.625802

The --spark-sql application option did not accept a path that started with "file:". For example, calling spark-submit with the following syntax resulted in a "ParseException" error:

--spark-sql file:parquet.spark.sql
5.4.623274

Rolling back a transaction with an active backup snapshot would sometimes cause a system crash.

5.4.526228

A regression in Version 5.4.x caused some equality conditions on CHAR values with trailing spaces to return wrong results.

5.4.426106

In some cases, the planner subquery estimation logic ignored casts. This behavior caused the wrong operators to be called and the database to shut down.

5.4.425417

On the Andromeda platform, logins to the SMC were slow. In Version 5.2.21, to mitigate this problem, the ybd-telegraf service is disabled by default.

5.4.424899

Instead of being garbage-collected, table deletes were included in full backups, making the backup bundle grow very large.

5.4.325534

Unsupported ybunload operations timed out instead of failing immediately.

5.4.324674

When a regular user ran a cross-database query that referenced a LANGUAGE SQL UDF in a remote database, the query failed with an error:

ERROR: function with OID nnnnnnn does not exist
5.4.323999

Incorrect implicit casting with string functions resulted in a full table scan.

5.4.225229

For security reasons, the google-gson dependency for backup and restore was upgraded to 2.9.1.

5.4.225071

On the Andromeda platform, spill queries assigned to the large resource pool ran out of memory.

5.4.225019

A query with multiple rewritten IN lists that constrained the same constants and referenced these constants in the GROUP BY clause returned the following error:

Variable not found in subplan target list
5.4.224898

CREATE EXTERNAL TABLE failed with the following error after a large number of table creations:

OutOfMemoryError: Direct buffer memory
5.4.224784

A common table expression (CTE) with an ORDER BY clause returned wrong results.

5.4.224779

ACLs defined with ALTER DEFAULT PRIVILEGES on the replication source system were not replicated to the target appliance. To change this behavior, add the following option to the WITH clause of the ALTER DATABASE ADD REPLICA statement:

user_resolution_mode 'PLACEHOLDER'
5.4.224765, 24660

A rare race condition between spilling and join cleaning code resulted in a worker crash. This crash was more likely to occur for join queries that spill to disk under heavy memory pressure.

5.4.224157

A worker crash caused by a race condition could occur while using partitioned tables when join queries involved spilling.

5.4.217888

The ybbackupctl --fix-bundle-paths command always returned an exit code of 0. In Version 5.4.2, if any backup paths cannot be fixed, this command correctly returns an error.

5.4.213312

The documentation was updated to clarify the description of the last_value column in the sys.sequence view.

5.4.124789

When an IN list subquery containing LIMIT and ORDER BY clauses was decorrelated, the IN list rewrite was incorrectly applied, causing wrong results. In Version 5.4.1, decorrelation of IN list subqueries containing LIMIT clauses results in the following error:

This form of correlated subquery is not supported.
5.4.124777

ORDER BY clauses were removed from IN list subqueries that included a LIMIT clause, causing queries to return wrong results.

5.4.024516

When temporary tables were being created, a rare race condition involving the allocation and vacuuming of associated ROWUNIQUE values could cause a system crash.

5.4.024453

During an upgrade to 5.2.18 or earlier, some query_id and session_id numbers were reduced, resulting in duplicated entries in the sys.log_query and sys.log_session views.

5.4.024365, 23953

Version 5.4.0 addresses some vulnerabilities to the SMC, Lime, and Postgres ports by disabling all weak CBC ciphers.

5.4.024255

Views created in Version 5.2.15 and earlier that include CASE expressions would sometimes return errors.

5.4.023977

During an upgrade, some tables appeared to be vacuumed when they were not, sometimes causing SQL statements to return the following error:

ERROR: could not access status of transaction
DETAIL: Could not open file "pg_clog/0000"
5.4.023954, 23952, 23951, 23950

Version 5.4.0 addresses some vulnerabilities in the Yellowbrick ssh configuration.

5.4.023912

An IN list rewrite would sometimes return wrong results because data was not cast to the correct data type.

5.4.023871

During an upgrade, a table containing duplicate keys that returned a duplicate key value error would cause the upgrade to fail.

5.4.023869, 23826

A memory allocation issue would cause a crash in the manager node stack.

5.4.023741

In some instances, a hung monitoring service was not terminated, causing a manager node failover to be unsuccessful and resulting in both manager nodes becoming secondary.

5.4.023694

Insufficient system pool memory would cause some GC operations to stall.

5.4.023307

A hung query was not present in the sys.query view and would not respond to yb_terminate_session, preventing a GC operation.

5.4.023305

When a blade was being booted, the ybcli blade status command incorrectly reported "Cluster Status: ERROR."

5.4.023094

Using ybcli maintenance mode for DNS changes could cause issues with the port listener.

5.4.023064, 22630

After an upgrade to 5.2.13 or 5.2.14, replication source databases unable to reach the target system were put in READONLY mode with no additional attempts to connect to the target. In Version 5.4.0, the system retries connections to the target system for 10 minutes, bringing the source database out of READONLY mode if the target is available.

5.4.023023

Using DISTINCT inside the MEDIAN function would cause the system to crash.

5.4.022986, 22750

Parsing a query with a complex nested case expression would sometimes cause out-of-memory errors and a system crash.

5.4.022886

After an upgrade, some front-end queries would return the following warning:

WARNING: out of shared memory.
5.4.022806

The installation preupgrader could terminate early when validating tables with only system attributes.

5.4.022715

Version 5.4.0 improves the ability to set multiple domains on the network manager.

5.4.022633

Toast tables and indexes related to schema-excluded objects were not excluded during the replication cycle, resulting in orphaned relations and ultimately causing a database restart.

5.4.022613

When executed in an aborted transaction, a ROLLBACK TO SAVEPOINT query would not be removed from the sys.query view.

5.4.022462

A CTAS statement that included a common table expression caused a Postgres crash.

5.4.022308

Character strings inside a NULLIF expression returned wrong results because auto-trimming was not applied to them.

5.4.022218

Queries including conversions of CHAR to DATE data type would sometimes return an out-of-range error.

5.4.022203

Issues during query execution caused simple correlated subquery usage in the predicate to result in a database crash.

5.4.022192

The description of ALTER DATABASE DROP REPLICA was improved in the 5.4 documentation.

5.4.022168

Misleading histogram statistics were being created for certain table distributions, leading to inefficient query plans and slower performance.

5.4.022142, 21881, 20518, 20514, 17683, 14502, 12218, 11324

In Version 5.4.0, performance of query compilation and code generation was improved.

5.4.022117

When a replicated view contained a UDF that was used by a subquery, selecting from the view on the destination database returned a "cache lookup failed" error.

5.4.022113

After an upgrade to 5.2, replication failed on the source system with backup errors because of a problem with duplicate "relfilenodes."

5.4.022106

A query with a very large IN list returned a "KE002: Out of Memory" error.

5.4.022103

A query with an IN list condition containing a GROUP BY, DISTINCT, or ORDER BY operation applied after the cast would return the following error:

ERROR: variable not found in subplan target list
5.4.021933

Using ALTER statements to rename database objects would allow new names to include line breaks, which resulted in errors when such objects were used in a CREATE TABLE AS statement. In Version 5.4.0, newlines are not permitted in object names.

5.4.021892

The pre-upgrade script returned a Python error when the --only flag was used. In Version 5.4.0, you can use this option to run a single check on all databases. You can use the --database option to run all pre-upgrade checks on a single database. You can run a single check on a single database by using both options together. (These command-line options are not supported in ybcli.)

5.4.021737

Failure of a GC operation resulted in a "Metadata storage full error."

5.4.021734

Performance of scans against the sys.fsinfo virtual table was improved.

5.4.021733, 21293

On 3-chassis and 4-chassis systems, the server crashed while running queries against the sys.fsinfo virtual table.

5.4.021722

When implicit casting was used for subqueries that included ORDER BY, GROUP BY, or DISTINCT clauses, the clauses were applied to the incorrect data type, sometimes resulting in a Postgres crash.

5.4.021717

INSERT statements that required implicit casts on numeric constants inserted garbage values into the table.

5.4.021711

An internal setting for worker spill space could not be retrieved and caused a Postgres crash.

5.4.021531, 21510

Improvements were made to the check_xfs.py script to prevent hanging.

5.4.021519

When a backup contained only a large number of table deletes, a ybrestore operation exceeded a configurable timeout in the restore service, causing the restore to error out although it was complete.

5.4.021439

The presence of unicode characters in short character strings could cause incorrect compression and wrong results when the strings were compared.

5.4.021389

Certain join queries returned a Generic UnexpectedException error when they contained a LIMIT clause but no ORDER BY clause.

5.4.021380

Memory for all joins was over-estimated as a result of a previous fix for skew handling.

5.4.021351

ybload failed to round up input data with a fractional second value of more than six digits of precision, resulting in the row being skipped.

5.4.021315

When a bulk load failed, a race condition could cause the SMC to report incorrect status and health for the cluster.

5.4.021290

Under certain conditions, running the ybdumpschema tool took significantly longer than expected.

5.4.021281

On the Andromeda platform, spilling scalability was improved for aggregations.

5.4.021249

To ensure that the Yellowbrick server and client tools are not subject to known log4j vulnerabilities, unused packages and their dependencies were removed from the log4j distribution used by Yellowbrick.

5.4.021214

On the Andromeda platform, a prioritization issue involving heartbeat signals would sometimes cause a heartbeat failure on large clusters.

5.4.021182

For loop joins, the planner sometimes did not choose the smaller table to hold in memory.

5.4.021166

Postgres crashed because of a problem with processing ACL settings when roles were set with the ALTER ROLE command.

5.4.021122

SEMI RIGHT joins could return wrong results because of a race condition. Any query of the following form could trigger this behavior, especially larger, partitioned queries:

SELECT a from foo where a IN (select b from bar);
5.4.021079

When a stored procedure that created a table with a CLUSTER ON clause was executed, Postgres crashed.

5.4.021069

A rare, intermittent worker crash caused by a race condition could occur on the Andromeda platform when queries involved spilling.

5.4.021052, 21008

When the geqo and geqo_threshold configuration parameters were set explicitly for a cursor-related query, the system crashed.

5.4.020972, 20967

A planner optimization prevented the deduplication of init plans, causing excessive memory consumption during query compilation.

5.4.020957

The ASCII function is now included in the Version 5.4 documentation.

5.4.020941

A transaction-consistency check in the FSCK code resulted in an assertion error.

5.4.020896

When a join key was also the target of a constant filter, the selectivity of the join was incorrectly estimated.

5.4.020886

Backups taken over AWS Private Link failed with a timeout error.

5.4.020859

In some cases, ybload could fail with an index out of bounds exception while generating a bad row file.

5.4.020849

The description of the w.resourcePool property for WLM rules was improved in the Version 5.4 documentation.

5.4.020835

The description of the ybbackup --exclude option was improved in the Version 5.4.0 documentation.

5.4.020833

A configuration parameter caused certain queries to return an "unrecognized node type" error.

5.4.020821

In Version 5.4.0, a section was added to the backup and restore documentation to explain how to give users access to encrypted table columns in a restored database.

5.4.020811

For CASE WHEN expressions with CHAR and VARCHAR types, the implicit casting hierarchy was not followed properly. In Version 5.4.0, a VARCHAR type is returned if there is at least one VARCHAR type in the branches of the CASE expression.

5.4.020810

A new configuration parameter was implemented to allow string functions such as TRIM to be ignored when selectivity of a clause that contains these functions is estimated.

5.4.020794

A misconfiguration of the code generation service during system startup would sometimes cause the service to be unavailable.

5.4.020760

A Postgres crash occurred because of problems with processing nested NOT IN subqueries.

5.4.020722

At some point in the cycle, a database restore would hang. All activities dropped to 0, as indicated by the read/write network statistics in the log.

5.4.020712

The used_bytes column in the sys.storage view is described incorrectly in the documentation of earlier versions. The used_bytes value is not a precise sum of other values in the view. You cannot derive the exact amount of space used per worker; used_bytes is an approximation, roughly equal to:

distributed_bytes + replicated_bytes + random_bytes + scratch_bytes + some overhead
5.4.020710

The SMC showed a value of 0 for temporary space when the limit was set to 20%.

5.4.020709

When a column name was used in the flags argument, the REGEXP_LIKE function returned the error "RPC channel broken" causing workers to crash. Version 5.4.0 returns the following expected error for this case:

ERROR: Flags found in REGEXP must be a string literal
5.4.020701

Under certain conditions, an INSERT query that was restarted in a different WLM resource pool could cause data loss. For more details, request the Support Bulletin on this topic from Customer Support.

5.4.020674

A CREATE VIEW statement with a CASE expression now returns VARCHAR columns with the correct width.

5.4.020653

A security issue made it possible for users to drop database objects that they did not own.

5.4.020640

The ybloadcat and ybdumpcat logs were missing during an upgrade. In Version 5.4.0, verbose mode is the default for upgrades so that pg.log files are redirected to the ybinstaller.log.

5.4.020638

The SMC displayed the creator of the WLM profile as the user who activated the profile, rather than the actual user who activated it.

5.4.020624

The output of ybdumpcat now includes the number of rows exported from each table.

5.4.020622

Improvements were made to the preupgrade script to log deletes for system catalog tables.

5.4.020621

Log messages were added to the pg.log when table delete records were inserted for pg_class or pg_type. When a DROP TABLE is executed, check the pg.log for a line of the form:

inserted td record oid <oid of dropped table> in relid 1259...
5.4.020619

Detection of composite primary-key/foreign-key relationships did not take the error margin for distinct values into account, resulting in the wrong join order being chosen.

5.4.020593

The number of distinct values was poorly estimated for expressions of the form:

func(var) = <const>

These estimates have been improved in Version 5.4.0.

5.4.020589

Using the ybd_analyze_after_writes configuration parameter returned an unexpected warning, which could interrupt automation scripts.

5.4.020586

For primary-key/foreign-key join estimation on composite keys, the cardinality was estimated as the size of the primary key, not the foreign key.

5.4.020577

The TO_NUMBER function returned a compile error when a NOT NULL constraint was defined on the column used for the first argument.

5.4.020538

A rare race condition could cause a restore or replication failure.

5.4.020517

The Infiniband route-determination algorithm was changed from Min Hop to LASH to prevent issues in large multi-chassis systems.

5.4.020492

Database restore operations could hang when network environment settings caused the termination of connections that were idle for a long time. In Version 5.4.0, the length of time that a network connection can remain idle for large data transfers has been reduced.

5.4.020465

The sys.cluster view logged duplicate rows for the same worker with multiple roles. Because of this problem, the cluster failed to start.

5.4.020458

Join queries run under low-memory conditions (in a slot only a little larger than the estimated memory requirement) triggered repeated spill cycles and failed with an assertion error.

5.4.020450

The compiler was overly aggressive in its memory consumption for large queries.

5.4.020448

A query against the sys.query view with an IN list condition returned wrong results.

5.4.020407

The NVL/COALESCE function returned wrong results because trailing space characters were not trimmed correctly.

5.4.020377

Entries from yb_deletes tables were included in sys.aalog (AutoAnalyzeLog), resulting in a reduction of disk space and slower auto-analyze performance.

5.4.020376

Database users granted "Manager" privileges in the SMC were unable to view queries executed by other users.

5.4.020356

Under memory pressure, a rare issue occurred with compute blades writing compiled queries to disk.

5.4.020310

In some instances, queries that should have been restartable were unable to restart.

5.4.020253

Several system views with privilege checks incorrectly omitted tables in other databases for non-superusers.

5.4.020172

DECIMAL values lost precision when implicit casting was used for subquery results in set operations, such as UNION and UNION ALL.

5.4.020015

An upgrader issue would prevent autovacuum from running on some tables, sometimes causing Postgres to emit the following warning:

WARNING: some databases have not been vacuumed in over 2 billion transactions
5.4.020014

Spilling a join that is skewed would sometimes result in an out-of-memory (OOM) condition if the skew value exceeded the memory.

5.4.019992

Version 5.4.0 increases memory requirements for all queries with table scans.

5.4.019973

Changes to join ordering introduced in Version 5.2.0 caused unintended changes to joins used in views. For example, LEFT joins became FULL joins. In Version 5.4.0, the original join ordering is restored for views.

5.4.019889

Version 5.4.0 improves cardinality estimations of composite keys.

5.4.019072

The Postgres acl_explode() function returned the following error when the ACL abbreviation "Z" was part of the input to the function:

SQL: ERROR: unrecognized aclright:8388608
5.4.019011

When parquet files with unsigned INT64 data were loaded via ybrelay and Spark, the load failed with the following error:

Parquet types not supported: INT64 (UINT_64)

To load unsigned INT64 data in Version 5.4.0, you can use the new ybload feature for loading parquet files, or you can upgrade to Spark 3.2.

5.4.018345

Under certain conditions, running the ybcli system status command on the Andromeda platform would incorrectly report the platform as Tinman.

5.4.018058

When queries failed with out-of-memory errors, a race condition resulted in a worker crash.

5.4.017916

The system could be left in a hung state when queries with very complex regular expressions were submitted. In Version 5.4.0, when a regex expression is too complex, the query returns an error that specifies the regex pattern that caused the problem:

Regex '<pattern>' is too complex. Exceeded maximum number of state transitions

In addition, "yielding" inside the regex engine has been introduced, such that before a regex pattern is declared as "too complex," system stability is improved for complex patterns that do not reach the maximum number of state transitions.

5.4.017695

Under certain conditions, compiling queries with large IN lists would return an out-of-memory error.

5.4.017371

A CTAS query that returned an error during parsing or planning logged a type of "create table as" in sys.log_query instead of the expected "ctas" type.

5.4.016831

Casting a two-digit year CHAR string stored in a table to a DATE data type failed with the following error:

ERROR: datetime field overflow
5.4.016806

After a manager node failover, a rare race condition could cause the logging service of the secondary node to stop working, resulting in a loss of logs on that node.

5.4.016474

When the result of an IN list subquery required a cast, the query failed with the following error:

Missing constant for test expression

In Version 5.4.0, the cast that is applied to the result of an IN list subquery is pushed down into the subquery.

5.4.015970

Queries with CASE WHEN expressions would sometimes become stuck in lime in a prepare state.

5.4.014178

Running the DESCRIBE command on tables with VARCHAR set as the default displayed an unnecessary cast.

5.4.011635

A query with a large number of OR clauses failed after fifteen minutes with the following error:

ERROR: Query too complex.
5.4.08719

Version 5.4.0 improves the compile time for queries containing nested subqueries.

CVE Fixes in This Version

The following CVEs are addressed in this version:

ReleaseCVE FixDescription
5.4.0CVE-2021-32027
5.4.0CVE-2021-23222
5.4.0CVE-2021-23214
5.4.0CVE-2019-10130
5.4.0CVE-2017-7548
5.4.0CVE-2017-7484
5.4.0CVE-2016-5424
5.4.0CVE-2016-5423
5.4.0CVE-2016-3065
5.4.0CVE-2016-2193
5.4.0CVE-2016-0773

Parent topic:Yellowbrick Documentation