log
Return log file entries that were recorded in the past 1 to 24 hours or tail one of the logs. You can return logs specific to a single compute blade, the database, the keystore, or the local manager node.
log blade [ 1-24 | all | tail ] [ <chassis>/<bay> | <uuid> | all ] [ <keyword> | "<search_string>" ]
log database [ 1-24 | all | tail ] [ <keyword> | "<search_string>" ]
log install [ <keyword> | "<search_string>" ]
log keystore [ 1-24 | all | tail ] [ <keyword> | "<search_string>" ]
log manager [ 1-24 | all | tail ] [ <keyword> | "<search_string>" ]
log ybcli [ 1-24 | all | tail ] [ <keyword> | "<search_string>" ]
Note: In all cases, 1-24
refers to a specific hour from 1 to 24, where 1
is the last hour and 24
is the last 24 hours. For <chassis>/<bay>
, specify a chassis number and a bay number. For example: 2/1
, 1/15
(without spaces).
All of the log
commands have a search feature. You can append a keyword or a text string to the end of any log
command, and only the log lines that contain the keyword or string are returned. A single keyword does not require quotes, but a string with multiple words does require quotes. The search is not case-sensitive. A single-keyword search will find words with a matching stem. See Log Search Examples.
- blade
Return the log for a specific blade or all blades. For a specific blade, use its chassis and bay number or its
UUID
. (Use thestatus blade
command to get the UUID.) For example:YBCLI (PRIMARY)> log blade tail 0/1 Press ctrl-c to stop tailing log.... 2018-07-10T17:53:42.042404-07:00 ybos-00000000-0000-0000-0000-38B8EBD00C1C YBDB[25486]: 23465685 Warn kernel.Work: waiting for resource RDMA Req: Send Req O on proc 28 2018-07-10T17:53:42.042423-07:00 ybos-00000000-0000-0000-0000-38B8EBD00C1C YBDB[25486]: 23465685 Warn kernel.Work: Cleaned up resource Timeout trackable on proc 21 2018-07-10T17:53:42.042434-07:00 ybos-00000000-0000-0000-0000-38B8EBD00C1C YBDB[25486]: 23465685 Warn kernel.Work: waiting for resource RDMA Req: Send Req O on proc 22 ...
To return logs for all blades (for all hours), use the following command:
log blade all all
You can also specify a number of hours for all blades. For example:
log blade 1 all
- database
Return the log for the database. This command must be run on the primary manager node.
YBCLI (PRIMARY)> log database tail Press ctrl-c to stop tailing log.... LEFT OUTER JOIN sys."vt_flush_tables" f ON f."tableId" = c.oid AND f."databaseName" = current_database() AND NOT f."expired" LEFT OUTER JOIN pg_catalog.pg_stat_all_tables t ON t."relid" = c."oid" WHERE c."relkind" = 'r' ; 2017-03-14 15:05:03.788 PDT 26246 FATAL: password authentication failed for user "ybd" 2017-03-14 15:05:03.788 PDT 26246 DETAIL: Connection matched pg_hba.conf line 96: "host all all 0.0.0.0/0 md5" ...
- install
Return the log for when the installer was last run on the system. For example:
YBCLI(31444) (PRIMARY - yb100-mgr1)> log install The following output is the contents of the log when the Yellowbrick installer was last run on this system. Yellowbrick Data appliance installer v2.0.0 Installing: v0.0.1-11160 Installing using the following parameters: {'core': False, 'fieldreset': False, 'force': False, 'rpmreinstall': False, 'ignoreoff': False, 'factory': True, 'nobackup': False, 'yy': True, 'rpmonly': False, 'rpmdisable': False, 'y': False, 'backup_dir': None, 'reinstall': False, 'fixperms': False} Blackout tenant : YB Blackout cluster : yb00 Blackout duration: 3600 stderr filename : /root/YB-yb00-create-blackout-stderr.out Blackout UUID : 2d46bf92-d500-4e23-845e-1b4e1d6ff5fc Starting backing up existing install Backup location: /mnt/ybdata/backup/backup-2019_02_22-0534-59/ Backup complete Installing YBD appliance version: 0.0.1-11160-release... ... ... System upgrade complete Install complete End of the install log.
- keystore
Return the keystore/encryption audit log (on systems with encrypted drives). For example:
YBCLI (PRIMARY)> log keystore tail Press ctrl-c to stop tailing log.... 2017-10-25T13:59:34.415317-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Executing -> keystore init 2017-10-25T13:59:34.419245-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Requires authentication 2017-10-25T13:59:57.254574-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Executing -> keystore init 2017-10-25T13:59:57.258328-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Requires authentication 2017-10-25T14:00:37.265908-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Executing -> keystore init 2017-10-25T14:00:37.270577-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Requires authentication 2017-10-25T14:04:32.013941-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Executing -> keystore init 2017-10-25T14:04:32.016892-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): keystore: Requires authentication 2017-10-25T14:04:37.550317-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): Keystore: Executing -> keystore unlock 2017-10-25T14:04:37.552872-07:00 yb00-mgr1 journal: YBCLI user ybdadmin (10.20.12.24): Keystore: Already unlocked ...
- manager
Return or tail the log for the local manager node. For example:
YBCLI (SECONDARY)> log manager all 2017-10-31T13:01:01.639817-07:00 yb00-mgr1 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1383" x-info="http://www.rsyslog.com"] rsyslogd was HUPed 2017-10-31T13:01:01.973358-07:00 yb00-mgr1 crmd[3756]: notice: Operation ybdkeystore_stop_0: ok (node=yb00-mgr1.yellowbrick.io, call=1421, rc=0, cib-update=3539, confirmed=true) 2017-10-31T13:01:01.974923-07:00 yb00-mgr1 crmd[3756]: notice: Initiating action 122: start ybdkeystore_start_0 on yb00-mgr1.yellowbrick.io (local) 2017-10-31T13:01:01.975659-07:00 yb00-mgr1 systemd: Reloading. ...
YBCLI (SECONDARY)> log manager tail Press ctrl-c to stop tailing log.... 2017-11-27T19:27:54.168847+01:00 yb00-mgr1 crmd[3725]: notice: Operation ybdfs_notify_0: ok (node=yb00-mgr1.yellowbrick.io, call=5315, rc=0, cib-update=0, confirmed=true) 2017-11-27T19:27:54.225340+01:00 yb00-mgr1 crmd[3725]: error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE 2017-11-27T19:27:54.569690+01:00 yb00-mgr1 ntpd[7901]: Deleting interface #35 BLADE_bond, 192.168.10.1#123, interface stats: received=0, sent=0, dropped=0, active_time=24477 secs 2017-11-27T19:27:54.729028+01:00 yb00-mgr1 systemd: Stopped YellowBrick Vault. ...
- ybcli
Return the
ybcli
command history. For example:YBCLI (PRIMARY - yb100-mgr0)> log ybcli tail Press ctrl-c to stop tailing log.... 2018-12-05 05:35:38 : INFO : Executing -> system appliance version 2018-12-05 05:35:38 : INFO : Completed -> system appliance version 2018-12-05 05:35:43 : INFO : Executing -> health cmp all 2018-12-05 05:35:45 : INFO : Completed -> health cmp all 2018-12-05 11:44:38 : INFO : YBCLI 2.0.0-9737 started 2018-12-05 11:44:47 : INFO : Executing -> help log 2018-12-05 11:45:41 : INFO : Executing -> log ybcli tail ...
Log Search Examples
The following examples show how you can search through the logs by specifying a single keyword or a longer search string. The first example searches the manager logs for the word installed
. Quotes are not required:
YBCLI(16789) (PRIMARY - yb00-mgr0)> log manager 24 installed
2019-04-16T13:41:34.155927-07:00 yb00-mgr0 installer2[10836]: Current version installed: v0.0.1-12607
2019-04-16T14:41:37.304598-07:00 yb00-mgr0 installer2[10894]: Current version installed: v0.0.1-12607
2019-04-16T14:44:41.598975-07:00 yb00-mgr0 installer2[10894]: New YBD appliance package installed. Continuing
The second example searches the manager logs for the string Current version installed
. Quotes are required:
YBCLI(16789) (PRIMARY - yb00-mgr0)> log manager 24 "Current version installed"
2019-04-16T13:41:34.155927-07:00 yb00-mgr0 installer2[10836]: Current version installed: v0.0.1-12607
2019-04-16T14:41:37.304598-07:00 yb00-mgr0 installer2[10894]: Current version installed: v0.0.1-12607
The third example searches the database logs for the string System not ready
:
YBCLI(16789) (PRIMARY - yb00-mgr0)> log database all "System not ready"
2019-04-16 14:50:59.330 PDT 7581 0 WM002 WARNING: System not ready
2019-04-16 14:50:59.601 PDT 7600 0 WM002 WARNING: System not ready
2019-04-16 14:51:00.790 PDT 7869 0 WM002 WARNING: System not ready
...
The fourth example searches the logs for a specific blade and returns all entries that contain fail
, Fail
, and words that begin with fail
or Fail
:
YBCLI(16789) (PRIMARY - yb00-mgr0)> log blade 24 1/1 fail
Chassis: 1
-----------
2019-04-16T14:44:42.960413-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D YBDB[4423]: 1 Inform kernel.LoggingDumper: Cluster Status: DCS: Failed to register with the cluster manager :DCS. Retrying... KE023|Missing registerWorker() response
2019-04-16T14:44:42.960413-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D YBDB[4423]: 1 Inform kernel.LoggingDumper: Cluster Status: DCS: Failed to register with the cluster manager :DCS. Retrying... KE023|Missing registerWorker() response
...
2019-04-16T14:47:52.951151-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D kernel: [ 0.699306] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
2019-04-16T14:47:52.951151-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D kernel: [ 0.699402] pci 0000:ff:12.0: BAR 2: failed to assign [mem size 0x00000040]
...
2019-04-16T14:47:52.951730-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D kernel: [ 9.117878] mgmt_bond: Setting fail_over_mac to active (1)
2019-04-16T14:47:52.951730-07:00 ybos-00000000-0000-0000-0000-38B8EBD0061D kernel: [ 9.176539] psmouse serio1: Failed to reset mouse on isa0060/serio1
Press any key for next section. 'q' to stop
...
Parent topic:ybcli Reference