Product Documentation

Appearance

All platforms

Using the Dry Run Option ​

Use the --dryrun option to test the ybload options you are setting before doing the actual load. For example:

$ ybload --dryrun -d premdb -t match --format csv --username bobr -W /home/bobr/match.csv
Password for user bobr: 
17:03:07.065 [ INFO] ABOUT CLIENT:
   app.cli_args         = --dryrun -d premdb -t match --format csv --username bobr -W /home/bobr/match.csv
   app.name_and_version = ybload version 2.0.0-10711
   java.home            = /usr/lib/jvm/java-8-oracle/jre
   java.version         = 1.8.0_101
   jvm.memory           = 981.50 MB (max=14.22 GB)
   jvm.name_and_version = Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
   jvm.options          = -Xmx16g, -Xms1g, -ea, -Dapp.name=ybload, -Dapp.pid=2823, -Dapp.repo=/usr/lib/ybtools/lib, -Dapp.home=/usr/lib/ybtools, -Dbasedir=/usr/lib/ybtools
   jvm.vendor           = Oracle Corporation
   os.name_and_version  = Linux 4.4.0-31-generic (amd64)

17:03:07.144 [ INFO] Gathering metadata on input files
17:03:07.487 [ INFO] Assuming source encoding matches database server encoding: LATIN9
17:03:07.759 [ INFO] Starting 1 source PreReaders
17:03:07.777 [ INFO] Auto-detected line separator = '\n'
17:03:07.784 [ INFO] Configuration (record/field separation):
   --format                           : CSV
   --delimiter                        : ,
   --linesep                          : \n
   --quote-char                       : "
   --escape-char                      : "
   --trim-white                       : false
   --skip-blank-lines                 : true
   --on-missing-field                 : ERROR
   --on-extra-field                   : ERROR
   --on-unescaped-embedded-quote      : ERROR
   Internal scanner                   : RecordScannerQuote_1_1
17:03:07.808 [ INFO] Using database locale: C
17:03:07.930 [ INFO] Bad rows will be written to /home/bobr/match.csv.20190204170307.bad
17:03:07.937 [ WARN] Running with --dryrun. Nothing will be written to database
17:03:07.938 [ INFO] Starting import of 1 files
17:03:08.392 [ INFO] Starting 4 segment readers
17:03:08.393 [ INFO] Opening transaction #1...
17:03:08.395 [ INFO] Opened transaction #1
17:03:08.471 [ INFO] Flushing last of 8606 rows for transaction #1
17:03:08.474 [ INFO] Committing 0 rows into transaction #1...
17:03:08.475 [ INFO] Committed transaction #1 after a total of 292604 bytes and 8606 good rows
17:03:08.477 [ INFO] READ:305.8KB(224.0KB/s). ROWS G/B: 8606/0( 6.16K/s). WRITE:285.7KB(209.3KB/s).  TIME E/R:   0:00:01/ --:--:--
17:03:08.477 [ INFO] SUCCESSFUL BULK LOAD (--dryrun): Loaded 8606 good rows in   0:00:01 (READ: 224.0KB/s WRITE: 209.3KB/s)

Based on this information, you know that you are ready to run the load without the --dryrun option and actually commit rows to the match table:

$ $ ybload -d premdb -t match --format csv --username bobr -W /home/bobr/match.csv
Password for user bobr: 
...

If you use the --dryrun option and your load is not set up correctly, you will see messages that help you identify the problem. For example, in this case, the wrong table name (newmatchstats) is specified, and none of the rows from the source file (match.csv) can be loaded:

$ ybload --dryrun -d premdb -t newmatchstats --username bobr -W /home/bobr/match.csv
Password for user bobr: 
...
17:35:19.139 [ERROR] Unable to separate row into fields for parsing: Too many fields (6 > 5)
Bad row = «1,1992-08-01 00:00:00,2,52,0-1,-  »
...

If necessary, you can go to the specified bad rows file to understand why these rows do not qualify for loading into this table. In this example, the source file contains one more column than the specified table:

$ more match.csv.20180521173518.bad
...
#error: lineByteRange(0-35)
#reason: Too many fields (6 > 5)
1,1992-08-01 00:00:00,2,52,0-1,-  

#error: lineByteRange(35-70)
#reason: Too many fields (6 > 5)
1,1992-08-01 00:00:00,2,55,0-1,-  
...