Loading Tables with Spark ​

This section describes how to bulk load data from source files that ybload cannot read directly, such as Avro and ORC files. Regardless of its source location (HDFS, NFS, S3, and so on), you can load data in different formats by running Apache Spark jobs that use the Yellowbrick ybrelay service to call ybload. Exported data flows from the Spark application platform to a ybrelay server running the ybrelay service, then is loaded into Yellowbrick database tables using ybload operations.

Follow these steps to bulk load Yellowbrick tables via Spark and ybrelay. Subsequent sections explain these steps in detail and provide examples.

1. Install and set up Apache Spark and the ybrelay service.
2. Define the parameters for a spark-submit command:

• Native Spark options
• Spark application options:
  • Yellowbrick database connectivity
  • ybrelay connectivity
  • General options
  • ybload options, if needed

3. Run the spark-submit command.
4. Monitor the resulting ybload operation.

In This Section

• Setting Up the ybrelay Service

• Setting up and Running a Spark Job

• ybrelay Options

• Spark and ybrelay Glossary

Parent topic:Loading Tables