Redshift Source implementation for Spark SQL
Wrapper of SQLContext that provide redshiftFile
method.
Add write functionality to DataFrame
Input format for text records saved with in-record delimiter and newline characters escaped.
Input format for text records saved with in-record delimiter and newline characters escaped.
For example, a record containing two fields: "a\n"
and "|b\\"
saved with delimiter |
should be the following:
a\\\n|\\|b\\\\\n
,
where the in-record |
, \r
, \n
, and \\
characters are escaped by \\
.
Users can configure the delimiter via RedshiftInputFormat$#KEY_DELIMITER.
Its default value RedshiftInputFormat$#DEFAULT_DELIMITER is set to match Redshift's UNLOAD
with the ESCAPE option:
UNLOAD ('select_statement') TO 's3://object_path_prefix' ESCAPE
org.apache.spark.SparkContext#newAPIHadoopFile