Package org.hpccsystems.spark

Provides access to data residing in HPCC Systems or Spark environments.

See: Description

Class Summary

Class	Description
FileFilterConverter	A helper class that translates Spark Filters into an HPCC Systems FileFilter.
GenericRowRecordAccessor	An implementation of IRecordAccessor that allows IRecordWriter to access Spark GenericRows.
GenericRowRecordBuilder	An implementation of IRecordBuilder that allows IRecordReader to create Spark GenericRows.
HpccFile	Access to file content on a collection of one or more HPCC clusters.
HpccFileWriter	A helper class that creates a job in Spark that writes a given RDD to HPCC Systems.
HpccRDD	The implementation of the RDD(GenericRowWithSchema) that allows reading a Dataset from HPCC Systems.
PySparkField
PySparkFieldConstructor
RowConstructor
SparkSchemaTranslator	A helper class that translates an HPCC Systems record defintion to a Spark Schema.

Package org.hpccsystems.spark Description

Provides access to data residing in HPCC Systems or Spark environments. The DFSClient from HPCC Systems is used to used to access HPCC Systems data files' metadata including the location and layout of the file, and also requests data file access privilages. An RDD is provided to read the file in parallel by file part. The main classes are:

• Content is the abstract class defining field content. There are concrete classes for each of the different content types.
• FieldType is an enumeration type listing the types of content.
• HpccPart implements the Spark Partition interface.
• HpccFile is the metadata for a file on an HPCC THOR cluster.
• HpccFileException is the general exception class.
• HpccRDD extends RDD(Record) class for Spark.
• HpccRemoteFileReader is the facade for the type of file reader.
• Record is the container class holding the data for a record from THOR.