com.github.mjakubowski84.parquet4s.parquet
Members list
Type members
Classlikes
Attributes
- Supertypes
- Self type
-
rotatingWriter.type
Value members
Concrete methods
Creates a fs2.Stream that reads Parquet data from the specified path. If there are multiple files at path then the order in which files are loaded is determined by underlying filesystem.
Path can refer to local file, HDFS, AWS S3, Google Storage, Azure, etc. Please refer to Hadoop client documentation or your data provider in order to know how to configure the connection.
Can read also partitioned directories. Filter applies also to partition values. Partition values are set as fields in read entities at path defined by partition name. Path can be a simple column name or a dot-separated path to nested field. Missing intermediate fields are automatically created for each read record.
Allows to turn on a projection over original file schema in order to boost read performance if not all columns are required to be read.
Builder allows to create a stream of data of given type or of generic records.
Creates a fs2.Stream that reads Parquet data from the specified path. If there are multiple files at path then the order in which files are loaded is determined by underlying filesystem.
Path can refer to local file, HDFS, AWS S3, Google Storage, Azure, etc. Please refer to Hadoop client documentation or your data provider in order to know how to configure the connection.
Can read also partitioned directories. Filter applies also to partition values. Partition values are set as fields in read entities at path defined by partition name. Path can be a simple column name or a dot-separated path to nested field. Missing intermediate fields are automatically created for each read record.
Allows to turn on a projection over original file schema in order to boost read performance if not all columns are required to be read.
Builder allows to create a stream of data of given type or of generic records.
Type parameters
- F
-
effect type
Attributes
- Returns
-
Builder of the fs2.Stream
Builds a fs2.Pipe that:
Builds a fs2.Pipe that:
- Is designed to write Parquet files indefinitely
- Is able to (optionally) partition data by a list of provided fields
- Flushes and rotates files after given number of rows is written to the partition or a given time period elapses
- Outputs incoming message after it is written but can write an effect of provided message transformation.
Builder allows to create a pipe for given data type or for generic records.
Type parameters
- F
-
effect type
Attributes
- Returns
-
fs2.Pipe builder
Builds a fs2.Pipe that writes Parquet data to single file at the specified path (including file name). The resulting stream returns nothing, that is, it doesn't emit any element.
Path can refer to local file, HDFS, AWS S3, Google Storage, Azure, etc. Please refer to Hadoop client documentation or your data provider in order to know how to configure the connection.
Builder allows to create a pipe for given data type or for generic records.
Builds a fs2.Pipe that writes Parquet data to single file at the specified path (including file name). The resulting stream returns nothing, that is, it doesn't emit any element.
Path can refer to local file, HDFS, AWS S3, Google Storage, Azure, etc. Please refer to Hadoop client documentation or your data provider in order to know how to configure the connection.
Builder allows to create a pipe for given data type or for generic records.
Type parameters
- F
-
effect type
Attributes
- Returns
-
fs2.Pipe builder