package
partition
Type Members
-
abstract
class
PartitionedOutputFormat[P, K, V] extends FileOutputFormat[P, (K, V)]
-
case class
PartitionedSink[P, K, V, B](subsink: DataSink[K, V, B], format: Class[_ <: PartitionedOutputFormat[P, K, V]], path: String, partition: (P) ⇒ String, overwrite: Boolean = false, check: OutputCheck = Sink.defaultOutputCheck, compression: Option[Compression] = None) extends DataSink[P, (K, V), (P, B)] with Product with Serializable
Value Members
-
This format creates a new record writer for each different path that's generated by the partition function Each record writer defines a specific OutputCommitter that will define a different work directory for a given key.
All the generated paths will be created under temporary dir/sink id in order to collect them more rapidly with just a rename of directories (see OutputChannel)