DataStream
Related Docs:
object DataStream
| package datastream
Returns the Schema for this stream.
Returns the Schema for this stream. This call will not cause a full evaluation, but only the operations required to retrieve a schema will occur. For example, on a stream backed by a JDBC source, an empty resultset will be obtained in order to query the metadata for the database columns.
Joins two streams together, such that the elements of the given frame are appended to the end of this streams.
Joins two streams together, such that the elements of the given frame are appended to the end of this streams. This operation is the same as a concat operation. This results in having numPartitions(a) + numPartitions(b)
Returns a new DataStream with the new field of type String added at the end.
Returns a new DataStream with the new field of type String added at the end. The value of this field for each Row is specified by the default value.
Returns a new DataStream with the given field added at the end.
Returns a new DataStream with the given field added at the end. The value of this field for each Row is specified by the default value. The value must be compatible with the field definition. Eg, an error will occur if the field has type Int and the default value was 1.3
Action which results in all the rows being returned in memory as a Vector.
Returns a new DataStream where k number of rows has been dropped.
Returns a new DataStream where k number of rows has been dropped. This operation requires a reshuffle.
Filters where the given field name matches the given predicate.
For each row in the stream, filter drops any rows which do not match the predicate.
Execute a side effecting function for every row in the stream, returning the same row.
Action which returns a scala.collection.CloseIterator, which will result in the lazy evaluation of the stream, element by element.
Combines two frames together such that the fields from this frame are joined with the fields of the given frame.
Combines two frames together such that the fields from this frame are joined with the fields of the given frame. Eg, if this frame has A,B and the given frame has C,D then the result will be A,B,C,D
Each stream has different partitions so we'll need to re-partition it to ensure we have an even distribution.
Returns a new DataStream which contains the given list of fields from the existing stream.
Foreach row, any values that match "from" will be replaced with "target".
Foreach row, any values that match "from" will be replaced with "target". This operation applies to all values for all rows.
Replaces any values that match "form" with the value "target".
Replaces any values that match "form" with the value "target". This operation only applies to the field name specified.
For each row, the value corresponding to the given fieldName is applied to the function.
For each row, the value corresponding to the given fieldName is applied to the function. The result of the function is the new value for that cell.
Returns a new DataStream where only each "k" row is retained.
Returns a new DataStream where only each "k" row is retained. Ie, if sample is 2, then on average, every other row will be returned. If sample is 10 then only 10% of rows will be returned. When running concurrently, the rows that are sampled will vary depending on the ordering that the workers pull through the rows. Each partition uses its own couter.
Returns a new DataStream with the same data as this stream, but where the field names have been sanitized by removing any occurances of the given characters.
Action which results in all the rows being returned in memory as a Vector.
Action which results in all the rows being returned in memory as a Vector. Alias for 'collect()'
Returns the same data but with an updated schema.
Returns the same data but with an updated schema. The field that matches the given name will have its datatype set to the given datatype.
A DataStream is kind of like a table of data. It has fields (like columns) and rows of data. Each row has an entry for each field (this may be null depending on the field definition).
It is a lazily evaluated data structure. Each operation on a stream will create a new derived stream, but those operations will only occur when a final action is performed.
You can create a DataStream from an IO source, such as a Parquet file or a Hive table, or you may create a fully evaluated one from an in memory structure. In the case of the former, the data will only be loaded on demand as an action is performed.
A DataStream is split into one or more partitions. Each partition can operate independantly of the others. For example, if you filter a stream, each partition will be filtered seperately, which allows it to be parallelized. If you write out a stream, each partition can be written out to individual files, again allowing parallelization.