io.kgraph.streaming

Class EdgeStream<K,EV>

java.lang.Object

io.kgraph.streaming.EdgeStream<K,EV>

Type Parameters:

K - the key type for edge and vertex identifiers.

EV - the value type for edges.

All Implemented Interfaces:

KGraphStream<K,Void,EV>

public class EdgeStream<K,EV>
extends Object
implements KGraphStream<K,Void,EV>

Represents a graph stream where the stream consists solely of edges.

Field Summary

Fields inherited from interface io.kgraph.streaming.KGraphStream

GLOBAL_KEY

Constructor Summary

Constructors

Constructor and Description
EdgeStream(org.apache.kafka.streams.kstream.KStream<Edge<K>,EV> edges, GraphSerialized<K,Void,EV> serialized) Creates a graph from an edge stream.

Method Summary

Modifier and Type	Method and Description
<VV> org.apache.kafka.streams.kstream.KStream<K,VV>	aggregate(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,Iterable<org.apache.kafka.streams.KeyValue<K,VV>>> edgeMapper, org.apache.kafka.streams.kstream.KeyValueMapper<K,VV,org.apache.kafka.streams.KeyValue<K,VV>> vertexMapper) The aggregate function splits the edge stream up into a vertex stream and applies a mapper on the resulting vertices
<S,T> org.apache.kafka.streams.kstream.KTable<org.apache.kafka.streams.kstream.Windowed<Short>,T>	aggregate(SummaryAggregation<K,EV,S,T> summaryAggregation) Applies an incremental aggregation on a graphstream and returns a stream of aggregation results
org.apache.kafka.streams.kstream.KStream<Edge<K>,Set<K>>	buildNeighborhood(boolean directed) Builds the neighborhood state by creating adjacency lists.
org.apache.kafka.streams.kstream.KStream<K,Long>	degrees() Get the degree stream
EdgeStream<K,EV>	distinct() Removes the duplicate edges by storing a neighborhood set for each vertex
org.apache.kafka.streams.kstream.KStream<Edge<K>,EV>	edges()
EdgeStream<K,EV>	filterEdges(org.apache.kafka.streams.kstream.Predicate<Edge<K>,EV> filter) Apply a filter to each edge in the graph stream
EdgeStream<K,EV>	filterVertices(org.apache.kafka.streams.kstream.Predicate<K,Void> filter) Apply a filter to each vertex in the graph stream Since this is an edge-only stream, the vertex filter can only access the key of vertices
<VV> org.apache.kafka.streams.kstream.KStream<Short,VV>	globalAggregate(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,Iterable<org.apache.kafka.streams.KeyValue<K,VV>>> edgeMapper, org.apache.kafka.streams.kstream.KeyValueMapper<K,VV,Iterable<org.apache.kafka.streams.KeyValue<Short,VV>>> vertexMapper, boolean collectUpdates) Returns a global aggregate on the previously split vertex stream
org.apache.kafka.streams.kstream.KStream<K,Long>	inDegrees() Get the in-degree stream
<NV> EdgeStream<K,NV>	mapEdges(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,org.apache.kafka.streams.KeyValue<Edge<K>,NV>> mapper, org.apache.kafka.common.serialization.Serde<NV> newValueSerde) Apply a function to the attribute of each edge in the graph stream.
org.apache.kafka.streams.kstream.KStream<Short,Long>	numberOfEdges()
org.apache.kafka.streams.kstream.KStream<Short,Long>	numberOfVertices()
org.apache.kafka.streams.kstream.KStream<K,Long>	outDegrees() Get the out-degree stream
EdgeStream<K,EV>	reverse()
KGraphWindowedStream<K,EV>	slice(org.apache.kafka.streams.kstream.Windows<? extends org.apache.kafka.streams.kstream.Window> size) Discretizes the edge stream into tumbling windows of the specified size.
KGraphWindowedStream<K,EV>	slice(org.apache.kafka.streams.kstream.Windows<? extends org.apache.kafka.streams.kstream.Window> size, EdgeDirection direction) Discretizes the edge stream into tumbling windows of the specified size.
EdgeStream<K,EV>	undirected()
EdgeStream<K,EV>	union(KGraphStream<K,Void,EV> graph)
org.apache.kafka.streams.kstream.KStream<K,Void>	vertices()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EdgeStream

public EdgeStream(org.apache.kafka.streams.kstream.KStream<Edge<K>,EV> edges,
                  GraphSerialized<K,Void,EV> serialized)

Creates a graph from an edge stream.

Parameters:

edges - a DataStream of edges.

serialized - the graph serialized

Method Detail

edges

public org.apache.kafka.streams.kstream.KStream<Edge<K>,EV> edges()

Specified by:

edges in interface KGraphStream<K,Void,EV>

Returns:

the edge DataStream.

vertices

public org.apache.kafka.streams.kstream.KStream<K,Void> vertices()

Specified by:

vertices in interface KGraphStream<K,Void,EV>

Returns:

the vertex DataStream.

mapEdges

public <NV> EdgeStream<K,NV> mapEdges(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,org.apache.kafka.streams.KeyValue<Edge<K>,NV>> mapper,
                                      org.apache.kafka.common.serialization.Serde<NV> newValueSerde)

Apply a function to the attribute of each edge in the graph stream.

Specified by:

mapEdges in interface KGraphStream<K,Void,EV>

Type Parameters:

NV - the new vertex value type

Parameters:

mapper - the map function to apply.

newValueSerde - the new value serde

Returns:

a new graph stream.

filterEdges

public EdgeStream<K,EV> filterEdges(org.apache.kafka.streams.kstream.Predicate<Edge<K>,EV> filter)

Apply a filter to each edge in the graph stream

Specified by:

filterEdges in interface KGraphStream<K,Void,EV>

Parameters:

filter - the filter function to apply.

Returns:

the filtered graph stream.

filterVertices

public EdgeStream<K,EV> filterVertices(org.apache.kafka.streams.kstream.Predicate<K,Void> filter)

Apply a filter to each vertex in the graph stream Since this is an edge-only stream, the vertex filter can only access the key of vertices

Specified by:

filterVertices in interface KGraphStream<K,Void,EV>

Parameters:

filter - the filter function to apply.

Returns:

the filtered graph stream.

numberOfEdges

public org.apache.kafka.streams.kstream.KStream<Short,Long> numberOfEdges()

Specified by:

numberOfEdges in interface KGraphStream<K,Void,EV>

Returns:

a data stream representing the number of all edges in the streamed graph, including possible duplicates

numberOfVertices

public org.apache.kafka.streams.kstream.KStream<Short,Long> numberOfVertices()

Specified by:

numberOfVertices in interface KGraphStream<K,Void,EV>

Returns:

a continuously improving data stream representing the number of vertices in the streamed graph

distinct

public EdgeStream<K,EV> distinct()

Removes the duplicate edges by storing a neighborhood set for each vertex

Specified by:

distinct in interface KGraphStream<K,Void,EV>

Returns:

a graph stream with no duplicate edges

undirected

public EdgeStream<K,EV> undirected()

Specified by:

undirected in interface KGraphStream<K,Void,EV>

Returns:

a graph stream where edges are undirected

reverse

public EdgeStream<K,EV> reverse()

Specified by:

reverse in interface KGraphStream<K,Void,EV>

Returns:

a graph stream with the edge directions reversed

degrees

public org.apache.kafka.streams.kstream.KStream<K,Long> degrees()

Get the degree stream

Specified by:

degrees in interface KGraphStream<K,Void,EV>

Returns:

a stream of vertices, with the degree as the vertex value

inDegrees

public org.apache.kafka.streams.kstream.KStream<K,Long> inDegrees()

Get the in-degree stream

Specified by:

inDegrees in interface KGraphStream<K,Void,EV>

Returns:

a stream of vertices, with the in-degree as the vertex value

outDegrees

public org.apache.kafka.streams.kstream.KStream<K,Long> outDegrees()

Get the out-degree stream

Specified by:

outDegrees in interface KGraphStream<K,Void,EV>

Returns:

a stream of vertices, with the out-degree as the vertex value

union

public EdgeStream<K,EV> union(KGraphStream<K,Void,EV> graph)

Specified by:

union in interface KGraphStream<K,Void,EV>

Parameters:

graph - the streamed graph to union with

Returns:

a streamed graph where the two edge streams are merged

aggregate

public <VV> org.apache.kafka.streams.kstream.KStream<K,VV> aggregate(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,Iterable<org.apache.kafka.streams.KeyValue<K,VV>>> edgeMapper,
                                                                     org.apache.kafka.streams.kstream.KeyValueMapper<K,VV,org.apache.kafka.streams.KeyValue<K,VV>> vertexMapper)

The aggregate function splits the edge stream up into a vertex stream and applies a mapper on the resulting vertices

Specified by:

aggregate in interface KGraphStream<K,Void,EV>

Type Parameters:

VV - the vertex value used

Parameters:

edgeMapper - the mapper that converts the edge stream to a vertex stream

vertexMapper - the mapper that aggregates vertex values

Returns:

a stream of vertices with the aggregated vertex value

globalAggregate

public <VV> org.apache.kafka.streams.kstream.KStream<Short,VV> globalAggregate(org.apache.kafka.streams.kstream.KeyValueMapper<Edge<K>,EV,Iterable<org.apache.kafka.streams.KeyValue<K,VV>>> edgeMapper,
                                                                               org.apache.kafka.streams.kstream.KeyValueMapper<K,VV,Iterable<org.apache.kafka.streams.KeyValue<Short,VV>>> vertexMapper,
                                                                               boolean collectUpdates)

Returns a global aggregate on the previously split vertex stream

Specified by:

globalAggregate in interface KGraphStream<K,Void,EV>

Type Parameters:

VV - the return value type

Parameters:

edgeMapper - the mapper that converts the edge stream to a vertex stream

vertexMapper - the mapper that aggregates vertex values

collectUpdates - boolean specifying whether the aggregate should only be collected when there is an update

Returns:

a stream of the aggregated values

buildNeighborhood

public org.apache.kafka.streams.kstream.KStream<Edge<K>,Set<K>> buildNeighborhood(boolean directed)

Builds the neighborhood state by creating adjacency lists.

Specified by:

buildNeighborhood in interface KGraphStream<K,Void,EV>

Parameters:

directed - if true, only the out-neighbors will be stored otherwise both directions are considered

Returns:

a stream of Tuple3, where the first 2 fields identify the edge processed and the third field is the adjacency list that was updated by processing this edge.

slice

public KGraphWindowedStream<K,EV> slice(org.apache.kafka.streams.kstream.Windows<? extends org.apache.kafka.streams.kstream.Window> size)

Discretizes the edge stream into tumbling windows of the specified size.

The edge stream is partitioned so that all neighbors of a vertex belong to the same partition. The KeyedStream is then windowed into tumbling time windows.

By default, each vertex is grouped with its outgoing edges. Use slice(Windows, EdgeDirection) to manually set the edge direction grouping.

Specified by:

slice in interface KGraphStream<K,Void,EV>

Parameters:

size - the size of the window

Returns:

a GraphWindowStream of the specified size

slice

public KGraphWindowedStream<K,EV> slice(org.apache.kafka.streams.kstream.Windows<? extends org.apache.kafka.streams.kstream.Window> size,
                                        EdgeDirection direction)
                                 throws IllegalArgumentException

Discretizes the edge stream into tumbling windows of the specified size.

The edge stream is partitioned so that all neighbors of a vertex belong to the same partition. The KeyedStream is then windowed into tumbling time windows.

Specified by:

slice in interface KGraphStream<K,Void,EV>

Parameters:

size - the size of the window

direction - the EdgeDirection to key by

Returns:

a GraphWindowStream of the specified size, keyed by

Throws:

IllegalArgumentException

aggregate

public <S,T> org.apache.kafka.streams.kstream.KTable<org.apache.kafka.streams.kstream.Windowed<Short>,T> aggregate(SummaryAggregation<K,EV,S,T> summaryAggregation)

Applies an incremental aggregation on a graphstream and returns a stream of aggregation results

Specified by:

aggregate in interface KGraphStream<K,Void,EV>

Type Parameters:

S - initial type

T - result type

Parameters:

summaryAggregation - the summary aggregation

Returns:

the aggregation results