org.apache.flink.graph

Class GraphCsvReader

java.lang.Object

org.apache.flink.graph.GraphCsvReader

public class GraphCsvReader
extends Object

A class to build a Graph using path(s) provided to CSV file(s) with optional vertex and edge data. The class also configures the CSV readers used to read edge and vertex data such as the field types, the delimiters (row and field), the fields that should be included or skipped, and other flags, such as whether to skip the initial line as the header. The configuration is done using the functions provided in the CsvReader class.

Field Summary

Fields

Modifier and Type	Field and Description
protected CsvReader	edgeReader
protected org.apache.flink.api.common.functions.MapFunction<?,?>	mapper
protected CsvReader	vertexReader

Constructor Summary

Constructors

Constructor and Description
GraphCsvReader(org.apache.flink.core.fs.Path edgePath, ExecutionEnvironment context)
GraphCsvReader(org.apache.flink.core.fs.Path edgePath, org.apache.flink.api.common.functions.MapFunction<K,VV> mapper, ExecutionEnvironment context)
GraphCsvReader(org.apache.flink.core.fs.Path vertexPath, org.apache.flink.core.fs.Path edgePath, ExecutionEnvironment context)
GraphCsvReader(String edgePath, ExecutionEnvironment context)
GraphCsvReader(String edgePath, org.apache.flink.api.common.functions.MapFunction<K,VV> mapper, ExecutionEnvironment context)
GraphCsvReader(String vertexPath, String edgePath, ExecutionEnvironment context)

Method Summary

Methods

Modifier and Type	Method and Description
<K,EV> Graph<K,org.apache.flink.types.NullValue,EV>	edgeTypes(Class<K> vertexKey, Class<EV> edgeValue) Creates a Graph from CSV input with edge values, but without vertex values.
GraphCsvReader	fieldDelimiterEdges(String delimiter) Configures the Delimiter that separates fields in a row for the CSV reader used to read the edges (',') is used by default.
GraphCsvReader	fieldDelimiterVertices(String delimiter) Configures the Delimiter that separates fields in a row for the CSV reader used to read the vertices (',') is used by default.
GraphCsvReader	ignoreCommentsEdges(String commentPrefix) Configures the string that starts comments for the Edge Csv Reader.
GraphCsvReader	ignoreCommentsVertices(String commentPrefix) Configures the string that starts comments for the Vertex Csv Reader.
GraphCsvReader	ignoreFirstLineEdges() Sets the CSV reader for the Edges file to ignore the first line.
GraphCsvReader	ignoreFirstLineVertices() Sets the CSV reader for the Vertices file to ignore the first line.
GraphCsvReader	ignoreInvalidLinesEdges() Sets the CSV reader for the Edges file to ignore any invalid lines.
GraphCsvReader	ignoreInvalidLinesVertices() Sets the CSV reader Vertices file to ignore any invalid lines.
GraphCsvReader	includeFieldsEdges(boolean... edgeFields) Configures which fields of the CSV file containing edges data should be included and which should be skipped.
GraphCsvReader	includeFieldsEdges(long mask) Configures which fields of the CSV file containing edges data should be included and which should be skipped.
GraphCsvReader	includeFieldsEdges(String mask) Configures which fields of the CSV file containing edges data should be included and which should be skipped.
GraphCsvReader	includeFieldsVertices(boolean... vertexFields) Configures which fields of the CSV file containing vertices data should be included and which should be skipped.
GraphCsvReader	includeFieldsVertices(long mask) Configures which fields of the CSV file containing vertices data should be included and which should be skipped.
GraphCsvReader	includeFieldsVertices(String mask) Configures which fields of the CSV file containing vertices data should be included and which should be skipped.
<K> Graph<K,org.apache.flink.types.NullValue,org.apache.flink.types.NullValue>	keyType(Class<K> vertexKey) Creates a Graph from CSV input without vertex values or edge values.
GraphCsvReader	lineDelimiterEdges(String delimiter) Configures the Delimiter that separates rows for the CSV reader used to read the edges ('\n') is used by default.
GraphCsvReader	lineDelimiterVertices(String delimiter) Configures the Delimiter that separates rows for the CSV reader used to read the vertices ('\n') is used by default.
GraphCsvReader	parseQuotedStringsEdges(char quoteCharacter) Enables quoted String parsing for Edge Csv Reader.
GraphCsvReader	parseQuotedStringsVertices(char quoteCharacter) Enables quoted String parsing for Vertex Csv Reader.
<K,VV,EV> Graph<K,VV,EV>	types(Class<K> vertexKey, Class<VV> vertexValue, Class<EV> edgeValue) Creates a Graph from CSV input with vertex values and edge values.
<K,VV> Graph<K,VV,org.apache.flink.types.NullValue>	vertexTypes(Class<K> vertexKey, Class<VV> vertexValue) Creates a Graph from CSV input without edge values.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

edgeReader

protected CsvReader edgeReader

vertexReader

protected CsvReader vertexReader

mapper

protected org.apache.flink.api.common.functions.MapFunction<?,?> mapper

Constructor Detail

GraphCsvReader

public GraphCsvReader(org.apache.flink.core.fs.Path vertexPath,
              org.apache.flink.core.fs.Path edgePath,
              ExecutionEnvironment context)

GraphCsvReader

public GraphCsvReader(org.apache.flink.core.fs.Path edgePath,
              ExecutionEnvironment context)

GraphCsvReader

public GraphCsvReader(org.apache.flink.core.fs.Path edgePath,
              org.apache.flink.api.common.functions.MapFunction<K,VV> mapper,
              ExecutionEnvironment context)

GraphCsvReader

public GraphCsvReader(String edgePath,
              ExecutionEnvironment context)

GraphCsvReader

public GraphCsvReader(String vertexPath,
              String edgePath,
              ExecutionEnvironment context)

GraphCsvReader

public GraphCsvReader(String edgePath,
              org.apache.flink.api.common.functions.MapFunction<K,VV> mapper,
              ExecutionEnvironment context)

Method Detail

types

public <K,VV,EV> Graph<K,VV,EV> types(Class<K> vertexKey,
                             Class<VV> vertexValue,
                             Class<EV> edgeValue)

Creates a Graph from CSV input with vertex values and edge values. The vertex values are specified through a vertices input file or a user-defined map function.

Parameters:

vertexKey - the type of the vertex IDs

vertexValue - the type of the vertex values

edgeValue - the type of the edge values

Returns:

a Graph with vertex and edge values.

edgeTypes

public <K,EV> Graph<K,org.apache.flink.types.NullValue,EV> edgeTypes(Class<K> vertexKey,
                                                            Class<EV> edgeValue)

Creates a Graph from CSV input with edge values, but without vertex values.

Parameters:

vertexKey - the type of the vertex IDs

edgeValue - the type of the edge values

Returns:

a Graph where the edges are read from an edges CSV file (with values).

keyType

public <K> Graph<K,org.apache.flink.types.NullValue,org.apache.flink.types.NullValue> keyType(Class<K> vertexKey)

Creates a Graph from CSV input without vertex values or edge values.

Parameters:

vertexKey - the type of the vertex IDs

Returns:

a Graph where the vertex IDs are read from the edges input file.

vertexTypes

public <K,VV> Graph<K,VV,org.apache.flink.types.NullValue> vertexTypes(Class<K> vertexKey,
                                                              Class<VV> vertexValue)

Creates a Graph from CSV input without edge values. The vertex values are specified through a vertices input file or a user-defined map function. If no vertices input file is provided, the vertex IDs are automatically created from the edges input file.

Parameters:

vertexKey - the type of the vertex IDs

vertexValue - the type of the vertex values

Returns:

a Graph where the vertex IDs and vertex values.

lineDelimiterEdges

public GraphCsvReader lineDelimiterEdges(String delimiter)

Configures the Delimiter that separates rows for the CSV reader used to read the edges ('\n') is used by default.

Parameters:

delimiter - The delimiter that separates the rows.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

lineDelimiterVertices

public GraphCsvReader lineDelimiterVertices(String delimiter)

Configures the Delimiter that separates rows for the CSV reader used to read the vertices ('\n') is used by default.

Parameters:

delimiter - The delimiter that separates the rows.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

fieldDelimiterVertices

public GraphCsvReader fieldDelimiterVertices(String delimiter)

Configures the Delimiter that separates fields in a row for the CSV reader used to read the vertices (',') is used by default.

Parameters:

delimiter - The delimiter that separates the fields in a row.

Returns:

The GraphCsv reader instance itself, to allow for fluent function chaining.

fieldDelimiterEdges

public GraphCsvReader fieldDelimiterEdges(String delimiter)

Configures the Delimiter that separates fields in a row for the CSV reader used to read the edges (',') is used by default.

Parameters:

delimiter - The delimiter that separates the fields in a row.

Returns:

The GraphCsv reader instance itself, to allow for fluent function chaining.

parseQuotedStringsEdges

public GraphCsvReader parseQuotedStringsEdges(char quoteCharacter)

Enables quoted String parsing for Edge Csv Reader. Field delimiters in quoted Strings are ignored. A String is parsed as quoted if it starts and ends with a quoting character and as unquoted otherwise. Leading or tailing whitespaces are not allowed.

Parameters:

quoteCharacter - The character which is used as quoting character.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

parseQuotedStringsVertices

public GraphCsvReader parseQuotedStringsVertices(char quoteCharacter)

Enables quoted String parsing for Vertex Csv Reader. Field delimiters in quoted Strings are ignored. A String is parsed as quoted if it starts and ends with a quoting character and as unquoted otherwise. Leading or tailing whitespaces are not allowed.

Parameters:

quoteCharacter - The character which is used as quoting character.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreCommentsVertices

public GraphCsvReader ignoreCommentsVertices(String commentPrefix)

Configures the string that starts comments for the Vertex Csv Reader. By default comments will be treated as invalid lines. This function only recognizes comments which start at the beginning of the line!

Parameters:

commentPrefix - The string that starts the comments.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreCommentsEdges

public GraphCsvReader ignoreCommentsEdges(String commentPrefix)

Configures the string that starts comments for the Edge Csv Reader. By default comments will be treated as invalid lines. This function only recognizes comments which start at the beginning of the line!

Parameters:

commentPrefix - The string that starts the comments.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsVertices

public GraphCsvReader includeFieldsVertices(boolean... vertexFields)

Configures which fields of the CSV file containing vertices data should be included and which should be skipped. The parser will look at the first n fields, where n is the length of the boolean array. The parser will skip over all fields where the boolean value at the corresponding position in the array is false. The result contains the fields where the corresponding position in the boolean array is true. The number of fields in the result is consequently equal to the number of times that true occurs in the fields array.

Parameters:

vertexFields - The array of flags that describes which fields are to be included from the CSV file for vertices.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsEdges

public GraphCsvReader includeFieldsEdges(boolean... edgeFields)

Configures which fields of the CSV file containing edges data should be included and which should be skipped. The parser will look at the first n fields, where n is the length of the boolean array. The parser will skip over all fields where the boolean value at the corresponding position in the array is false. The result contains the fields where the corresponding position in the boolean array is true. The number of fields in the result is consequently equal to the number of times that true occurs in the fields array.

Parameters:

edgeFields - The array of flags that describes which fields are to be included from the CSV file for edges.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsVertices

public GraphCsvReader includeFieldsVertices(String mask)

Configures which fields of the CSV file containing vertices data should be included and which should be skipped. The positions in the string (read from position 0 to its length) define whether the field at the corresponding position in the CSV schema should be included. parser will look at the first n fields, where n is the length of the mask string The parser will skip over all fields where the character at the corresponding position in the string is '0', 'F', or 'f' (representing the value false). The result contains the fields where the corresponding position in the boolean array is '1', 'T', or 't' (representing the value true).

Parameters:

mask - The string mask defining which fields to include and which to skip.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsEdges

public GraphCsvReader includeFieldsEdges(String mask)

Configures which fields of the CSV file containing edges data should be included and which should be skipped. The positions in the string (read from position 0 to its length) define whether the field at the corresponding position in the CSV schema should be included. parser will look at the first n fields, where n is the length of the mask string The parser will skip over all fields where the character at the corresponding position in the string is '0', 'F', or 'f' (representing the value false). The result contains the fields where the corresponding position in the boolean array is '1', 'T', or 't' (representing the value true).

Parameters:

mask - The string mask defining which fields to include and which to skip.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsVertices

public GraphCsvReader includeFieldsVertices(long mask)

Configures which fields of the CSV file containing vertices data should be included and which should be skipped. The bits in the value (read from least significant to most significant) define whether the field at the corresponding position in the CSV schema should be included. parser will look at the first n fields, where n is the position of the most significant non-zero bit. The parser will skip over all fields where the character at the corresponding bit is zero, and include the fields where the corresponding bit is one.

Examples:

• A mask of 0x7 would include the first three fields.
• A mask of 0x26 (binary 100110 would skip the first fields, include fields two and three, skip fields four and five, and include field six.

Parameters:

mask - The bit mask defining which fields to include and which to skip.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

includeFieldsEdges

public GraphCsvReader includeFieldsEdges(long mask)

Configures which fields of the CSV file containing edges data should be included and which should be skipped. The bits in the value (read from least significant to most significant) define whether the field at the corresponding position in the CSV schema should be included. parser will look at the first n fields, where n is the position of the most significant non-zero bit. The parser will skip over all fields where the character at the corresponding bit is zero, and include the fields where the corresponding bit is one.

Examples:

• A mask of 0x7 would include the first three fields.
• A mask of 0x26 (binary 100110 would skip the first fields, include fields two and three, skip fields four and five, and include field six.

Parameters:

mask - The bit mask defining which fields to include and which to skip.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreFirstLineEdges

public GraphCsvReader ignoreFirstLineEdges()

Sets the CSV reader for the Edges file to ignore the first line. This is useful for files that contain a header line.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreFirstLineVertices

public GraphCsvReader ignoreFirstLineVertices()

Sets the CSV reader for the Vertices file to ignore the first line. This is useful for files that contain a header line.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreInvalidLinesEdges

public GraphCsvReader ignoreInvalidLinesEdges()

Sets the CSV reader for the Edges file to ignore any invalid lines. This is useful for files that contain an empty line at the end, multiple header lines or comments. This would throw an exception otherwise.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.

ignoreInvalidLinesVertices

public GraphCsvReader ignoreInvalidLinesVertices()

Sets the CSV reader Vertices file to ignore any invalid lines. This is useful for files that contain an empty line at the end, multiple header lines or comments. This would throw an exception otherwise.

Returns:

The GraphCSVReader instance itself, to allow for fluent function chaining.