org.apache.fluo.recipes.spark

Class FluoSparkHelper

java.lang.Object

org.apache.fluo.recipes.spark.FluoSparkHelper

public class FluoSparkHelper
extends Object

Helper methods for using Fluo with Spark

Since:

1.0.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FluoSparkHelper.BulkImportOptions Optional settings for Bulk Imports

Constructor Summary

Constructors

Constructor and Description
FluoSparkHelper(org.apache.fluo.api.config.FluoConfiguration fluoConfig, org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.Path tempBaseDir)

Method Summary

Modifier and Type	Method and Description
void	bulkImportKvToAccumulo(org.apache.spark.api.java.JavaPairRDD<org.apache.accumulo.core.data.Key,org.apache.accumulo.core.data.Value> data, String accumuloTable, FluoSparkHelper.BulkImportOptions opts) Bulk import Key/Value data into specified Accumulo table.
void	bulkImportKvToFluo(org.apache.spark.api.java.JavaPairRDD<org.apache.accumulo.core.data.Key,org.apache.accumulo.core.data.Value> data, FluoSparkHelper.BulkImportOptions opts) Bulk import Key/Value data into into Fluo table (obtained from Fluo configuration).
void	bulkImportRcvToAccumulo(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> data, String accumuloTable, FluoSparkHelper.BulkImportOptions opts) Bulk import RowColumn/Value data into specified Accumulo table.
void	bulkImportRcvToFluo(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> data, FluoSparkHelper.BulkImportOptions opts) Bulk import RowColumn/Value data into Fluo table (obtained from Fluo configuration).
org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes>	readFromFluo(org.apache.spark.api.java.JavaSparkContext ctx) Reads all data from a snapshot in Fluo and returns it as a RowColumn/Value RDD.
static org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes>	toPairRDD(org.apache.spark.api.java.JavaRDD<org.apache.fluo.api.data.RowColumnValue> rcvRDD) Converts RowColumnValue RDD to RowColumn/Bytes PairRDD
static org.apache.spark.api.java.JavaRDD<org.apache.fluo.api.data.RowColumnValue>	toRcvRDD(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> pairRDD) Converts RowColumn/Bytes PairRDD to RowColumnValue RDD

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FluoSparkHelper

public FluoSparkHelper(org.apache.fluo.api.config.FluoConfiguration fluoConfig,
                       org.apache.hadoop.conf.Configuration hadoopConfig,
                       org.apache.hadoop.fs.Path tempBaseDir)

Method Detail

toPairRDD

public static org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> toPairRDD(org.apache.spark.api.java.JavaRDD<org.apache.fluo.api.data.RowColumnValue> rcvRDD)

Converts RowColumnValue RDD to RowColumn/Bytes PairRDD

Parameters:

rcvRDD - RowColumnValue RDD to convert

Returns:

RowColumn/Bytes PairRDD

toRcvRDD

public static org.apache.spark.api.java.JavaRDD<org.apache.fluo.api.data.RowColumnValue> toRcvRDD(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> pairRDD)

Converts RowColumn/Bytes PairRDD to RowColumnValue RDD

Parameters:

pairRDD - RowColumn/Bytes PairRDD

Returns:

RowColumnValue RDD

readFromFluo

public org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> readFromFluo(org.apache.spark.api.java.JavaSparkContext ctx)

Reads all data from a snapshot in Fluo and returns it as a RowColumn/Value RDD.

Parameters:

ctx - Java Spark context

Returns:

RowColumn/Value RDD containing all data in Fluo

bulkImportRcvToFluo

public void bulkImportRcvToFluo(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> data,
                                FluoSparkHelper.BulkImportOptions opts)

Bulk import RowColumn/Value data into Fluo table (obtained from Fluo configuration). This method will repartition RDD using the current split points of the Fluo table, creating one partition per tablet in the table. This is done so that one RFile is created per tablet for bulk import.

Parameters:

data - RowColumn/Value data to import

opts - Bulk import options

bulkImportKvToFluo

public void bulkImportKvToFluo(org.apache.spark.api.java.JavaPairRDD<org.apache.accumulo.core.data.Key,org.apache.accumulo.core.data.Value> data,
                               FluoSparkHelper.BulkImportOptions opts)

Bulk import Key/Value data into into Fluo table (obtained from Fluo configuration). This method does not repartition data. One RFile will be created for each partition in the passed in RDD. Ensure the RDD is reasonably partitioned before calling this method.

Parameters:

data - Key/Value data to import

opts - Bulk import options

bulkImportRcvToAccumulo

public void bulkImportRcvToAccumulo(org.apache.spark.api.java.JavaPairRDD<org.apache.fluo.api.data.RowColumn,org.apache.fluo.api.data.Bytes> data,
                                    String accumuloTable,
                                    FluoSparkHelper.BulkImportOptions opts)

Bulk import RowColumn/Value data into specified Accumulo table. This method will repartition RDD using the current split points of the specified table, creating one partition per tablet in the table. This is done so that one RFile is created per tablet for bulk import.

Parameters:

data - RowColumn/Value data to import

accumuloTable - Accumulo table used for import

opts - Bulk import options

bulkImportKvToAccumulo

public void bulkImportKvToAccumulo(org.apache.spark.api.java.JavaPairRDD<org.apache.accumulo.core.data.Key,org.apache.accumulo.core.data.Value> data,
                                   String accumuloTable,
                                   FluoSparkHelper.BulkImportOptions opts)

Bulk import Key/Value data into specified Accumulo table. This method does not repartition data. One RFile will be created for each partition in the passed in RDD. Ensure the RDD is reasonably partitioned before calling this method.

Parameters:

data - Key/value data to import

accumuloTable - Accumulo table used for import

opts - Bulk import options