com.datastax.spark.connector.rdd.partitioner
CassandraPartitionedRDD
Related Doc:
package partitioner
com.datastax.spark.connector.rdd.partitioner
This RDD was partitioned using the Replica Partitioner so we can use that to get preferred location data
This RDD was partitioned using the Replica Partitioner so we can use that to get preferred location data
This method currently uses thrift to determine the local endpoint and rpc endpoints from whatever endpoint the partition belongs to.
This method currently uses thrift to determine the local endpoint and rpc endpoints from whatever endpoint the partition belongs to. This gives us a better chance of matching the bound interface of the Spark Executor. In addition we will add the HostName and HostAddress that we get for each of these endpoints to cover all of the logical choices.
Uses the data from RDD to join with a Cassandra table without retrieving the entire table.
Uses the data from RDD to join with a Cassandra table without retrieving the entire table. Any RDD which can be used to saveToCassandra can be used to joinWithCassandra as well as any RDD which only specifies the partition Key of a Cassandra Table. This method executes single partition requests against the Cassandra Table and accepts the functional modifiers that a normal com.datastax.spark.connector.rdd.CassandraTableScanRDD takes.
By default this method only uses the Partition Key for joining but any combination of columns which are acceptable to C* can be used in the join. Specify columns using joinColumns as a parameter or the on() method.
Example With Prior Repartitioning:
val source = sc.parallelize(keys).map(x => new KVRow(x)) val repart = source.repartitionByCassandraReplica(keyspace, tableName, 10) val someCass = repart.joinWithCassandraTable(keyspace, tableName)
Example Joining on Clustering Columns:
val source = sc.parallelize(keys).map(x => (x, x * 100)) val someCass = source.joinWithCassandraTable(keyspace, wideTable).on(SomeColumns("key", "group"))
Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row.
Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row. The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.
Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName.
Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName.
Calling this method before using joinWithCassandraTable will ensure that requests will be coordinator
local. partitionsPerHost Controls the number of Spark Partitions that will be created in this repartitioning
event.
The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.
Saves the data from RDD to a new table with definition taken from the
ColumnMapper for this class.
Saves the data from RDD to a new table with definition taken from the
ColumnMapper for this class.
keyspace where to create a new table
name of the table to create; the table must not exist
Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.
additional configuration object allowing to set consistency level, batch size, etc.
optional, implicit connector to Cassandra
factory for obtaining the row writer to be used to extract column values from items of the RDD
a column mapper determining the definition of the table
Saves the data from RDD to a new table defined by the given TableDef.
Saves the data from RDD to a new table defined by the given TableDef.
First it creates a new table with all columns from the TableDef
and then it saves RDD content in the same way as saveToCassandra.
The table must not exist prior to this call.
table definition used to create a new table
Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.
additional configuration object allowing to set consistency level, batch size, etc.
optional, implicit connector to Cassandra
factory for obtaining the row writer to be used to extract column values from items of the RDD
Saves the data from RDD to a Cassandra table.
Saves the data from RDD to a Cassandra table. Uses the specified column names.
the name of the Keyspace to use
the name of the Table to use
additional configuration object allowing to set consistency level, batch size, etc.
Applies a function to each item, and groups consecutive items having the same value together.
Applies a function to each item, and groups consecutive items having the same value together.
Contrary to groupBy, items from the same group must be already next to each other in the
original collection. Works locally on each partition, so items from different
partitions will never be placed in the same group.
Groups items with the same key, assuming the items with the same key are next to each other in the collection.
Groups items with the same key, assuming the items with the same key are next to each other
in the collection. It does not perform shuffle, therefore it is much faster than using
much more universal Spark RDD groupByKey. For this method to be useful with Cassandra tables,
the key must represent a prefix of the primary key, containing at least the partition key of the
Cassandra table.
(cassandraPartitionedRDD: RDDFunctions[T]).sparkContext
(Since version 1.0.0) use mapPartitionsWithIndex and filter
(Since version 1.0.0) use mapPartitionsWithIndex and flatMap
(Since version 1.0.0) use mapPartitionsWithIndex and foreach
(Since version 1.2.0) use TaskContext.get
(Since version 0.7.0) use mapPartitionsWithIndex
(Since version 1.0.0) use mapPartitionsWithIndex
(Since version 1.0.0) use collect
RDD created by repartitionByCassandraReplica with preferred locations mapping to the CassandraReplicas each partition was created for.