accumulo

Type Members

1. class AccumuloLayerCopier extends LayerCopier[LayerId]

   

2. class AccumuloLayerManager extends LayerManager[LayerId]

   

3. class AccumuloLayerReader extends FilteringLayerReader[LayerId]

   

4. class AccumuloLayerReindexer extends LayerReindexer[LayerId]

   

5. class AccumuloLayerWriter extends LayerWriter[LayerId]

   

6. class AccumuloSparkLayerProvider extends AccumuloCollectionLayerProvider with LayerReaderProvider with LayerWriterProvider

   

   Provides AccumuloAttributeStore instance for URI with accumulo scheme.

   Provides AccumuloAttributeStore instance for URI with accumulo scheme. ex: accumulo://[user[:password]@]zookeeper/instance-name[?attributes=table1[&layers=table2]]

   Attributes table name is optional, not provided default value will be used. Layers table name is required to instantiate a LayerWriter

7. sealed trait AccumuloWriteStrategy extends Serializable

   

8. case class HdfsWriteStrategy(ingestPath: Path) extends AccumuloWriteStrategy with Product with Serializable

   

   This strategy will perfom Accumulo bulk ingest.

   This strategy will perfom Accumulo bulk ingest. Bulk ingest requires that sorted records be written to the filesystem, preferbly HDFS, before Accumulo is able to ingest them. After the ingest is finished the nodes will likely go through a period of high load as they perform major compactions.

   Note: Giving relative URLs will cause HDFS to use the fs.defaultFS property in core-site.xml. If not specified this will default to local ('file:/') system, this is undesriable.

   ingestPath

   Path where spark will write RDD records for ingest

9. class SocketWriteStrategy extends AccumuloWriteStrategy

   

   This strategy will create one BatchWriter per partition and attempt to stream the records to the target tablets.

   This strategy will create one BatchWriter per partition and attempt to stream the records to the target tablets. In order to gain some parallism this strategy will create a number of splits in the target table equal to the number of tservers in the cluster. This is suitable for smaller ingests, or where HdfsWriteStrategy is otherwise not possible.

   This strategy will not create splits before starting to write. If you wish to do that use AccumuloUtils.getSplits first.

   There is a problem in Accumulo 1.6 (fixed in 1.7) where the split creation does not wait for the resulting empty tablets to distribute through the cluster before returning. This will create a warm-up period where the pressure the ingest writers on that node will delay tablet re-balancing.

   The speed of the ingest can be improved by setting tserver.wal.sync.method=hflush in accumulo shell. Note: this introduces higher chance of data loss due to sudden node failure.

   BatchWriter is notified of the tablet migrations and will follow them around the cluster.