Package org.apache.accumulo.core.client.mapreduce

Class InputFormatBase<K,​V>

  • Direct Known Subclasses:
    AccumuloInputFormat, AccumuloRowInputFormat
    @Deprecated
public abstract class InputFormatBase<K,​V>
extends AbstractInputFormat<K,​V>
    Deprecated.
    since 2.0.0; Use org.apache.accumulo.hadoop.mapreduce instead from the accumulo-hadoop-mapreduce.jar
    This abstract InputFormat class allows MapReduce jobs to use Accumulo as the source of K,V pairs.

    Subclasses must implement a InputFormat.createRecordReader(InputSplit, TaskAttemptContext) to provide a RecordReader for K,V.

    A static base class, RecordReaderBase, is provided to retrieve Accumulo Key/Value pairs, but one must implement its RecordReader.nextKeyValue() to transform them to the desired generic types K,V.

    See AccumuloInputFormat for an example implementation.

    • Constructor Detail

      • InputFormatBase

        public InputFormatBase()
        Deprecated.

    • Method Detail

      • getInputTableName

        protected static String getInputTableName​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Gets the table name from the configuration.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        the table name
        Since:
        1.5.0
        See Also:
        setInputTableName(Job, String)

      • setInputTableName

        public static void setInputTableName​(org.apache.hadoop.mapreduce.Job job,
                                     String tableName)
        Deprecated.
        Sets the name of the input table, over which this job will scan.
        Parameters:
        job - the Hadoop job instance to be configured
        tableName - the table to use when the tablename is null in the write call
        Since:
        1.5.0

      • setRanges

        public static void setRanges​(org.apache.hadoop.mapreduce.Job job,
                             Collection<Range> ranges)
        Deprecated.
        Sets the input ranges to scan for the single input table associated with this job.
        Parameters:
        job - the Hadoop job instance to be configured
        ranges - the ranges that will be mapped over
        Since:
        1.5.0
        See Also:
        TableOperations.splitRangeByTablets(String, Range, int)

      • getRanges

        protected static List<Range> getRanges​(org.apache.hadoop.mapreduce.JobContext context)
                                throws IOException
        Deprecated.
        Gets the ranges to scan over from a job.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        the ranges
        Throws:
        IOException
        Since:
        1.5.0
        See Also:
        setRanges(Job, Collection)

      • fetchColumns

        public static void fetchColumns​(org.apache.hadoop.mapreduce.Job job,
                                Collection<Pair<org.apache.hadoop.io.Text,​org.apache.hadoop.io.Text>> columnFamilyColumnQualifierPairs)
        Deprecated.
        Restricts the columns that will be mapped over for this job for the default input table.
        Parameters:
        job - the Hadoop job instance to be configured
        columnFamilyColumnQualifierPairs - a pair of Text objects corresponding to column family and column qualifier. If the column qualifier is null, the entire column family is selected. An empty set is the default and is equivalent to scanning the all columns.
        Since:
        1.5.0

      • getFetchedColumns

        protected static Set<Pair<org.apache.hadoop.io.Text,​org.apache.hadoop.io.Text>> getFetchedColumns​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Gets the columns to be mapped over from this job.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        a set of columns
        Since:
        1.5.0
        See Also:
        fetchColumns(Job, Collection)

      • addIterator

        public static void addIterator​(org.apache.hadoop.mapreduce.Job job,
                               IteratorSetting cfg)
        Deprecated.
        Encode an iterator on the single input table for this job.
        Parameters:
        job - the Hadoop job instance to be configured
        cfg - the configuration of the iterator
        Since:
        1.5.0

      • getIterators

        protected static List<IteratorSetting> getIterators​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Gets a list of the iterator settings (for iterators to apply to a scanner) from this configuration.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        a list of iterators
        Since:
        1.5.0
        See Also:
        addIterator(Job, IteratorSetting)

      • setAutoAdjustRanges

        public static void setAutoAdjustRanges​(org.apache.hadoop.mapreduce.Job job,
                                       boolean enableFeature)
        Deprecated.
        Controls the automatic adjustment of ranges for this job. This feature merges overlapping ranges, then splits them to align with tablet boundaries. Disabling this feature will cause exactly one Map task to be created for each specified range. The default setting is enabled. *

        By default, this feature is enabled.

        Parameters:
        job - the Hadoop job instance to be configured
        enableFeature - the feature is enabled if true, disabled otherwise
        Since:
        1.5.0
        See Also:
        setRanges(Job, Collection)

      • getAutoAdjustRanges

        protected static boolean getAutoAdjustRanges​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Determines whether a configuration has auto-adjust ranges enabled. Must be enabled when setBatchScan(Job, boolean) is true.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        false if the feature is disabled, true otherwise
        Since:
        1.5.0
        See Also:
        setAutoAdjustRanges(Job, boolean)

      • setScanIsolation

        public static void setScanIsolation​(org.apache.hadoop.mapreduce.Job job,
                                    boolean enableFeature)
        Deprecated.
        Controls the use of the IsolatedScanner in this job.

        By default, this feature is disabled.

        Parameters:
        job - the Hadoop job instance to be configured
        enableFeature - the feature is enabled if true, disabled otherwise
        Since:
        1.5.0

      • isIsolated

        protected static boolean isIsolated​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Determines whether a configuration has isolation enabled.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        true if the feature is enabled, false otherwise
        Since:
        1.5.0
        See Also:
        setScanIsolation(Job, boolean)

      • setLocalIterators

        public static void setLocalIterators​(org.apache.hadoop.mapreduce.Job job,
                                     boolean enableFeature)
        Deprecated.
        Controls the use of the ClientSideIteratorScanner in this job. Enabling this feature will cause the iterator stack to be constructed within the Map task, rather than within the Accumulo TServer. To use this feature, all classes needed for those iterators must be available on the classpath for the task.

        By default, this feature is disabled.

        Parameters:
        job - the Hadoop job instance to be configured
        enableFeature - the feature is enabled if true, disabled otherwise
        Since:
        1.5.0

      • usesLocalIterators

        protected static boolean usesLocalIterators​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Determines whether a configuration uses local iterators.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        true if the feature is enabled, false otherwise
        Since:
        1.5.0
        See Also:
        setLocalIterators(Job, boolean)

      • setOfflineTableScan

        public static void setOfflineTableScan​(org.apache.hadoop.mapreduce.Job job,
                                       boolean enableFeature)
        Deprecated.
        Enable reading offline tables. By default, this feature is disabled and only online tables are scanned. This will make the map reduce job directly read the table's files. If the table is not offline, then the job will fail. If the table comes online during the map reduce job, it is likely that the job will fail.

        To use this option, the map reduce user will need access to read the Accumulo directory in HDFS.

        Reading the offline table will create the scan time iterator stack in the map process. So any iterators that are configured for the table will need to be on the mapper's classpath.

        One way to use this feature is to clone a table, take the clone offline, and use the clone as the input table for a map reduce job. If you plan to map reduce over the data many times, it may be better to the compact the table, clone it, take it offline, and use the clone for all map reduce jobs. The reason to do this is that compaction will reduce each tablet in the table to one file, and it is faster to read from one file.

        There are two possible advantages to reading a tables file directly out of HDFS. First, you may see better read performance. Second, it will support speculative execution better. When reading an online table speculative execution can put more load on an already slow tablet server.

        By default, this feature is disabled.

        Parameters:
        job - the Hadoop job instance to be configured
        enableFeature - the feature is enabled if true, disabled otherwise
        Since:
        1.5.0

      • isOfflineScan

        protected static boolean isOfflineScan​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Determines whether a configuration has the offline table scan feature enabled.
        Parameters:
        context - the Hadoop context for the configured job
        Returns:
        true if the feature is enabled, false otherwise
        Since:
        1.5.0
        See Also:
        setOfflineTableScan(Job, boolean)

      • setBatchScan

        public static void setBatchScan​(org.apache.hadoop.mapreduce.Job job,
                                boolean enableFeature)
        Deprecated.
        Controls the use of the BatchScanner in this job. Using this feature will group Ranges by their source tablet, producing an InputSplit per tablet rather than per Range. This batching helps to reduce overhead when querying a large number of small ranges. (ex: when doing quad-tree decomposition for spatial queries)

        In order to achieve good locality of InputSplits this option always clips the input Ranges to tablet boundaries. This may result in one input Range contributing to several InputSplits.

        Note: that the value of setAutoAdjustRanges(Job, boolean) is ignored and is assumed to be true when BatchScan option is enabled.

        This configuration is incompatible with:

        By default, this feature is disabled.

        Parameters:
        job - the Hadoop job instance to be configured
        enableFeature - the feature is enabled if true, disabled otherwise
        Since:
        1.7.0

      • isBatchScan

        public static boolean isBatchScan​(org.apache.hadoop.mapreduce.JobContext context)
        Deprecated.
        Determines whether a configuration has the BatchScanner feature enabled.
        Parameters:
        context - the Hadoop context for the configured job
        Since:
        1.7.0
        See Also:
        setBatchScan(Job, boolean)

      • setSamplerConfiguration

        public static void setSamplerConfiguration​(org.apache.hadoop.mapreduce.Job job,
                                           SamplerConfiguration samplerConfig)
        Deprecated.
        Causes input format to read sample data. If sample data was created using a different configuration or a tables sampler configuration changes while reading data, then the input format will throw an error.
        Parameters:
        job - the Hadoop job instance to be configured
        samplerConfig - The sampler configuration that sample must have been created with inorder for reading sample data to succeed.
        Since:
        1.8.0
        See Also:
        ScannerBase.setSamplerConfiguration(SamplerConfiguration)