org.apache.hadoop.hbase.mapreduce

Class TableSnapshotInputFormatImpl

java.lang.Object

org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl

@InterfaceAudience.Private
public class TableSnapshotInputFormatImpl
extends Object

Hadoop MR API-agnostic implementation for mapreduce over table snapshots.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TableSnapshotInputFormatImpl.InputSplit Implementation class for InputSplit logic common between mapred and mapreduce.
static class	TableSnapshotInputFormatImpl.RecordReader Implementation class for RecordReader logic common between mapred and mapreduce.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
static String	NUM_SPLITS_PER_REGION For MapReduce jobs running multiple mappers per region, determines number of splits to generate per region.
protected static String	RESTORE_DIR_KEY
static boolean	SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_DEFAULT
static String	SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_KEY Whether to calculate the block location for splits.
static String	SNAPSHOT_INPUTFORMAT_ROW_LIMIT_PER_INPUTSPLIT In some scenario, scan limited rows on each InputSplit for sampling data extraction
static String	SPLIT_ALGO For MapReduce jobs running multiple mappers per region, determines what split algorithm we should be using to find split points for scanners.

Constructor Summary

Constructors

Constructor and Description
TableSnapshotInputFormatImpl()

Method Summary

Modifier and Type	Method and Description
static Scan	extractScanFromConf(org.apache.hadoop.conf.Configuration conf)
static List<String>	getBestLocations(org.apache.hadoop.conf.Configuration conf, HDFSBlocksDistribution blockDistribution)
static List<HRegionInfo>	getRegionInfosFromManifest(SnapshotManifest manifest)
static SnapshotManifest	getSnapshotManifest(org.apache.hadoop.conf.Configuration conf, String snapshotName, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileSystem fs)
static RegionSplitter.SplitAlgorithm	getSplitAlgo(org.apache.hadoop.conf.Configuration conf)
static List<TableSnapshotInputFormatImpl.InputSplit>	getSplits(org.apache.hadoop.conf.Configuration conf)
static List<TableSnapshotInputFormatImpl.InputSplit>	getSplits(Scan scan, SnapshotManifest manifest, List<HRegionInfo> regionManifests, org.apache.hadoop.fs.Path restoreDir, org.apache.hadoop.conf.Configuration conf)
static List<TableSnapshotInputFormatImpl.InputSplit>	getSplits(Scan scan, SnapshotManifest manifest, List<HRegionInfo> regionManifests, org.apache.hadoop.fs.Path restoreDir, org.apache.hadoop.conf.Configuration conf, RegionSplitter.SplitAlgorithm sa, int numSplits)
static void	setInput(org.apache.hadoop.conf.Configuration conf, String snapshotName, org.apache.hadoop.fs.Path restoreDir) Configures the job to use TableSnapshotInputFormat to read from a snapshot.
static void	setInput(org.apache.hadoop.conf.Configuration conf, String snapshotName, org.apache.hadoop.fs.Path restoreDir, RegionSplitter.SplitAlgorithm splitAlgo, int numSplitsPerRegion) Configures the job to use TableSnapshotInputFormat to read from a snapshot.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

RESTORE_DIR_KEY

protected static final String RESTORE_DIR_KEY

See Also:

Constant Field Values

SPLIT_ALGO

public static final String SPLIT_ALGO

For MapReduce jobs running multiple mappers per region, determines what split algorithm we should be using to find split points for scanners.

See Also:

Constant Field Values

NUM_SPLITS_PER_REGION

public static final String NUM_SPLITS_PER_REGION

For MapReduce jobs running multiple mappers per region, determines number of splits to generate per region.

See Also:

Constant Field Values

SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_KEY

public static final String SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_KEY

Whether to calculate the block location for splits. Default to true. If the computing layer runs outside of HBase cluster, the block locality does not master. Setting this value to false could skip the calculation and save some time. Set access modifier to "public" so that these could be accessed by test classes of both org.apache.hadoop.hbase.mapred and org.apache.hadoop.hbase.mapreduce.

See Also:

Constant Field Values

SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_DEFAULT

public static final boolean SNAPSHOT_INPUTFORMAT_LOCALITY_ENABLED_DEFAULT

See Also:

Constant Field Values

SNAPSHOT_INPUTFORMAT_ROW_LIMIT_PER_INPUTSPLIT

public static final String SNAPSHOT_INPUTFORMAT_ROW_LIMIT_PER_INPUTSPLIT

In some scenario, scan limited rows on each InputSplit for sampling data extraction

See Also:

Constant Field Values

Constructor Detail

TableSnapshotInputFormatImpl

public TableSnapshotInputFormatImpl()

Method Detail

getSplits

public static List<TableSnapshotInputFormatImpl.InputSplit> getSplits(org.apache.hadoop.conf.Configuration conf)
                                                               throws IOException

Throws:

IOException

getSplitAlgo

public static RegionSplitter.SplitAlgorithm getSplitAlgo(org.apache.hadoop.conf.Configuration conf)
                                                  throws IOException

Throws:

IOException

getRegionInfosFromManifest

public static List<HRegionInfo> getRegionInfosFromManifest(SnapshotManifest manifest)

getSnapshotManifest

public static SnapshotManifest getSnapshotManifest(org.apache.hadoop.conf.Configuration conf,
                                                   String snapshotName,
                                                   org.apache.hadoop.fs.Path rootDir,
                                                   org.apache.hadoop.fs.FileSystem fs)
                                            throws IOException

Throws:

IOException

extractScanFromConf

public static Scan extractScanFromConf(org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Throws:

IOException

getSplits

public static List<TableSnapshotInputFormatImpl.InputSplit> getSplits(Scan scan,
                                                                      SnapshotManifest manifest,
                                                                      List<HRegionInfo> regionManifests,
                                                                      org.apache.hadoop.fs.Path restoreDir,
                                                                      org.apache.hadoop.conf.Configuration conf)
                                                               throws IOException

Throws:

IOException

getSplits

public static List<TableSnapshotInputFormatImpl.InputSplit> getSplits(Scan scan,
                                                                      SnapshotManifest manifest,
                                                                      List<HRegionInfo> regionManifests,
                                                                      org.apache.hadoop.fs.Path restoreDir,
                                                                      org.apache.hadoop.conf.Configuration conf,
                                                                      RegionSplitter.SplitAlgorithm sa,
                                                                      int numSplits)
                                                               throws IOException

Throws:

IOException

getBestLocations

public static List<String> getBestLocations(org.apache.hadoop.conf.Configuration conf,
                                            HDFSBlocksDistribution blockDistribution)

setInput

public static void setInput(org.apache.hadoop.conf.Configuration conf,
                            String snapshotName,
                            org.apache.hadoop.fs.Path restoreDir)
                     throws IOException

Configures the job to use TableSnapshotInputFormat to read from a snapshot.

Parameters:

conf - the job to configuration

snapshotName - the name of the snapshot to read from

restoreDir - a temporary directory to restore the snapshot into. Current user should have write permissions to this directory, and this should not be a subdirectory of rootdir. After the job is finished, restoreDir can be deleted.

Throws:

IOException - if an error occurs

setInput

public static void setInput(org.apache.hadoop.conf.Configuration conf,
                            String snapshotName,
                            org.apache.hadoop.fs.Path restoreDir,
                            RegionSplitter.SplitAlgorithm splitAlgo,
                            int numSplitsPerRegion)
                     throws IOException

Configures the job to use TableSnapshotInputFormat to read from a snapshot.

Parameters:

conf - the job to configure

snapshotName - the name of the snapshot to read from

restoreDir - a temporary directory to restore the snapshot into. Current user should have write permissions to this directory, and this should not be a subdirectory of rootdir. After the job is finished, restoreDir can be deleted.

numSplitsPerRegion - how many input splits to generate per one region

splitAlgo - SplitAlgorithm to be used when generating InputSplits

Throws:

IOException - if an error occurs