org.apache.drill.exec.planner

Class FileSystemPartitionDescriptor

java.lang.Object

org.apache.drill.exec.planner.AbstractPartitionDescriptor

org.apache.drill.exec.planner.FileSystemPartitionDescriptor

All Implemented Interfaces:

Iterable<List<PartitionLocation>>, PartitionDescriptor

public class FileSystemPartitionDescriptor
extends AbstractPartitionDescriptor

Partition descriptor for file system based tables.

Field Summary

Fields inherited from class org.apache.drill.exec.planner.AbstractPartitionDescriptor

locationSuperList, sublistsCreated

Fields inherited from interface org.apache.drill.exec.planner.PartitionDescriptor

PARTITION_BATCH_SIZE

Constructor Summary

Constructors

Constructor and Description
FileSystemPartitionDescriptor(PlannerSettings settings, org.apache.calcite.rel.core.TableScan scanRel)

Method Summary

Modifier and Type	Method and Description
protected void	createPartitionSublists() Create sublists of the partition locations, each sublist of size at most PartitionDescriptor.PARTITION_BATCH_SIZE
org.apache.calcite.rel.core.TableScan	createTableScan(List<PartitionLocation> newPartitionLocation, boolean wasAllPartitionsPruned) Create a new TableScan rel node, given the lists of new partitions or new files to SCAN.
org.apache.calcite.rel.core.TableScan	createTableScan(List<PartitionLocation> newPartitionLocation, org.apache.hadoop.fs.Path cacheFileRoot, boolean wasAllPartitionsPruned, MetadataContext metaContext) Create a new TableScan rel node, given the lists of new partitions or new files to scan and a path to a metadata cache file
org.apache.hadoop.fs.Path	getBaseTableLocation()
protected org.apache.commons.lang3.tuple.Pair<Collection<org.apache.hadoop.fs.Path>,Boolean>	getFileLocationsAndStatus()
Integer	getIdIfValid(String name) Check to see if the name is a partition name.
int	getMaxHierarchyLevel() Maximum level of partition nesting/ hierarchy supported
String	getName(int index)
int	getPartitionHierarchyIndex(String partitionName) Get the hierarchy index of the given partition For eg: if we have the partition laid out as follows 1997/q1/jan then getPartitionHierarchyIndex("jan") => 2
DrillTable	getTable()
TypeProtos.MajorType	getVectorType(SchemaPath column, PlannerSettings plannerSettings) Method returns the Major type associated with the given column
boolean	isPartitionName(String name) Given a column name return boolean to indicate if its a partition column or not
void	populatePartitionVectors(ValueVector[] vectors, List<PartitionLocation> partitions, BitSet partitionColumnBitSet, Map<Integer,String> fieldNameMap) Creates an in memory representation of all the partitions.
boolean	supportsMetadataCachePruning()

Methods inherited from class org.apache.drill.exec.planner.AbstractPartitionDescriptor

iterator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

FileSystemPartitionDescriptor

public FileSystemPartitionDescriptor(PlannerSettings settings,
                                     org.apache.calcite.rel.core.TableScan scanRel)

Method Detail

getPartitionHierarchyIndex

public int getPartitionHierarchyIndex(String partitionName)

Description copied from interface: PartitionDescriptor

Get the hierarchy index of the given partition For eg: if we have the partition laid out as follows 1997/q1/jan then getPartitionHierarchyIndex("jan") => 2

Parameters:

partitionName - Partition name

Returns:

the index of specified partition name in the hierarchy

isPartitionName

public boolean isPartitionName(String name)

Description copied from interface: PartitionDescriptor

Given a column name return boolean to indicate if its a partition column or not

Parameters:

name - of Partition

Returns:

true, if this is the partition name and vise versa.

getIdIfValid

public Integer getIdIfValid(String name)

Description copied from interface: PartitionDescriptor

Check to see if the name is a partition name.

Parameters:

name - The field name you want to compare to partition names.

Returns:

Return index if valid, otherwise return null;

getMaxHierarchyLevel

public int getMaxHierarchyLevel()

Description copied from interface: PartitionDescriptor

Maximum level of partition nesting/ hierarchy supported

Returns:

maximum supported level number of partition hierarchy

getTable

public DrillTable getTable()

populatePartitionVectors

public void populatePartitionVectors(ValueVector[] vectors,
                                     List<PartitionLocation> partitions,
                                     BitSet partitionColumnBitSet,
                                     Map<Integer,String> fieldNameMap)

Description copied from interface: PartitionDescriptor

Creates an in memory representation of all the partitions. For each level of partitioning we will create a value vector which this method will populate for all the partitions with the values of the partitioning key

Parameters:

vectors - - Array of vectors in the container that need to be populated

partitions - - List of all the partitions that exist in the table

partitionColumnBitSet - - Partition columns selected in the query

fieldNameMap - - Maps field ordinal to the field name

getVectorType

public TypeProtos.MajorType getVectorType(SchemaPath column,
                                          PlannerSettings plannerSettings)

Description copied from interface: PartitionDescriptor

Method returns the Major type associated with the given column

Parameters:

column - - column whose type should be determined

getName

public String getName(int index)

getBaseTableLocation

public org.apache.hadoop.fs.Path getBaseTableLocation()

createPartitionSublists

protected void createPartitionSublists()

Description copied from class: AbstractPartitionDescriptor

Create sublists of the partition locations, each sublist of size at most PartitionDescriptor.PARTITION_BATCH_SIZE

Specified by:

createPartitionSublists in class AbstractPartitionDescriptor

getFileLocationsAndStatus

protected org.apache.commons.lang3.tuple.Pair<Collection<org.apache.hadoop.fs.Path>,Boolean> getFileLocationsAndStatus()

createTableScan

public org.apache.calcite.rel.core.TableScan createTableScan(List<PartitionLocation> newPartitionLocation,
                                                             org.apache.hadoop.fs.Path cacheFileRoot,
                                                             boolean wasAllPartitionsPruned,
                                                             MetadataContext metaContext)
                                                      throws Exception

Description copied from interface: PartitionDescriptor

Create a new TableScan rel node, given the lists of new partitions or new files to scan and a path to a metadata cache file

Specified by:

createTableScan in interface PartitionDescriptor

Overrides:

createTableScan in class AbstractPartitionDescriptor

Throws:

Exception

createTableScan

public org.apache.calcite.rel.core.TableScan createTableScan(List<PartitionLocation> newPartitionLocation,
                                                             boolean wasAllPartitionsPruned)
                                                      throws Exception

Description copied from interface: PartitionDescriptor

Create a new TableScan rel node, given the lists of new partitions or new files to SCAN.

Throws:

Exception

supportsMetadataCachePruning

public boolean supportsMetadataCachePruning()

Specified by:

supportsMetadataCachePruning in interface PartitionDescriptor

Overrides:

supportsMetadataCachePruning in class AbstractPartitionDescriptor