org.apache.drill.exec.store.dfs

Class FileSystemPlugin

java.lang.Object

org.apache.drill.exec.store.AbstractStoragePlugin

org.apache.drill.exec.store.dfs.FileSystemPlugin

All Implemented Interfaces:

AutoCloseable, SchemaFactory, StoragePlugin

public class FileSystemPlugin
extends AbstractStoragePlugin

A Storage engine associated with a Hadoop FileSystem Implementation. Examples include HDFS, MapRFS, QuantacastFileSystem, LocalFileSystem, as well Apache Drill specific CachedFileSystem, ClassPathFileSystem and LocalSyncableFileSystem. Tables are file names, directories and path patterns. This storage engine delegates to FSFormatEngines but shares references to the FileSystem configuration and path management.

Field Summary

Fields inherited from class org.apache.drill.exec.store.AbstractStoragePlugin

context

Fields inherited from interface org.apache.drill.exec.store.SchemaFactory

DEFAULT_WS_NAME

Constructor Summary

Constructors

Constructor and Description
FileSystemPlugin(FileSystemConfig config, DrillbitContext context, String name)

Method Summary

Modifier and Type	Method and Description
StoragePluginConfig	getConfig() Method returns a Jackson serializable object that extends a StoragePluginConfig.
FormatPlugin	getFormatPlugin(FormatPluginConfig config) If format plugin configuration is for named format plugin, will return format plugin from pre-loaded list by name.
FormatPlugin	getFormatPlugin(String name)
org.apache.hadoop.conf.Configuration	getFsConf()
Set<? extends org.apache.calcite.plan.RelOptRule>	getOptimizerRules(OptimizerRulesContext optimizerContext, PlannerPhase phase) TODO: Move this method to StoragePlugin interface in next major version release.
AbstractGroupScan	getPhysicalScan(String userName, JSONOptions selection, List<SchemaPath> columns) Get the physical scan operator for the particular GroupScan (read) node.
AbstractGroupScan	getPhysicalScan(String userName, JSONOptions selection, List<SchemaPath> columns, SessionOptionManager options, MetadataProviderManager metadataProviderManager) Get the physical scan operator for the particular GroupScan (read) node.
AbstractGroupScan	getPhysicalScan(String userName, JSONOptions selection, SessionOptionManager options) Get the physical scan operator for the particular GroupScan (read) node.
AbstractGroupScan	getPhysicalScan(String userName, JSONOptions selection, SessionOptionManager options, MetadataProviderManager metadataProviderManager) Get the physical scan operator for the particular GroupScan (read) node.
protected FormatCreator	newFormatCreator(FileSystemConfig config, DrillbitContext context, org.apache.hadoop.conf.Configuration fsConf) Creates a new FormatCreator instance.
void	registerSchemas(SchemaConfig schemaConfig, org.apache.calcite.schema.SchemaPlus parent) Register the schemas provided by this SchemaFactory implementation under the given parent schema.
boolean	supportsRead() Indicates if Drill can read the table from this format.

Methods inherited from class org.apache.drill.exec.store.AbstractStoragePlugin

close, getContext, getLogicalOptimizerRules, getName, getOptimizerRules, getPhysicalOptimizerRules, getPhysicalScan, getPhysicalScan, start, supportsWrite

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileSystemPlugin

public FileSystemPlugin(FileSystemConfig config,
                        DrillbitContext context,
                        String name)
                 throws ExecutionSetupException

Throws:

ExecutionSetupException

Method Detail

newFormatCreator

protected FormatCreator newFormatCreator(FileSystemConfig config,
                                         DrillbitContext context,
                                         org.apache.hadoop.conf.Configuration fsConf)

Creates a new FormatCreator instance. To be used by subclasses to return custom formats if required. Note that this method is called by the constructor, which fields may not be initialized yet.

Parameters:

config - the plugin configuration

context - the drillbit context

Returns:

a new FormatCreator instance

supportsRead

public boolean supportsRead()

Description copied from interface: StoragePlugin

Indicates if Drill can read the table from this format.

Specified by:

supportsRead in interface StoragePlugin

Overrides:

supportsRead in class AbstractStoragePlugin

getConfig

public StoragePluginConfig getConfig()

Description copied from interface: StoragePlugin

Method returns a Jackson serializable object that extends a StoragePluginConfig.

Returns:

an extension of StoragePluginConfig

getPhysicalScan

public AbstractGroupScan getPhysicalScan(String userName,
                                         JSONOptions selection,
                                         SessionOptionManager options)
                                  throws IOException

Description copied from interface: StoragePlugin

Get the physical scan operator for the particular GroupScan (read) node.

Specified by:

getPhysicalScan in interface StoragePlugin

Overrides:

getPhysicalScan in class AbstractStoragePlugin

Parameters:

userName - User whom to impersonate when when reading the contents as part of Scan.

selection - The configured storage engine specific selection.

options - (optional) session options

Returns:

The physical scan operator for the particular GroupScan (read) node.

Throws:

IOException

getPhysicalScan

public AbstractGroupScan getPhysicalScan(String userName,
                                         JSONOptions selection,
                                         SessionOptionManager options,
                                         MetadataProviderManager metadataProviderManager)
                                  throws IOException

Description copied from interface: StoragePlugin

Get the physical scan operator for the particular GroupScan (read) node.

Specified by:

getPhysicalScan in interface StoragePlugin

Overrides:

getPhysicalScan in class AbstractStoragePlugin

Parameters:

userName - User whom to impersonate when when reading the contents as part of Scan.

selection - The configured storage engine specific selection.

options - (optional) session options

metadataProviderManager - manager for handling metadata providers

Returns:

The physical scan operator for the particular GroupScan (read) node.

Throws:

IOException

getPhysicalScan

public AbstractGroupScan getPhysicalScan(String userName,
                                         JSONOptions selection,
                                         List<SchemaPath> columns)
                                  throws IOException

Description copied from interface: StoragePlugin

Get the physical scan operator for the particular GroupScan (read) node.

Specified by:

getPhysicalScan in interface StoragePlugin

Overrides:

getPhysicalScan in class AbstractStoragePlugin

Parameters:

userName - User whom to impersonate when when reading the contents as part of Scan.

selection - The configured storage engine specific selection.

columns - (optional) The list of column names to scan from the data source.

Returns:

The physical scan operator for the particular GroupScan (read) node.

Throws:

IOException

getPhysicalScan

public AbstractGroupScan getPhysicalScan(String userName,
                                         JSONOptions selection,
                                         List<SchemaPath> columns,
                                         SessionOptionManager options,
                                         MetadataProviderManager metadataProviderManager)
                                  throws IOException

Description copied from interface: StoragePlugin

Get the physical scan operator for the particular GroupScan (read) node.

Specified by:

getPhysicalScan in interface StoragePlugin

Overrides:

getPhysicalScan in class AbstractStoragePlugin

Parameters:

userName - User whom to impersonate when when reading the contents as part of Scan.

selection - The configured storage engine specific selection.

columns - (optional) The list of column names to scan from the data source.

options - (optional) session options

metadataProviderManager - manager for handling metadata providers

Returns:

The physical scan operator for the particular GroupScan (read) node.

Throws:

IOException

registerSchemas

public void registerSchemas(SchemaConfig schemaConfig,
                            org.apache.calcite.schema.SchemaPlus parent)
                     throws IOException

Description copied from interface: SchemaFactory

Register the schemas provided by this SchemaFactory implementation under the given parent schema.

Parameters:

schemaConfig - Configuration for schema objects.

parent - Reference to parent schema.

Throws:

IOException - in case of error during schema registration

getFormatPlugin

public FormatPlugin getFormatPlugin(String name)

getFormatPlugin

public FormatPlugin getFormatPlugin(FormatPluginConfig config)

If format plugin configuration is for named format plugin, will return format plugin from pre-loaded list by name. For other cases will try to find format plugin by its configuration, if not present will attempt to create one.

Specified by:

getFormatPlugin in interface StoragePlugin

Overrides:

getFormatPlugin in class AbstractStoragePlugin

Parameters:

config - format plugin configuration

Returns:

format plugin for given configuration if found, null otherwise

getOptimizerRules

public Set<? extends org.apache.calcite.plan.RelOptRule> getOptimizerRules(OptimizerRulesContext optimizerContext,
                                                                           PlannerPhase phase)

Description copied from class: AbstractStoragePlugin

TODO: Move this method to StoragePlugin interface in next major version release.

Overrides:

getOptimizerRules in class AbstractStoragePlugin

getFsConf

public org.apache.hadoop.conf.Configuration getFsConf()