org.apache.iceberg.hadoop

Class HadoopCatalog

java.lang.Object

org.apache.iceberg.BaseMetastoreCatalog

org.apache.iceberg.hadoop.HadoopCatalog

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.iceberg.catalog.Catalog, org.apache.iceberg.catalog.SupportsNamespaces

public class HadoopCatalog
extends BaseMetastoreCatalog
implements java.io.Closeable, org.apache.iceberg.catalog.SupportsNamespaces

HadoopCatalog provides a way to use table names like db.table to work with path-based tables under a common location. It uses a specified directory under a specified filesystem as the warehouse directory, and organizes multiple levels directories that mapped to the database, namespace and the table respectively. The HadoopCatalog takes a location as the warehouse directory. When creating a table such as $db.$tbl, it creates $db/$tbl directory under the warehouse directory, and put the table metadata into that directory. The HadoopCatalog now supports Catalog.createTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.Schema, org.apache.iceberg.PartitionSpec, java.lang.String, java.util.Map<java.lang.String, java.lang.String>), Catalog.dropTable(org.apache.iceberg.catalog.TableIdentifier), the Catalog.renameTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.catalog.TableIdentifier) is not supported yet. Note: The HadoopCatalog requires that the underlying file system supports atomic rename.

Constructor Summary

Constructors

Constructor and Description
HadoopCatalog(org.apache.hadoop.conf.Configuration conf) The constructor of the HadoopCatalog.
HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation) The constructor of the HadoopCatalog.
HadoopCatalog(java.lang.String name, org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation) The constructor of the HadoopCatalog.

Method Summary

Modifier and Type	Method and Description
void	close()
void	createNamespace(org.apache.iceberg.catalog.Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
org.apache.iceberg.Table	createTable(org.apache.iceberg.catalog.TableIdentifier identifier, org.apache.iceberg.Schema schema, org.apache.iceberg.PartitionSpec spec, java.lang.String location, java.util.Map<java.lang.String,java.lang.String> properties)
protected java.lang.String	defaultWarehouseLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)
boolean	dropNamespace(org.apache.iceberg.catalog.Namespace namespace)
boolean	dropTable(org.apache.iceberg.catalog.TableIdentifier identifier, boolean purge)
protected boolean	isValidIdentifier(org.apache.iceberg.catalog.TableIdentifier identifier)
java.util.List<org.apache.iceberg.catalog.Namespace>	listNamespaces(org.apache.iceberg.catalog.Namespace namespace)
java.util.List<org.apache.iceberg.catalog.TableIdentifier>	listTables(org.apache.iceberg.catalog.Namespace namespace)
java.util.Map<java.lang.String,java.lang.String>	loadNamespaceMetadata(org.apache.iceberg.catalog.Namespace namespace)
protected java.lang.String	name()
protected TableOperations	newTableOps(org.apache.iceberg.catalog.TableIdentifier identifier)
boolean	removeProperties(org.apache.iceberg.catalog.Namespace namespace, java.util.Set<java.lang.String> properties)
void	renameTable(org.apache.iceberg.catalog.TableIdentifier from, org.apache.iceberg.catalog.TableIdentifier to)
boolean	setProperties(org.apache.iceberg.catalog.Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)

Methods inherited from class org.apache.iceberg.BaseMetastoreCatalog

dropTableData, fullTableName, loadTable, newCreateTableTransaction, newReplaceTableTransaction, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.iceberg.catalog.SupportsNamespaces

createNamespace, listNamespaces, namespaceExists

Methods inherited from interface org.apache.iceberg.catalog.Catalog

createTable, createTable, createTable, dropTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists

Constructor Detail

HadoopCatalog

public HadoopCatalog(java.lang.String name,
                     org.apache.hadoop.conf.Configuration conf,
                     java.lang.String warehouseLocation)

The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.

Parameters:

name - The catalog name

conf - The Hadoop configuration

warehouseLocation - The location used as warehouse directory

HadoopCatalog

public HadoopCatalog(org.apache.hadoop.conf.Configuration conf,
                     java.lang.String warehouseLocation)

The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.

Parameters:

conf - The Hadoop configuration

warehouseLocation - The location used as warehouse directory

HadoopCatalog

public HadoopCatalog(org.apache.hadoop.conf.Configuration conf)

The constructor of the HadoopCatalog. It gets the value of fs.defaultFS property from the passed Hadoop configuration as its default file system, and use the default directory iceberg/warehouse as the warehouse directory.

Parameters:

conf - The Hadoop configuration

Method Detail

name

protected java.lang.String name()

Specified by:

name in class BaseMetastoreCatalog

listTables

public java.util.List<org.apache.iceberg.catalog.TableIdentifier> listTables(org.apache.iceberg.catalog.Namespace namespace)

Specified by:

listTables in interface org.apache.iceberg.catalog.Catalog

createTable

public org.apache.iceberg.Table createTable(org.apache.iceberg.catalog.TableIdentifier identifier,
                                            org.apache.iceberg.Schema schema,
                                            org.apache.iceberg.PartitionSpec spec,
                                            java.lang.String location,
                                            java.util.Map<java.lang.String,java.lang.String> properties)

Specified by:

createTable in interface org.apache.iceberg.catalog.Catalog

Overrides:

createTable in class BaseMetastoreCatalog

isValidIdentifier

protected boolean isValidIdentifier(org.apache.iceberg.catalog.TableIdentifier identifier)

Overrides:

isValidIdentifier in class BaseMetastoreCatalog

newTableOps

protected TableOperations newTableOps(org.apache.iceberg.catalog.TableIdentifier identifier)

Specified by:

newTableOps in class BaseMetastoreCatalog

defaultWarehouseLocation

protected java.lang.String defaultWarehouseLocation(org.apache.iceberg.catalog.TableIdentifier tableIdentifier)

Specified by:

defaultWarehouseLocation in class BaseMetastoreCatalog

dropTable

public boolean dropTable(org.apache.iceberg.catalog.TableIdentifier identifier,
                         boolean purge)

Specified by:

dropTable in interface org.apache.iceberg.catalog.Catalog

renameTable

public void renameTable(org.apache.iceberg.catalog.TableIdentifier from,
                        org.apache.iceberg.catalog.TableIdentifier to)

Specified by:

renameTable in interface org.apache.iceberg.catalog.Catalog

createNamespace

public void createNamespace(org.apache.iceberg.catalog.Namespace namespace,
                            java.util.Map<java.lang.String,java.lang.String> meta)

Specified by:

createNamespace in interface org.apache.iceberg.catalog.SupportsNamespaces

listNamespaces

public java.util.List<org.apache.iceberg.catalog.Namespace> listNamespaces(org.apache.iceberg.catalog.Namespace namespace)

Specified by:

listNamespaces in interface org.apache.iceberg.catalog.SupportsNamespaces

dropNamespace

public boolean dropNamespace(org.apache.iceberg.catalog.Namespace namespace)

Specified by:

dropNamespace in interface org.apache.iceberg.catalog.SupportsNamespaces

setProperties

public boolean setProperties(org.apache.iceberg.catalog.Namespace namespace,
                             java.util.Map<java.lang.String,java.lang.String> properties)

Specified by:

setProperties in interface org.apache.iceberg.catalog.SupportsNamespaces

removeProperties

public boolean removeProperties(org.apache.iceberg.catalog.Namespace namespace,
                                java.util.Set<java.lang.String> properties)

Specified by:

removeProperties in interface org.apache.iceberg.catalog.SupportsNamespaces

loadNamespaceMetadata

public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(org.apache.iceberg.catalog.Namespace namespace)

Specified by:

loadNamespaceMetadata in interface org.apache.iceberg.catalog.SupportsNamespaces

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException