Class

org.apache.spark.sql.execution.columnar.impl

BaseColumnFormatRelation

Related Doc: package impl

Permalink

abstract class BaseColumnFormatRelation extends JDBCAppendableRelation with PartitionedDataSourceScan with RowInsertableRelation with MutableRelation

This class acts as a DataSource provider for column format tables provided Snappy. It uses GemFireXD as actual datastore to physically locate the tables. Column tables can be used for storing data in columnar compressed format. A example usage is given below.

val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) snc.createTable(tableName, "column", dataDF.schema, props) dataDF.write.insertInto(tableName)

This provider scans underlying tables in parallel and is aware of the data partition. It does not introduces a shuffle if simple table query is fired. One can insert a single or multiple rows into this table as well as do a bulk insert by a Spark DataFrame. Bulk insert example is shown above.

Linear Supertypes
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BaseColumnFormatRelation
  2. MutableRelation
  3. RowInsertableRelation
  4. PartitionedDataSourceScan
  5. JDBCAppendableRelation
  6. Product
  7. Equals
  8. Serializable
  9. Serializable
  10. NativeTableRowLevelSecurityRelation
  11. RowLevelSecurityRelation
  12. Logging
  13. IndexableRelation
  14. PlanInsertableRelation
  15. NativeTableRelation
  16. DestroyRelation
  17. InsertableRelation
  18. PrunedUnsafeFilteredScan
  19. BaseRelation
  20. AnyRef
  21. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BaseColumnFormatRelation(_table: String, _provider: String, _mode: SaveMode, _userSchema: StructType, schemaExtensions: String, ddlExtensionForShadowTable: String, _origOptions: CaseInsensitiveMutableHashMap[String], _externalStore: ExternalStore, partitioningColumns: Seq[String], _context: SQLContext, _relationInfo: (RelationInfo, Option[LocalRegion]))

    Permalink

Abstract Value Members

  1. abstract def getColocatedTable: Option[String]

    Permalink
    Definition Classes
    PartitionedDataSourceScan

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. var _relationInfoAndRegion: (RelationInfo, Option[LocalRegion])

    Permalink
    Attributes
    protected[this]
    Definition Classes
    NativeTableRowLevelSecurityRelation
  5. var _schema: StructType

    Permalink
    Attributes
    protected[this]
    Definition Classes
    NativeTableRowLevelSecurityRelation
  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. def buildRowBufferRDD(partitionEvaluator: () ⇒ Array[Partition], requiredColumns: Array[String], filters: Array[Expression], useResultSet: Boolean, projection: Array[Int]): RDD[Any]

    Permalink
  8. def buildUnsafeScan(requiredColumns: Array[String], filters: Array[Expression]): (RDD[Any], Seq[RDD[InternalRow]])

    Permalink
  9. def buildUnsafeScanForSampledRelation(requiredColumns: Array[String], filters: Array[Expression]): (RDD[Any], RDD[Any], Seq[RDD[InternalRow]])

    Permalink
  10. def canEqual(that: Any): Boolean

    Permalink
    Definition Classes
    JDBCAppendableRelation → Equals
  11. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. final val connFactory: () ⇒ Connection

    Permalink
  13. final def connProperties: ConnectionProperties

    Permalink
  14. val connectionType: ConnectionType.Value

    Permalink
  15. def createActualTables(conn: Connection): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    BaseColumnFormatRelationNativeTableRowLevelSecurityRelation
  16. def createIndex(indexIdent: TableIdentifier, tableIdent: TableIdentifier, indexColumns: Seq[(String, Option[SortDirection])], options: Map[String, String]): Unit

    Permalink

    Create an index on a table.

    Create an index on a table.

    indexIdent

    Index Identifier which goes in the catalog

    tableIdent

    Table identifier on which the index is created.

    indexColumns

    Columns on which the index has to be created with the direction of sorting. Direction can be specified as None.

    options

    Options for indexes. For e.g. column table index - ("COLOCATE_WITH"->"CUSTOMER"). row table index - ("INDEX_TYPE"->"GLOBAL HASH") or ("INDEX_TYPE"->"UNIQUE")

    Definition Classes
    JDBCAppendableRelationIndexableRelation
  17. def createTable(conn: Connection, tableStr: String, tableName: String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    JDBCAppendableRelation
  18. def createTable(mode: SaveMode): Unit

    Permalink
  19. val ddlExtensionForShadowTable: String

    Permalink
  20. var delayRollover: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    JDBCAppendableRelation
  21. def destroy(ifExists: Boolean): Unit

    Permalink

    Destroy and cleanup this relation.

    Destroy and cleanup this relation. It may include, but not limited to, dropping the external table that this relation represents.

    Definition Classes
    BaseColumnFormatRelationDestroyRelation
  22. final def dialect: JdbcDialect

    Permalink
    Attributes
    protected
    Definition Classes
    JDBCAppendableRelationNativeTableRelation
  23. def dropIndex(indexIdent: TableIdentifier, tableIdent: TableIdentifier, ifExists: Boolean): Unit

    Permalink

    Drops an index on this table

    Drops an index on this table

    indexIdent

    Index identifier

    tableIdent

    Table identifier

    ifExists

    Drop if exists

    Definition Classes
    JDBCAppendableRelationIndexableRelation
  24. def enableOrDisableRowLevelSecurity(tableIdent: TableIdentifier, enableRowLevelSecurity: Boolean): Unit

    Permalink
  25. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. def equals(that: Any): Boolean

    Permalink
    Definition Classes
    JDBCAppendableRelation → Equals → AnyRef → Any
  27. def executeUpdate(sql: String, defaultSchema: String): Int

    Permalink

    Execute a DML SQL and return the number of rows affected.

    Execute a DML SQL and return the number of rows affected.

    Definition Classes
    NativeTableRelation
  28. val externalStore: ExternalStore

    Permalink
    Definition Classes
    JDBCAppendableRelation
  29. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  30. def flushRowBuffer(): Unit

    Permalink
  31. def getBasicInsertPlan(relation: LogicalRelation, child: SparkPlan): SparkPlan

    Permalink
  32. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  33. def getColumnBatchParams: (Int, Int, String)

    Permalink
    Definition Classes
    JDBCAppendableRelation
  34. def getColumnBatchStatistics(schema: Seq[AttributeReference]): PartitionStatistics

    Permalink
  35. def getCompressionCodec: String

    Permalink
    Definition Classes
    JDBCAppendableRelation
  36. def getDeletePlan(relation: LogicalRelation, child: SparkPlan, keyColumns: Seq[Attribute]): SparkPlan

    Permalink

    Get a spark plan to delete rows the relation.

    Get a spark plan to delete rows the relation. The result of SparkPlan execution should be a count of number of updated rows.

    Definition Classes
    BaseColumnFormatRelationMutableRelation
  37. def getInsertPlan(relation: LogicalRelation, child: SparkPlan): SparkPlan

    Permalink

    Get a spark plan for insert.

    Get a spark plan for insert. The result of SparkPlan execution should be a count of number of inserted rows.

    Definition Classes
    PlanInsertableRelation
  38. def getKeyColumns: Seq[String]

    Permalink

    Get the "key" columns for the table that need to be projected out by UPDATE and DELETE operations for affecting the selected rows.

    Get the "key" columns for the table that need to be projected out by UPDATE and DELETE operations for affecting the selected rows.

    Definition Classes
    BaseColumnFormatRelationMutableRelation
  39. def getPrimaryKeyColumns(session: SnappySession): Seq[String]

    Permalink

    Get key columns of the column table

    Get key columns of the column table

    Definition Classes
    BaseColumnFormatRelationMutableRelation
  40. def getUpdatePlan(relation: LogicalRelation, child: SparkPlan, updateColumns: Seq[Attribute], updateExpressions: Seq[Expression], keyColumns: Seq[Attribute]): SparkPlan

    Permalink

    Get a spark plan to update rows in the relation.

    Get a spark plan to update rows in the relation. The result of SparkPlan execution should be a count of number of updated rows.

    Definition Classes
    BaseColumnFormatRelationMutableRelation
  41. def hashCode(): Int

    Permalink
    Definition Classes
    JDBCAppendableRelation → AnyRef → Any
  42. def initializeLogIfNecessary(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  43. def insert(rows: Seq[Row]): Int

    Permalink

    Insert a sequence of rows into the table represented by this relation.

    Insert a sequence of rows into the table represented by this relation.

    rows

    the rows to be inserted

    returns

    number of rows inserted

    Definition Classes
    BaseColumnFormatRelationRowInsertableRelation
  44. def insert(data: DataFrame, overwrite: Boolean): Unit

    Permalink
    Definition Classes
    JDBCAppendableRelation → InsertableRelation
  45. final def isDebugEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  46. final def isInfoEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  47. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  48. def isPartitioned: Boolean

    Permalink
  49. def isRowLevelSecurityEnabled: Boolean

    Permalink
  50. final def isRowTable: Boolean

    Permalink
  51. final def isTraceEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  52. final var levelFlags: Int

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  53. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  54. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  55. def logDebug(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  56. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  57. def logError(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  58. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  59. def logInfo(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  60. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  62. def logTrace(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  63. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  64. def logWarning(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  65. final var log_: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. val mode: SaveMode

    Permalink
    Definition Classes
    JDBCAppendableRelation
  67. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  68. val needConversion: Boolean

    Permalink
    Definition Classes
    JDBCAppendableRelation → BaseRelation
  69. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  70. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  71. def numBuckets: Int

    Permalink
  72. val origOptions: CaseInsensitiveMutableHashMap[String]

    Permalink
    Definition Classes
    JDBCAppendableRelation
  73. def partitionColumns: Seq[String]

    Permalink

    Get the partitioning columns for the table, if any.

    Get the partitioning columns for the table, if any.

    Definition Classes
    BaseColumnFormatRelationMutableRelationPartitionedDataSourceScan
  74. val partitioningColumns: Seq[String]

    Permalink
  75. val provider: String

    Permalink
    Definition Classes
    JDBCAppendableRelation
  76. def refreshTableSchema(invalidateCached: Boolean, fetchFromStore: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    NativeTableRowLevelSecurityRelation
  77. lazy val region: LocalRegion

    Permalink
  78. def relationInfo: RelationInfo

    Permalink
  79. def relationInfoAndRegion: (RelationInfo, Option[LocalRegion])

    Permalink
    Attributes
    protected
    Definition Classes
    NativeTableRowLevelSecurityRelation
  80. def resetLogger(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  81. final def resolvedName: String

    Permalink
  82. lazy val rowInsertStr: String

    Permalink
  83. def scanTable(tableName: String, requiredColumns: Array[String], filters: Array[Expression], _ignore: () ⇒ Int): (RDD[Any], Array[Int])

    Permalink
  84. def schema: StructType

    Permalink
    Definition Classes
    NativeTableRowLevelSecurityRelation → BaseRelation
  85. val schemaExtensions: String

    Permalink
  86. lazy val schemaName: String

    Permalink
  87. def sizeInBytes: Long

    Permalink
    Definition Classes
    JDBCAppendableRelation → BaseRelation
  88. val sqlContext: SQLContext

    Permalink
    Definition Classes
    JDBCAppendableRelation → BaseRelation
  89. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  90. val table: String

    Permalink

    Name of this table as stored in catalog.

    Name of this table as stored in catalog.

    Definition Classes
    JDBCAppendableRelationNativeTableRelation
  91. lazy val tableName: String

    Permalink
  92. def toString(): String

    Permalink
    Definition Classes
    BaseColumnFormatRelationJDBCAppendableRelation → AnyRef → Any
  93. def truncate(): Unit

    Permalink

    Truncate the table represented by this relation.

    Truncate the table represented by this relation.

    Definition Classes
    BaseColumnFormatRelationDestroyRelation
  94. def unhandledFilters(filters: Seq[Expression]): Seq[Expression]

    Permalink

    Returns the list of Expressions that this datasource may not be able to handle.

    Returns the list of Expressions that this datasource may not be able to handle. By default, this function will return all filters, as it is always safe to double evaluate an Expression.

    Definition Classes
    BaseColumnFormatRelationPrunedUnsafeFilteredScan
  95. def unhandledFilters(filters: Array[Filter]): Array[Filter]

    Permalink
    Definition Classes
    BaseRelation
  96. val userSchema: StructType

    Permalink
    Definition Classes
    JDBCAppendableRelation
  97. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  98. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  99. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  100. def withKeyColumns(relation: LogicalRelation, keyColumns: Seq[String]): LogicalRelation

    Permalink

    If required inject the key columns in the original relation.

    If required inject the key columns in the original relation.

    Definition Classes
    MutableRelation
  101. def withTableWriteLock(f: () ⇒ SparkPlan): SparkPlan

    Permalink
  102. def withoutUserSchema: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    BaseColumnFormatRelationNativeTableRowLevelSecurityRelation

Inherited from MutableRelation

Inherited from RowInsertableRelation

Inherited from PartitionedDataSourceScan

Inherited from JDBCAppendableRelation

Inherited from Product

Inherited from Equals

Inherited from Serializable

Inherited from Serializable

Inherited from RowLevelSecurityRelation

Inherited from Logging

Inherited from IndexableRelation

Inherited from PlanInsertableRelation

Inherited from NativeTableRelation

Inherited from DestroyRelation

Inherited from InsertableRelation

Inherited from PrunedUnsafeFilteredScan

Inherited from BaseRelation

Inherited from AnyRef

Inherited from Any

Ungrouped