com.microsoft.azure.cosmosdb.spark

CosmosDBSpark

object CosmosDBSpark extends LoggingTrait with Serializable

The CosmosDBSpark allow fast creation of RDDs, DataFrames or Datasets from CosmosDBSpark.

Since

1.0

Linear Supertypes
Serializable, Serializable, LoggingTrait, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CosmosDBSpark
  2. Serializable
  3. Serializable
  4. LoggingTrait
  5. AnyRef
  6. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class Builder extends AnyRef

    Builder for configuring and creating a CosmosDBSpark

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def builder(): Builder

    Create a builder for configuring the CosmosDBSpark

    Create a builder for configuring the CosmosDBSpark

    returns

    a CosmosDBSession Builder

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. val defaultSource: String

    The default source string for creating DataFrames from CosmosDB

  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  16. var lastUpsertSetting: Option[Boolean]

    For verfication purpose

  17. var lastWritingBatchSize: Option[Int]

  18. def load(jsc: JavaSparkContext, readConfig: Config): JavaCosmosDBRDD

    Load data from CosmosDB

    Load data from CosmosDB

    jsc

    the Spark context containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  19. def load(jsc: JavaSparkContext): JavaCosmosDBRDD

    Load data from CosmosDB

    Load data from CosmosDB

    jsc

    the Spark context containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  20. def load[D](sparkSession: SparkSession, readConfig: Config, clazz: Class[D]): Dataset[D]

    Load data from CosmosDB

    Load data from CosmosDB

    D

    The bean class defining the schema for the data

    sparkSession

    the SparkSession containing the CosmosDB connection configuration

    clazz

    the class of the data contained in the RDD

    returns

    a CosmosDBRDD

  21. def load[D <: Product](sparkSession: SparkSession, readConfig: Config)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[D]): DataFrame

    Load data from CosmosDB

    Load data from CosmosDB

    D

    The optional class defining the schema for the data

    sparkSession

    the SparkSession containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  22. def load[D <: Product](sparkSession: SparkSession)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[D]): DataFrame

    Load data from CosmosDB

    Load data from CosmosDB

    D

    The optional class defining the schema for the data

    sparkSession

    the SparkSession containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  23. def load(sc: SparkContext, readConfig: Config): CosmosDBRDD

    Load data from CosmosDB

    Load data from CosmosDB

    sc

    the Spark context containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  24. def load(sc: SparkContext): CosmosDBRDD

    Load data from CosmosDB

    Load data from CosmosDB

    sc

    the Spark context containing the CosmosDB connection configuration

    returns

    a CosmosDBRDD

  25. def log: Logger

    Attributes
    protected
    Definition Classes
    LoggingTrait

  26. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  27. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  28. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  29. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  30. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  31. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  32. def logName: String

    Attributes
    protected
    Definition Classes
    LoggingTrait

  33. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  34. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  36. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    LoggingTrait

  37. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  38. final def notify(): Unit

    Definition Classes
    AnyRef

  39. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  40. def read(sparkSession: SparkSession): DataFrameReader

    Creates a DataFrameReader with CosmosDB as the source

    Creates a DataFrameReader with CosmosDB as the source

    sparkSession

    the SparkSession

    returns

    the DataFrameReader

  41. def save[D](javaRDD: JavaRDD[D], writeConfig: Config, clazz: Class[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the writeConfig for the database information Requires a codec for the data type

    D

    the type of the data in the RDD

    javaRDD

    the RDD data to save to CosmosDB

    writeConfig

    the com.microsoft.azure.cosmosdb.spark.config.Config

    clazz

    the class of the data contained in the RDD

    returns

    the javaRDD

  42. def save(javaRDD: JavaRDD[Document], writeConfig: Config): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database information

    javaRDD

    the RDD data to save to CosmosDB

    writeConfig

    the com.microsoft.azure.cosmosdb.spark.config.Config

    returns

    the javaRDD

  43. def save[D](javaRDD: JavaRDD[D], clazz: Class[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database and collection information Requires a codec for the data type

    D

    the type of the data in the RDD

    javaRDD

    the RDD data to save to CosmosDB

    clazz

    the class of the data contained in the RDD

    returns

    the javaRDD

  44. def save(javaRDD: JavaRDD[Document]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database and collection information

    javaRDD

    the RDD data to save to CosmosDB

    returns

    the javaRDD

  45. def save(dataFrameWriter: DataFrameWriter[_], writeConfig: Config): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    dataFrameWriter

    the DataFrameWriter save to CosmosDB

    writeConfig

    the writeConfig

  46. def save(dataFrameWriter: DataFrameWriter[_]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database and collection information

    dataFrameWriter

    the DataFrameWriter save to CosmosDB

  47. def save[D](dataset: Dataset[D], writeConfig: Config)(implicit arg0: ClassTag[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Note: If the dataFrame contains an _id field the data will upserted and replace any existing documents in the collection.

    D
    dataset

    the dataset to save to CosmosDB

    writeConfig

    the writeConfig

    Since

    1.1.0

  48. def save[D](dataset: Dataset[D])(implicit arg0: ClassTag[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database and collection information

    Note: If the dataFrame contains an _id field the data will upserted and replace any existing documents in the collection.

    D
    dataset

    the dataset to save to CosmosDB

    Since

    1.1.0

  49. def save[D](rdd: RDD[D], writeConfig: Config)(implicit arg0: ClassTag[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    D

    the type of the data in the RDD

    rdd

    the RDD data to save to CosmosDB

    writeConfig

    the writeConfig

  50. def save[D](rdd: RDD[D])(implicit arg0: ClassTag[D]): Unit

    Save data to CosmosDB

    Save data to CosmosDB

    Uses the SparkConf for the database and collection information Requires a codec for the data type

    D

    the type of the data in the RDD

    rdd

    the RDD data to save to CosmosDB

  51. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  52. def toString(): String

    Definition Classes
    AnyRef → Any

  53. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  54. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  55. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  56. def write(dataFrame: DataFrame): DataFrameWriter[Row]

    Creates a DataFrameWriter with the CosmosDB underlying output data source.

    Creates a DataFrameWriter with the CosmosDB underlying output data source.

    dataFrame

    the DataFrame to convert into a DataFrameWriter

    returns

    the DataFrameWriter

Inherited from Serializable

Inherited from Serializable

Inherited from LoggingTrait

Inherited from AnyRef

Inherited from Any

Ungrouped