Trait

com.mongodb.spark.config

MongoInputConfig

Related Doc: package config

Permalink

trait MongoInputConfig extends MongoCompanionConfig

Mongo input configurations

Configurations used when reading from MongoDB

Configuration Properties

The prefix when using sparkConf is: spark.mongodb.input. followed by the property name:

Since

1.0

See also

com.mongodb.spark.config.ReadConfig$

Linear Supertypes
MongoCompanionConfig, Serializable, Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MongoInputConfig
  2. MongoCompanionConfig
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. abstract type Self

    Permalink

    The type of the MongoConfig

    The type of the MongoConfig

    Definition Classes
    MongoCompanionConfig

Abstract Value Members

  1. abstract def apply(options: Map[String, String], default: Option[Self]): Self

    Permalink

    Create a configuration from the values in the Map, using the optional default configuration for any default values.

    Create a configuration from the values in the Map, using the optional default configuration for any default values.

    Note: Values in the map do not need to be prefixed with the configPrefix.

    options

    a map of properties and their string values

    default

    the optional default configuration, used for determining the default values for the properties

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
  2. abstract def create(options: Map[String, String], default: Self): Self

    Permalink

    Create a configuration easily from the Java API using the values in the Map, using the optional default configuration for any default values.

    Create a configuration easily from the Java API using the values in the Map, using the optional default configuration for any default values.

    Note: Values in the map do not need to be prefixed with the configPrefix.

    options

    a map of properties and their string values

    default

    the optional default configuration, used for determining the default values for the properties

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
  3. abstract def create(options: Map[String, String]): Self

    Permalink

    Create a configuration easily from the Java API using the values in the Map

    Create a configuration easily from the Java API using the values in the Map

    Note: Values in the map do not need to be prefixed with the configPrefix.

    options

    a map of properties and their string values

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
  4. abstract def create(sparkConf: SparkConf, options: Map[String, String]): Self

    Permalink

    Create a configuration from the sparkConf

    Create a configuration from the sparkConf

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkConf

    the spark configuration

    options

    overloaded parameters

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  5. abstract def create(sparkConf: SparkConf): Self

    Permalink

    Create a configuration easily from the Java API using the sparkConf

    Create a configuration easily from the Java API using the sparkConf

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkConf

    the spark configuration

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  6. abstract def create(sparkSession: SparkSession): Self

    Permalink

    Create a configuration easily from the Java API using the JavaSparkContext

    Create a configuration easily from the Java API using the JavaSparkContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkSession

    the SparkSession

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  7. abstract def create(javaSparkContext: JavaSparkContext): Self

    Permalink

    Create a configuration easily from the Java API using the JavaSparkContext

    Create a configuration easily from the Java API using the JavaSparkContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    javaSparkContext

    the java spark context

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  8. abstract def create(sqlContext: SQLContext): Self

    Permalink

    Create a configuration easily from the Java API using the JavaSparkContext

    Create a configuration easily from the Java API using the JavaSparkContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sqlContext

    the SQL context

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) As of Spark 2.0 SQLContext was replaced by SparkSession. Use the SparkSession method instead

    See also

    configPrefix

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val allowDiskUseProperty: String

    Permalink

    The allow disk use property

    The allow disk use property

    Enables writing to temporary files

    Since

    2.3.1

  5. def apply(options: Map[String, String]): Self

    Permalink

    Create a configuration from the values in the Map

    Create a configuration from the values in the Map

    Note: Values in the map do not need to be prefixed with the configPrefix.

    options

    a map of properties and their string values

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
  6. def apply(sparkConf: SparkConf, options: Map[String, String]): Self

    Permalink

    Create a configuration from the sparkConf

    Create a configuration from the sparkConf

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkConf

    the spark configuration

    options

    overloaded parameters

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  7. def apply(sparkConf: SparkConf): Self

    Permalink

    Create a configuration from the sparkConf

    Create a configuration from the sparkConf

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkConf

    the spark configuration

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  8. def apply(sparkSession: SparkSession): Self

    Permalink

    Create a configuration from the sqlContext

    Create a configuration from the sqlContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkSession

    the SparkSession

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  9. def apply(sparkContext: SparkContext): Self

    Permalink

    Create a configuration from the sparkContext

    Create a configuration from the sparkContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sparkContext

    the spark context

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    See also

    configPrefix

  10. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  11. val batchSizeProperty: String

    Permalink

    The batch size property

    The batch size property

    The size of batches used by the underlying cursor. Smaller batches will result in more round trips to MongoDB.

    Default: The servers default

    Since

    2.4.1

  12. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. val collationProperty: String

    Permalink

    The collation property

    The collation property

    The json representation of a Collation. Created via Collation.asDocument.toJson.

    Since

    2.3

  14. def collectionName(collectionNameProperty: String, options: Map[String, String], default: Option[String] = None): String

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  15. val collectionNameProperty: String

    Permalink

    The collection name property

  16. val configPrefix: String

    Permalink

    The configuration prefix string for the current configuration scope

    The configuration prefix string for the current configuration scope

    Definition Classes
    MongoInputConfigMongoCompanionConfig
  17. def connectionString(options: Map[String, String]): ConnectionString

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  18. def databaseName(databaseNameProperty: String, options: Map[String, String], default: Option[String] = None): String

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  19. val databaseNameProperty: String

    Permalink

    The database name property

  20. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  21. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  22. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  23. def getBoolean(newValue: Option[String], existingValue: Option[Boolean] = None, defaultValue: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  24. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  25. def getInt(newValue: Option[String], existingValue: Option[Int] = None, defaultValue: Int): Int

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  26. def getOptionsFromConf(sparkConf: SparkConf): Map[String, String]

    Permalink

    Gets an options map from the SparkConf

    Gets an options map from the SparkConf

    sparkConf

    the SparkConf

    returns

    the options

    Definition Classes
    MongoCompanionConfig
  27. def getString(newValue: Option[String], existingValue: Option[String] = None, defaultValue: String): String

    Permalink
    Attributes
    protected
    Definition Classes
    MongoCompanionConfig
  28. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  29. val hintProperty: String

    Permalink

    The hint property

    The hint property

    The json representation of a hint document

    Since

    2.3

  30. val inferSchemaMapTypeEnabledProperty: String

    Permalink

    The infer schema MapType enabled property

    The infer schema MapType enabled property

    A boolean flag to enable or disable MapType infer. If this flag is enabled, large compatible struct types will be inferred to a MapType instead.

    Default: true

    Since

    2.3

  31. val inferSchemaMapTypeMinimumKeysProperty: String

    Permalink

    The infer schema MapType minimum keys property

    The infer schema MapType minimum keys property

    The minimum keys property controls how large a struct must be before a MapType should be inferred.

    Default: 250

    Since

    2.3

  32. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  33. val localThresholdProperty: String

    Permalink

    The localThreshold property

    The localThreshold property

    The local threshold in milliseconds is used when choosing among multiple MongoDB servers to send a request. Only servers whose ping time is less than or equal to the server with the fastest ping time *plus* the local threshold will be chosen.

    For example when choosing which MongoS to send a request through a localThreshold of 0 would pick the MongoS with the fastest ping time.

    Default: 15 ms

  34. val mongoURIProperty: String

    Permalink

    The mongo URI property

    The mongo URI property

    Represents a connection string.

    Any values set in the connection string will override any default values for the configuration.

    Definition Classes
    MongoCompanionConfig
  35. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  36. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  37. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  38. val partitionerOptionsProperty: String

    Permalink

    The partitioner options property

    The partitioner options property

    Represents a map of options for customising the configuration of a partitioner. Default: Map.empty[String, String]

  39. val partitionerProperty: String

    Permalink

    The partition property

    The partition property

    Represents the name of the partitioner to use when partitioning the data in the collection. Default: MongoDefaultPartitioner

  40. val pipelineIncludeFiltersAndProjectionsProperty: String

    Permalink

    The sql include pipeline filters and projections property

    The sql include pipeline filters and projections property

    A boolean flag to enable or disable pushing down filters and projections into MongoDB when using spark sql. A false value will be expensive as all data will be sent to spark and filtered in Spark.

    Default: true

    Since

    2.3

  41. val pipelineIncludeNullFiltersProperty: String

    Permalink

    The sql include null filters in the pipeline property

    The sql include null filters in the pipeline property

    A boolean flag to enable or disable pushing null value checks into MongoDB when using spark sql. These ensure that the value exists and is not null for each not nullable field.

    Default: true

    Since

    2.3

  42. val pipelineProperty: String

    Permalink

    The pipeline property

    The pipeline property

    Enables custom aggregation pipelines to applied to the collection before sending to Spark. When configuring this should either be an extended json representation of a list of documents:

    """[{"$match": {"closed": false}}, {"$project": {"status": 1, "name": 1, "description": 1}}]"""

    Or the extended json syntax of a single document:

    """{"$match": {"closed": false}}"""

    Note: Custom aggregation pipelines must work with the partitioner strategy. Some stages such as $group may not work as expected.

    Since

    2.3.1

  43. val readConcernLevelProperty: String

    Permalink

    The ReadConcern level property

    The ReadConcern level property

    Default: DEFAULT

    See also

    ReadConcernConfig

  44. val readPreferenceNameProperty: String

    Permalink

    The ReadPreference name property

    The ReadPreference name property

    Default: primary

    See also

    ReadPreferenceConfig

  45. val readPreferenceTagSetsProperty: String

    Permalink

    The ReadPreference tags property

    The ReadPreference tags property

    See also

    ReadPreferenceConfig

  46. val registerSQLHelperFunctions: String

    Permalink
  47. val registerSQLHelperFunctionsProperty: String

    Permalink

    Register SQL Helper functions

    Register SQL Helper functions

    The SQL helper functions allow easy querying of Bson types inside SQL queries

    Since

    1.1

  48. val samplePoolSizeProperty: String

    Permalink

    The sample pool size property

    The sample pool size property

    The size of the pool to take a sample from, used when there is no $sample support or if there is a pushed down aggregation. Can be used to significantly reduce the costs of inferring the schema. A negative value disables limiting when using $sample and will sample from the whole collection.

    Default: 10000

    Since

    2.3.1

  49. val sampleSizeProperty: String

    Permalink

    The sample size property

    The sample size property

    Used when sampling data from MongoDB to determine the Schema. Should be equal or less than the sample pool size. Default: 1000

  50. def stripPrefix(options: Map[String, String]): Map[String, String]

    Permalink

    Strip the prefix from options

    Strip the prefix from options

    options

    options that may contain the prefix

    returns

    prefixLess options

    Definition Classes
    MongoCompanionConfig
  51. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  52. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  53. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  54. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  55. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def apply(sqlContext: SQLContext): Self

    Permalink

    Create a configuration from the sqlContext

    Create a configuration from the sqlContext

    Uses the prefixed properties that are set in the Spark configuration to create the config.

    sqlContext

    the SQL context

    returns

    the configuration

    Definition Classes
    MongoCompanionConfig
    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) As of Spark 2.0 SQLContext was replaced by SparkSession. Use the SparkSession method instead

    See also

    configPrefix

Inherited from MongoCompanionConfig

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped