config

Type Members

1. class Conf extends Serializable with Configurable

   

   Conf is a class that holds data of configuration.

   Conf is a class that holds data of configuration. Internally, Conf use a ConcurrentHashMap for saving the configuration arguments.

   We can instantiate a Conf directly from a Scala Map or a Config from the typesafe config library.

   val conf1 = Conf(myMap)  // equivalent to Conf.fromMap(myMap)
   val conf2 = Conf(myConfig)  //equivalent to Conf.fromConfig(myConfig)

   We can also instantiate manually

   val conf3 = new Conf()
   conf3.set("k", "v")
   conf3.set(Map("k1" -> "v1", "k2" -> "v2"))
   conf3 += anotherConf

   Configuration arguments can be retrieved by calling the get() method. Conf provide also several implicit converters that handle the type conversion.

   For example:

   val double = conf.getAs[Double]("dKey")
   val intArray = conf.getAs[Array[Int]]("aiKey")

   Annotations

   @Evolving()

2. abstract class ConfigLoader extends Logging

   

   ConfigLoader loads configurations by using typesafe.com's config library.

   ConfigLoader loads configurations by using typesafe.com's config library.

   You should set the application environment, then ConfigLoad will read the configuration file [app_env].conf in your resources directory.

   Multiple solutions are possible to configure application environment:
   

   - Otherwise you can set the JVM property app.environment. For example, by adding -Dapp.environment=local in your command, ConfigLoader will read the conf file local.conf

   - You can also create app.environment in the default configuration file (example: application.conf, application.properties)

   • The last solution is to overwrite directly the value of confPath with the name of you configuration file. Then ConfigLoader will read directly the given file

   The priority of the above solution is:

   configPath >> setAppEnv >> JVM property >> `app.environment` in default config file

   If none of the above parameters are set, ConfigLoader will try to read its fallback configuration file

   Annotations

   @Evolving()

3. abstract class ConnectorConf extends Conf

   

4. class DeltaConnectorConf extends ConnectorConf

   

5. class DynamoDBConnectorConf extends ConnectorConf

   

6. class FileConnectorConf extends ConnectorConf

   

   Annotations

   @Evolving()

7. class JDBCConnectorConf extends ConnectorConf

   

8. class StructuredStreamingConnectorConf extends ConnectorConf

   

   Configuration parameters: Spark documentation

   Configuration parameters: Spark documentation