com.coxautodata.waimak.configuration
CaseClassConfigParser
Related Doc:
package configuration
URI of the properties file used by the PropertiesFilePropertyProviderBuilder object.
URI of the properties file used by the PropertiesFilePropertyProviderBuilder object. Used when CONFIG_PROPERTY_PROVIDER_BUILDER_MODULES includes PropertiesFilePropertyProviderBuilder. File will be opened using an Hadoop FileSystem object, therefore URI must be supported by your Hadoop libraries and configuration must be present in the HadoopConfiguration on the SparkSession.
Comma separated list of property provider builder object names to instantiate.
Comma separated list of property provider builder object names to instantiate. Set this to have the config parser use the custom objects to search for configuration parameter values in these property providers.
Number of retries used when requesting parameter values from property providers
Timeout in milliseconds used when requesting parameter values from property providers
Prefix to add to parameters when looking in the Spark conf.
Prefix to add to parameters when looking in the Spark conf. For example, if looking for parameter
args.arg1 then the parser will look for spark.args.arg1 in the Spark conf by default.
This can be disabled by setting this property to an empty string.
Populate a Case Class from an instance of SparkConf.
Populate a Case Class from an instance of SparkConf. It will attempt to cast the configuration values to the correct types, and most primitive, Option[primitive], and List/Seq/Vector[primitive] types are supported. Default arguments in the case class will also be respected. Option types will not be set to None if they are not specified in the SparkConf unless None is the default value in the case class. The separator for collection types (List,Seq,Vector) is by default "," but can be changed with '@separator("..")' annotation on the parameter in the case class. The parameters keys that are looked up will be of the form: {prefix}{parameter}, e.g. for case class Ex(key: String) and prefix="example.prefix.", then the key will have the form "example.prefix.key" By default, properties in the SparkConf will be looked up with an additional prefix (see SPARK_CONF_PROPERTY_PREFIX). The order in which properties are looked up are: 1) Spark Conf, 2) Additional conf map, 3) Property Providers (in order they were specified if multiple were given)
Case class type to construct
Instance of SparkFlowContext containing a spark session with configuration
Prefix to assign to a Key when looking in SparkConf
An additional set of properties to search. Preference is given to SparkConf values if the key exists in both this additionalConf and SparkConf
An instantiated case class populated from the SparkConf instance and default arguments
Takes a value of type A and a msg to log, returning a and logging the message at the desired level
Takes a value of type A and a msg to log, returning a and logging the message at the desired level
a
Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
a
logAndReturn(1, (num: Int) => s"number: $num", Info) // In the log we would see a log corresponding to "number 1"