Package

org.tupol.spark

tools

Permalink

package tools

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. tools
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. abstract class FileStreamingSqlProcessor extends SparkApp[FileStreamingSqlProcessorContext, (StreamingQuery, DataFrame)]

    Permalink

    The StreamingSqlProcessor is a base class that can support multiple implementation, mainly designed to support registering custom SQL functions (UDFs) to make them available while running the queries.

  2. case class FileStreamingSqlProcessorContext(inputTables: Map[String, FileStreamDataSourceConfiguration], inputVariables: Map[String, String], outputConfig: FileStreamDataSinkConfiguration, sql: String) extends Product with Serializable

    Permalink
  3. case class FormatConverterContext(input: FormatAwareDataSourceConfiguration, output: FormatAwareDataSinkConfiguration) extends Product with Serializable

    Permalink

    Context class for the FormatConverter

    Context class for the FormatConverter

    input

    data source

    output

    data sink

  4. abstract class SqlProcessor extends SparkApp[SqlProcessorContext, DataFrame]

    Permalink

    The SqlProcessor is a base class that can support multiple implementation, mainly designed to support registering custom SQL functions (UDFs) to make them available while running the queries.

  5. case class SqlProcessorContext(inputTables: Map[String, FormatAwareDataSourceConfiguration], inputVariables: Map[String, String], outputConfig: FormatAwareDataSinkConfiguration, sql: String) extends Product with Serializable

    Permalink
  6. case class StreamingFormatConverterContext(input: FormatAwareStreamingSourceConfiguration, output: FormatAwareStreamingSinkConfiguration) extends Product with Serializable

    Permalink

    Context class for the StreamingFormatConverter

    Context class for the StreamingFormatConverter

    input

    data source

    output

    data sink

Value Members

  1. object FileStreamingSqlProcessorContext extends Configurator[FileStreamingSqlProcessorContext] with Logging with Serializable

    Permalink
  2. object FormatConverter extends SparkFun[FormatConverterContext, DataFrame]

    Permalink

    Load a file into a DataFrame and save it as a file in the specified path.

    Load a file into a DataFrame and save it as a file in the specified path.

  3. object FormatConverterContext extends Configurator[FormatConverterContext] with Serializable

    Permalink
  4. object SimpleFileStreamingSqlProcessor extends FileStreamingSqlProcessor

    Permalink

    This is the most basic implementation of the SQL processor and it registers no custom functions

  5. object SimpleSqlProcessor extends SqlProcessor

    Permalink

    This is the most basic implementation of the SQL processor and it registers no custom functions

  6. object SqlProcessorContext extends Configurator[SqlProcessorContext] with Logging with Serializable

    Permalink
  7. object StreamingFormatConverter extends SparkApp[StreamingFormatConverterContext, StreamingQuery]

    Permalink

    Load a file into a org.apache.spark.sql.DataFrame and save it as a file in the specified path.

    Load a file into a org.apache.spark.sql.DataFrame and save it as a file in the specified path.

  8. object StreamingFormatConverterContext extends Configurator[StreamingFormatConverterContext] with Serializable

    Permalink
  9. package info

    Permalink
  10. def replaceVariables(text: String, variables: Map[String, String]): String

    Permalink

    Replace variables marked by {{}} inside the given text from the map holding variable names as keys and the values to be replaced as values.

Inherited from AnyRef

Inherited from Any

Ungrouped