Class/Object

io.smartdatalake.workflow.dataobject

WebserviceFileDataObject

Related Docs: object WebserviceFileDataObject | package dataobject

Permalink

case class WebserviceFileDataObject(id: DataObjectId, url: String, additionalHeaders: Map[String, String] = Map(), timeouts: Option[HttpTimeoutConfig] = None, readTimeoutMs: Option[Int] = None, authMode: Option[AuthMode] = None, mimeType: Option[String] = None, writeMethod: WebserviceWriteMethod = WebserviceWriteMethod.Post, proxy: Option[HttpProxyConfig] = None, followRedirects: Boolean = false, partitionDefs: Seq[WebservicePartitionDefinition] = Seq(), partitionLayout: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry) extends FileRefDataObject with CanCreateInputStream with CanCreateOutputStream with SmartDataLakeLogger with Product with Serializable

DataObject to call webservice and return response as InputStream This is implemented as FileRefDataObject because the response is treated as some file content. FileRefDataObjects support partitioned data. For a WebserviceFileDataObject partitions are mapped as query parameters to create query string. All possible query parameter values must be given in configuration.

partitionDefs

list of partitions with list of possible values for every entry

partitionLayout

definition of partitions in query string. Use %<partitionColName>% as placeholder for partition column value in layout.

Linear Supertypes
Serializable, Serializable, Product, Equals, CanCreateOutputStream, CanCreateInputStream, FileRefDataObject, FileDataObject, CanHandlePartitions, DataObject, AtlasExportable, SmartDataLakeLogger, ParsableFromConfig[DataObject], SdlConfigObject, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WebserviceFileDataObject
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. CanCreateOutputStream
  7. CanCreateInputStream
  8. FileRefDataObject
  9. FileDataObject
  10. CanHandlePartitions
  11. DataObject
  12. AtlasExportable
  13. SmartDataLakeLogger
  14. ParsableFromConfig
  15. SdlConfigObject
  16. AnyRef
  17. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new WebserviceFileDataObject(id: DataObjectId, url: String, additionalHeaders: Map[String, String] = Map(), timeouts: Option[HttpTimeoutConfig] = None, readTimeoutMs: Option[Int] = None, authMode: Option[AuthMode] = None, mimeType: Option[String] = None, writeMethod: WebserviceWriteMethod = WebserviceWriteMethod.Post, proxy: Option[HttpProxyConfig] = None, followRedirects: Boolean = false, partitionDefs: Seq[WebservicePartitionDefinition] = Seq(), partitionLayout: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

    Permalink

    partitionDefs

    list of partitions with list of possible values for every entry

    partitionLayout

    definition of partitions in query string. Use %<partitionColName>% as placeholder for partition column value in layout.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val additionalHeaders: Map[String, String]

    Permalink
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def atlasName: String

    Permalink
    Definition Classes
    DataObjectAtlasExportable
  7. def atlasQualifiedName(prefix: String): String

    Permalink
    Definition Classes
    AtlasExportable
  8. val authMode: Option[AuthMode]

    Permalink
  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def createInputStream(query: String)(implicit session: SparkSession): InputStream

    Permalink

    Same as getResponse, but returns response as InputStream

    Same as getResponse, but returns response as InputStream

    query

    it should be possible to define the partition to read as query string, but this is not yet implemented

    Definition Classes
    WebserviceFileDataObject → CanCreateInputStream
  11. def createOutputStream(path: String, overwrite: Boolean)(implicit session: SparkSession): OutputStream

    Permalink

    path

    is ignored for webservices

    overwrite

    is ignored for webservices

    session

    implicit spark session

    returns

    outputstream that writes to WebService once it's closed

    Definition Classes
    WebserviceFileDataObject → CanCreateOutputStream
  12. def deleteAll(implicit session: SparkSession): Unit

    Permalink

    Delete all data.

    Delete all data. This is used to implement SaveMode.Overwrite.

    Definition Classes
    FileRefDataObject
  13. def deleteFileRefs(fileRefs: Seq[FileRef])(implicit session: SparkSession): Unit

    Permalink

    Delete given files.

    Delete given files. This is used to cleanup files after they are processed.

    Definition Classes
    FileRefDataObject
  14. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  15. def expectedPartitionsCondition: Option[String]

    Permalink

    Definition of partitions that are expected to exists.

    Definition of partitions that are expected to exists. This is used to validate that partitions being read exists and don't return no data. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false example: "elements['yourColName'] > 2017"

    returns

    true if partition is expected to exist.

    Definition Classes
    WebserviceFileDataObjectCanHandlePartitions
  16. def extractPartitionValuesFromPath(filePath: String)(implicit session: SparkSession): PartitionValues

    Permalink

    Extract partition values from a given file path

    Extract partition values from a given file path

    Attributes
    protected
    Definition Classes
    FileRefDataObject
  17. def factory: FromConfigFactory[DataObject]

    Permalink

    Returns the factory that can parse this type (that is, type CO).

    Returns the factory that can parse this type (that is, type CO).

    Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

    returns

    the factory (object) for this class.

    Definition Classes
    WebserviceFileDataObject → ParsableFromConfig
  18. val fileName: String

    Permalink

    Definition of fileName.

    Definition of fileName. Default is an asterix to match everything. This is concatenated with the partition layout to search for files.

    Definition Classes
    FileRefDataObject
  19. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  20. val followRedirects: Boolean

    Permalink
  21. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  22. def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Permalink

    Handle class cast exception when getting objects from instance registry

    Handle class cast exception when getting objects from instance registry

    Attributes
    protected
    Definition Classes
    DataObject
  23. def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    DataObject
  24. def getFileRefs(partitionValues: Seq[PartitionValues])(implicit session: SparkSession): Seq[FileRef]

    Permalink

    For WebserviceFileDataObject, every partition is mapped to one FileRef

    For WebserviceFileDataObject, every partition is mapped to one FileRef

    partitionValues

    List of partition values to be filtered. If empty all files in root path of DataObject will be listed.

    returns

    List of FileRefs

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
  25. def getPartitionString(partitionValues: PartitionValues)(implicit session: SparkSession): Option[String]

    Permalink

    get partition values formatted by partition layout

    get partition values formatted by partition layout

    Definition Classes
    FileRefDataObject
  26. def getPath(implicit session: SparkSession): String

    Permalink

    Method for subclasses to override the base path for this DataObject.

    Method for subclasses to override the base path for this DataObject. This is for instance needed if pathPrefix is defined in a connection.

    Definition Classes
    FileRefDataObject
  27. def getResponse(query: Option[String] = None): Array[Byte]

    Permalink

    Calls webservice and returns response

    Calls webservice and returns response

    query

    optional URL with replaced placeholders to call

    returns

    Response as Array[Byte]

  28. def getSearchPaths(partitionValues: Seq[PartitionValues])(implicit session: SparkSession): Seq[(PartitionValues, String)]

    Permalink

    prepare paths to be searched

    prepare paths to be searched

    Attributes
    protected
    Definition Classes
    FileRefDataObject
  29. def housekeepingMode: Option[HousekeepingMode]

    Permalink

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions.

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions. Default is None.

    Definition Classes
    DataObject
  30. val id: DataObjectId

    Permalink

    A unique identifier for this instance.

    A unique identifier for this instance.

    Definition Classes
    WebserviceFileDataObjectDataObject → SdlConfigObject
  31. implicit val instanceRegistry: InstanceRegistry

    Permalink
  32. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  33. def listPartitions(implicit session: SparkSession, context: ActionPipelineContext): Seq[PartitionValues]

    Permalink

    List partition values defined for this web service.

    List partition values defined for this web service. Note that this is a fixed list.

    Definition Classes
    WebserviceFileDataObjectCanHandlePartitions
  34. lazy val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
  35. val metadata: Option[DataObjectMetadata]

    Permalink

    Additional metadata for the DataObject

    Additional metadata for the DataObject

    Definition Classes
    WebserviceFileDataObjectDataObject
  36. val mimeType: Option[String]

    Permalink
  37. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  38. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  39. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  40. val partitionDefs: Seq[WebservicePartitionDefinition]

    Permalink

    list of partitions with list of possible values for every entry

  41. val partitionLayout: Option[String]

    Permalink

    definition of partitions in query string.

    definition of partitions in query string. Use %<partitionColName>% as placeholder for partition column value in layout.

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
  42. def partitions: Seq[String]

    Permalink

    Definition of partition columns

    Definition of partition columns

    Definition Classes
    WebserviceFileDataObjectCanHandlePartitions
  43. def path: String

    Permalink

    No root path needed for Webservice.

    No root path needed for Webservice. It can be included in webserviceOptions.url.

    Definition Classes
    WebserviceFileDataObject → FileDataObject
  44. def postResponse(body: Array[Byte], query: Option[String] = None): Array[Byte]

    Permalink

    Calls webservice POST method with binary data as body

    Calls webservice POST method with binary data as body

    body

    post body as Byte Array, type will be determined by Tika

    query

    optional URL with replaced placeholders to call

    returns

    Response as Array[Byte]

  45. def postWrite(partitionValues: Seq[PartitionValues])(implicit session: SparkSession, context: ActionPipelineContext): Unit

    Permalink

    Runs operations after writing to DataObject

    Runs operations after writing to DataObject

    Definition Classes
    WebserviceFileDataObjectDataObject
  46. def prepare(implicit session: SparkSession, context: ActionPipelineContext): Unit

    Permalink

    Prepare & test DataObject's prerequisits

    Prepare & test DataObject's prerequisits

    This runs during the "prepare" operation of the DAG.

    Definition Classes
    WebserviceFileDataObject → FileDataObject → DataObject
  47. val proxy: Option[HttpProxyConfig]

    Permalink
  48. val readTimeoutMs: Option[Int]

    Permalink
  49. def relativizePath(filePath: String)(implicit session: SparkSession): String

    Permalink

    Make a given path relative to this DataObjects base path

    Make a given path relative to this DataObjects base path

    Definition Classes
    WebserviceFileDataObject → FileDataObject
  50. val saveMode: SDLSaveMode

    Permalink

    Overwrite or Append new data.

    Overwrite or Append new data. When writing partitioned data, this applies only to partitions concerned.

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
  51. val separator: Char

    Permalink

    default separator for paths

    default separator for paths

    Attributes
    protected
    Definition Classes
    FileDataObject
  52. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  53. val tika: Tika

    Permalink
  54. val timeouts: Option[HttpTimeoutConfig]

    Permalink
  55. def toStringShort: String

    Permalink
    Definition Classes
    DataObject
  56. def translateFileRefs(fileRefs: Seq[FileRef])(implicit session: SparkSession, context: ActionPipelineContext): Seq[FileRef]

    Permalink

    Given some FileRefs for another DataObject, translate the paths to the root path of this DataObject

    Given some FileRefs for another DataObject, translate the paths to the root path of this DataObject

    Definition Classes
    FileRefDataObject
  57. val url: String

    Permalink
  58. def validateSchemaHasPartitionCols(df: DataFrame, role: String): Unit

    Permalink

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  59. def validateSchemaHasPrimaryKeyCols(df: DataFrame, primaryKeyCols: Seq[String], role: String): Unit

    Permalink

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  60. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  61. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  62. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  63. val writeMethod: WebserviceWriteMethod

    Permalink

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from CanCreateOutputStream

Inherited from CanCreateInputStream

Inherited from FileRefDataObject

Inherited from FileDataObject

Inherited from CanHandlePartitions

Inherited from DataObject

Inherited from AtlasExportable

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[DataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped