Packages

trait SparkFunctions extends AnyRef

Library of all spark functions which implements different abinitio functions used in abinitio workflows.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkFunctions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class RecordIterator extends Serializable
  2. class StringAsStream extends Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def YJJJ_to_YYYYJJJ(in_date: Column, ref_date: Column): Column

    Converts 1 digit julian year to 4 digits julian year.

    Converts 1 digit julian year to 4 digits julian year.

    in_date

    date in Julian in "YJJJ" format

    ref_date

    date in "yyyyMMdd" format

    returns

    a date in "YYYYJJJ"

    Annotations
    @Py4JWhitelist()
  5. lazy val adjustCenturyDateInCyyFormat: UserDefinedFunction

    begining of input should have Cyy

  6. def adjustStringRegexPattern(input: String): String
  7. def alteryxFlattenSchema(dataFrame: DataFrame, jsonParsedColumnName: String, sparkSession: SparkSession): DataFrame
    Annotations
    @Py4JWhitelist()
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. lazy val bigDecimalToPackedBytes: UserDefinedFunction
  10. lazy val bv_all_zeros: UserDefinedFunction
  11. lazy val bv_and: UserDefinedFunction
  12. lazy val bv_count_one_bits: UserDefinedFunction
  13. lazy val bv_difference: UserDefinedFunction
  14. lazy val bv_from_index_vector: UserDefinedFunction
  15. lazy val bv_indices: UserDefinedFunction
  16. lazy val bv_or: UserDefinedFunction
  17. lazy val bv_vector_or: UserDefinedFunction
  18. lazy val canonical_representation: UserDefinedFunction
  19. lazy val char_string: UserDefinedFunction
  20. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  21. def computeChecksum(df: DataFrame): DataFrame
    Annotations
    @Py4JWhitelist()
  22. def convertInputBytesToStructType(input: Any, typeInfo: Seq[String], startByte: Int = 0): Row

    Method used for abinitio's reinterpret_as function to read necessary bytes from byteArray for input data and convert into struct format as per provided in typeInfo sequence.

    Method used for abinitio's reinterpret_as function to read necessary bytes from byteArray for input data and convert into struct format as per provided in typeInfo sequence.

    TypeInfo can have multiple entries, each could be either decimal or string type. Depending on the argument passed within decimal or string bytes are read from input byte array.

    If decimal or string argument has some integer then that many bytes are read from input byte array or if decimal or string has some string delimiter as its argument then from the current position bytes are read until string delimiter is found in input byte array.

  23. def createDataFrameFromData(inputData: String, delimiter: String, columnName: String, columnType: String, sparkSession: SparkSession): DataFrame

    Method to read values from inputData and create dataframe with column name as columnName and column type as columnType for the values in inputData delimiter by delimiter.

    Method to read values from inputData and create dataframe with column name as columnName and column type as columnType for the values in inputData delimiter by delimiter.

    Annotations
    @Py4JWhitelist()
  24. lazy val cross_join_index_range: UserDefinedFunction
  25. def date_add_months(inputDate: Column, months: Int): Column

    Returns the internal representation of a date resulting from adding (or subtracting) a number of months to the specified date.

    Returns the internal representation of a date resulting from adding (or subtracting) a number of months to the specified date.

    inputDate

    in yyyy-MM-dd format

    Annotations
    @Py4JWhitelist()
  26. def date_difference_days(laterDate: Column, earlierDate: Column): Column

    Computes number of days between two specified dates in "yyyyMMdd" format

    Computes number of days between two specified dates in "yyyyMMdd" format

    laterDate

    input date

    earlierDate

    input date

    returns

    number of days between laterDate and earlierDate or null if either one is null

    Annotations
    @Py4JWhitelist()
  27. lazy val date_month_end: UserDefinedFunction
  28. def date_to_int(input: Column): Column
    Annotations
    @Py4JWhitelist()
  29. lazy val datetime_add: UserDefinedFunction
  30. def datetime_add_months(input: Column, months: Int): Column

    Returns the internal representation of a timestamp resulting from adding (or subtracting) a number of months to the specified timestamp.

    Returns the internal representation of a timestamp resulting from adding (or subtracting) a number of months to the specified timestamp.

    input

    timestamp in yyyy-MM-dd HH:mm:ss.SSSS format

    Annotations
    @Py4JWhitelist()
  31. lazy val datetime_difference: UserDefinedFunction
  32. def datetime_difference_hours(end: Column, start: Column): Column

    Returns the number of hours between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Returns the number of hours between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Annotations
    @Py4JWhitelist()
  33. def datetime_difference_minutes(end: Column, start: Column): Column

    Returns the number of minutes between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Returns the number of minutes between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Annotations
    @Py4JWhitelist()
  34. def datetime_difference_seconds(end: Column, start: Column): Column

    Returns the number of minutes between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Returns the number of minutes between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.

    Annotations
    @Py4JWhitelist()
  35. def datetime_from_unixtime(seconds: Column): Column
    Annotations
    @Py4JWhitelist()
  36. def decimal_lpad(input: Column, len: Int, char_to_pad_with: String = "0", decimal_point_char: String = "."): Column

    Method uses a java regex to identify decimal numbers from input string.

    Method uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by regex. 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [0-9]+(\$$decimal_point_char)[0-9]+ and (0\$$decimal_point_char)[0-9]+ regex

    After extracting decimal number this code checks if length of decimal number is more than len parameter or not. If length is more than len parameter then it simply returns this extracted decimal number. Otherwise it first left pad decimal number with char_to_pad_with to make its length equal to len parameter and then adjusts minus sign (-) to left most part of decimal number.

    input

    input string.

    len

    length of characters.

    char_to_pad_with

    character to left pad with. default value is "0"

    decimal_point_char

    A string that specifies the character that represents the decimal point.

    returns

    a decimal string of the specified length or longer, left-padded with a specified character as needed and trimmed of leading zeros.

    Annotations
    @Py4JWhitelist()
  37. def decimal_lrepad(input: Column, len: Int, char_to_pad_with: String = "0", decimal_point_char: String = "."): Column

    Method uses a java regex to identify decimal numbers from input string.

    Method uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by combination of [1-9][0-9]*[0-9] and [1-9]+ regex 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [1-9][0-9]*(\\\$$decimal_point_char)[0-9]+ and (0\\\$$decimal_point_char)[0-9]*[0-9] regex

    After extracting decimal number this code checks if length of decimal number is more than len parameter or not. If length is more than len parameter then it simply returns this extracted decimal number. Otherwise it first left pad decimal number with char_to_pad_with to make its length equal to len parameter and then adjusts minus sign (-) to left most part of decimal number.

    input

    input string.

    len

    length of characters.

    char_to_pad_with

    character to left pad with. default value is "0"

    decimal_point_char

    A string that specifies the character that represents the decimal point.

    returns

    a decimal string of the specified length or longer, left-padded with a specified character as needed and trimmed of leading zeros.

    Annotations
    @Py4JWhitelist()
  38. def decimal_round(input: Column, places: Int): Column
    Annotations
    @Py4JWhitelist()
  39. def decimal_round_down(input: Column, right_digits: Int): Column

    Function returns a value which is rounded down to right_digits number of digits to the right of decimal point.

    Function returns a value which is rounded down to right_digits number of digits to the right of decimal point.

    Annotations
    @Py4JWhitelist()
  40. def decimal_round_even(input: Column, places: Int): Column
    Annotations
    @Py4JWhitelist()
  41. def decimal_round_up(input: Column, places: Int): Column

    Returns a number rounded up to a specified number of places to the right of the decimal point.

    Returns a number rounded up to a specified number of places to the right of the decimal point.

    Annotations
    @Py4JWhitelist()
  42. def decimal_strip(input: Column, decimal_point_char: String = "."): Column

    Function uses a java regex to identify decimal numbers from input string.

    Function uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by combination of [1-9][0-9 ]*[0-9] and [1-9]+ regex 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [1-9][0-9]*(\$$decimal_point_char)[0-9 ]+ and (0\$$decimal_point_char)[0-9 ]*[0-9] regex

    After extracting decimal number this code looks for minus sign before extracted number in input and appends it with decimal number if found minus sign.

    In the end it replaces all whitespaces with empty string in the final resultant decimal number.

    input

    input string

    decimal_point_char

    A string that specifies the character that represents the decimal point.

    returns

    a decimal from a string that has been trimmed of leading zeros and non-numeric characters.

    Annotations
    @Py4JWhitelist()
  43. def decimal_truncate(input: Column, number_of_places: Column): Column
    Annotations
    @Py4JWhitelist()
  44. lazy val decodeBytes: UserDefinedFunction
  45. lazy val decodeString: UserDefinedFunction
  46. lazy val decode_datetime: UserDefinedFunction

    UDF to get record of type decode_datetime_type.

    UDF to get record of type decode_datetime_type. This record will have all its fields populated with corresponding entries in input date/timestamp.

    Returned record will have following schema.

    integer(8) year; integer(8) month; integer(8) day; integer(8) hour; integer(8) minute; integer(8) second; integer(8) microsecond;

    Note: Supported Input time is in yyyy-MM-dd HH:mm:ss.SSSSSS or yyyy-MM-dd HH:mm:ss or yyyy-MM-dd formats only. Additional handling is done to support timestamp retrieved from now() function call.

  47. lazy val decode_datetime_as_local: UserDefinedFunction
  48. lazy val directory_listing: UserDefinedFunction
  49. lazy val directory_listing_dir_only: UserDefinedFunction
  50. def directory_listing_scala(path: String, filePrefix: String): Column
    Annotations
    @Py4JWhitelist()
  51. lazy val encodeBytes: UserDefinedFunction
  52. lazy val encodeString: UserDefinedFunction
  53. lazy val encode_date: UserDefinedFunction

    integer values specifying days relative to January 1, 1900.

    integer values specifying days relative to January 1, 1900. This function returns the internal representation of a date given the year, month, and date. encode_date returns the internal representation of the date specified by the year 1998, the month 5, and the day 18:encode_date(1998, 5, 18) = 35931

  54. def ends_with(input: Column, suffix: String): Column

    Returns true if string columns ends with given suffix

    Returns true if string columns ends with given suffix

    Annotations
    @Py4JWhitelist()
  55. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  56. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  57. lazy val eval: UserDefinedFunction

    Method to return the result of evaluating a string expression in the context of a specified input column.

    Method to return the result of evaluating a string expression in the context of a specified input column. Here input column could be struct type record, simple column, array type etc. Here expr could be reference to nested column inside input column or any expression which requires values from input column for its evaulation.

    Note: Current implementation only supports scenerio where input column is of struct type and expr is simply dot separated column reference to input struct.

  58. lazy val file_information: UserDefinedFunction

    UDF to get file information for passed input file path.

  59. def findFirstElement(input: Column, default: Column = lit(null)): Column
    Annotations
    @Py4JWhitelist()
  60. def findFirstNonBlankElement(input: Column, default: Column): Column
    Annotations
    @Py4JWhitelist()
  61. def findLastElement(input: Column, default: Column = lit(null)): Column
    Annotations
    @Py4JWhitelist()
  62. def first_defined(expr1: Column, expr2: Column): Column

    Method to identify and return first non null expression.

    Method to identify and return first non null expression.

    Annotations
    @Py4JWhitelist()
  63. lazy val first_defined_for_double_Udf: UserDefinedFunction
  64. def flattenStructSchema(schema: StructType, prefix: String = null): Array[Column]
    Annotations
    @Py4JWhitelist()
  65. lazy val force_error: UserDefinedFunction
  66. def format_decimal(input: Column, scale: Int): Column
    Annotations
    @Py4JWhitelist()
  67. def from_sv(input: Column, separator: String, schema: StructType): Column
    Annotations
    @Py4JWhitelist()
  68. def from_xml(content: Column, schema: StructType): Column
    Annotations
    @Py4JWhitelist()
  69. def from_xml(content: Column, schemaJSON: String): Column
    Annotations
    @Py4JWhitelist()
  70. def generateDataFrameWithSequenceColumn(start: Int, end: Int, columnName: String, sparkSession: SparkSession): DataFrame

    Method to create dataframe with single column containing increasing sequence id from start to end.

    Method to create dataframe with single column containing increasing sequence id from start to end.

    Annotations
    @Py4JWhitelist()
  71. def generate_sequence(start: Int, end: Int, step: Int = 1): Column

    Function to create sequence of array between two passed numbers

    Function to create sequence of array between two passed numbers

    start

    starting point of generated sequence

    end

    terminating point of generated sequence.

    returns

    column containing sequence of integers.

    Annotations
    @Py4JWhitelist()
  72. lazy val generate_sequence: UserDefinedFunction

    UDF to generate column with sequence of integers between two passed start and end columns.

  73. def getAlias(column: Column): String
  74. lazy val getByteFromByteArray: UserDefinedFunction

    UDF to get last Byte from ByteArray of input data.

  75. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  76. def getColumnInSecondArrayByFirstNonBlankPositionInFirstArray(nonBlankEntryExpr: Column, firstArray: Column, secondArray: Column): Column
    Annotations
    @Py4JWhitelist()
  77. def getContentAsStream(content: String): StringAsStream
  78. lazy val getDefaultedElseTrimmed: UserDefinedFunction
  79. def getFebruaryDay(year: Column): Column

    Computes number of days in February month in a given year

    Computes number of days in February month in a given year

    year

    year whose number of days in February needs to be calculated

    returns

    number of days

    Annotations
    @Py4JWhitelist()
  80. def getFieldFromStructByPosition(column: Column, position: Int): Column

    Method to get field at specific position from struct column

    Method to get field at specific position from struct column

    Annotations
    @Py4JWhitelist()
  81. lazy val getIntArrayFromByteArray: UserDefinedFunction

    UDF to get long comprising of last 8 Bytes from ByteArray of input data.

  82. lazy val getIntFromByteArray: UserDefinedFunction

    UDF to get integer comprising of last 4 Bytes from ByteArray of input data.

  83. lazy val getLongArrayFromByteArray: UserDefinedFunction

    UDF to get long comprising of last 8 Bytes from ByteArray of input data.

  84. lazy val getLongFromByteArray: UserDefinedFunction

    UDF to get long comprising of last 8 Bytes from ByteArray of input data.

  85. def getMTimeDataframe(filepath: String, format: String, spark: SparkSession): DataFrame
    Annotations
    @Py4JWhitelist()
  86. lazy val getShortFromByteArray: UserDefinedFunction

    UDF to get short comprising of last 2 Bytes from ByteArray of input data.

  87. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  88. lazy val hash_MD5: UserDefinedFunction
  89. lazy val hash_SHA1: UserDefinedFunction
  90. def hash_SHA512(input: Column): Column
    Annotations
    @Py4JWhitelist()
  91. def hash_value(input: Column, keys: Seq[String], hashAlgorithm: String): Column
    Annotations
    @Py4JWhitelist()
  92. lazy val instr_extended_udf: UserDefinedFunction
  93. lazy val instr_udf: UserDefinedFunction
  94. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  95. lazy val isNotEqualToValue: UserDefinedFunction
  96. lazy val isNotNullAndNotBlank: UserDefinedFunction
  97. lazy val isNullOrBlank: UserDefinedFunction
  98. def isNullOrEmpty(input: Column): Column

    Method to check if current column is null or has empty value.

    Method to check if current column is null or has empty value.

    Annotations
    @Py4JWhitelist()
  99. def is_ascii(input: Column): Column

    Checks if a string is ascii

    Checks if a string is ascii

    input

    column to be checked

    returns

    true if the input string is ascii otherwise false

    Annotations
    @Py4JWhitelist()
  100. def is_blank(input: Column): Column

    Method to identify if input string is a blank string or not.

    Method to identify if input string is a blank string or not.

    input

    input string.

    returns

    return 1 if given string contains all blank character or is a zero length string, otherwise it returns 0

    Annotations
    @Py4JWhitelist()
  101. lazy val is_blank_udf: UserDefinedFunction
  102. lazy val is_bzero: UserDefinedFunction

    Tests whether an object is composed of all binary zero bytes.

    Tests whether an object is composed of all binary zero bytes. This function returns: 1. 1 if obj contains only binary zero bytes or is a zero-length string 2. 0 if obj contains any non-zero bytes 3. NULL if obj is NULL

  103. def is_not_blank(input: Column): Column
    Annotations
    @Py4JWhitelist()
  104. def is_not_null(input: Column): Column
    Annotations
    @Py4JWhitelist()
  105. def is_numeric_ascii(input: Column): Column

    Checks if an input string contains only ascii code and numbers

    Checks if an input string contains only ascii code and numbers

    input

    string to be checked

    returns

    true if input string contains only ascii code and numbers or null if input is null

    Annotations
    @Py4JWhitelist()
  106. def is_valid(input: Column, isNullable: Boolean, formatInfo: Option[Any], len: Option[Seq[Int]]): Column

    Method to identify if passed input column is a valid expression after typecasting to passed dataType.

    Method to identify if passed input column is a valid expression after typecasting to passed dataType. Also while typecasting if len is present then this function also makes sure the max length of input column after typecasting operation is not greater than len.

    input

    input column expression to be identified if is valid.

    formatInfo

    datatype to which input column expression must be typecasted. If datatype is a string then it is treated as timestamp format. If it is a list of string then it is treated as having current timestamp format and and new timestamp format to which input column needs to be typecasted.

    len

    max length of input column after typecasting it to dataType.

    returns

    0 if input column is not valid after typecasting or 1 if it is valid.

    Annotations
    @Py4JWhitelist()
  107. def is_valid(input: Column, isNullable: Boolean, formatInfo: Option[Any]): Column
    Annotations
    @Py4JWhitelist()
  108. def is_valid(input: Column, formatInfo: Option[Any], len: Option[Seq[Int]]): Column
    Annotations
    @Py4JWhitelist()
  109. def is_valid(input: Column, formatInfo: Option[Any]): Column
    Annotations
    @Py4JWhitelist()
  110. def is_valid(input: Column, isNullable: Boolean): Column
    Annotations
    @Py4JWhitelist()
  111. def is_valid(input: Column): Column
    Annotations
    @Py4JWhitelist()
  112. def is_valid_date(dateFormat: String, inDate: Column): Column

    Validates date against a input format

    Validates date against a input format

    dateFormat

    A pattern such as yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.SSSS or dd.MM.yyyy

    inDate

    Input date to be validated

    returns

    true if the input date is valid otherwise false

    Annotations
    @Py4JWhitelist()
  113. def is_valid_python_bridge(input: Column, isNullable: Boolean, formatSerialized: Option[Any], len: Option[Seq[Int]]): Column
    Annotations
    @Py4JWhitelist()
  114. def lastElementInCurrentWindow(input: Column): Column
    Annotations
    @Py4JWhitelist()
  115. lazy val make_byte_flags: UserDefinedFunction

    UDF to return a flag for each character if it is present or not in input String.

  116. def make_constant_vector(size: Int, seedVal: Int): Array[Int]

    Method to create array of size "size" containing seedVal as each entry

    Method to create array of size "size" containing seedVal as each entry

    Annotations
    @Py4JWhitelist()
  117. def make_constant_vector(size: Int, seedVal: Column): Column

    Method to create array of size "size" containing seedVal as each entry

    Method to create array of size "size" containing seedVal as each entry

    Annotations
    @Py4JWhitelist()
  118. def math_max(input: Column*): Column
    Annotations
    @Py4JWhitelist()
  119. def math_min(input: Column*): Column
    Annotations
    @Py4JWhitelist()
  120. lazy val multi_regex_match: UserDefinedFunction
  121. def multi_regex_replace_with_char_conversion(input: Column, charSet: Column, replaceStr: Column, replacement0: String, replacement1: String, pattern: String*): Column
    Annotations
    @Py4JWhitelist()
  122. lazy val multifile_information: UserDefinedFunction

    UDF to get multifile information for passed input file path.

  123. lazy val murmur: UserDefinedFunction

    UDF for murmur hash generation for any column type

  124. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  125. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  126. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  127. def now(): Column

    Method to get current timestamp.

    Method to get current timestamp.

    returns

    current timestamp in YYYYMMddHHmmssSSSSSS format.

    Annotations
    @Py4JWhitelist()
  128. def numberOfPartitions(in: DataFrame): Column
    Annotations
    @Py4JWhitelist()
  129. lazy val number_grouping: UserDefinedFunction

    udf to group input decimal into multiple groups separated by separator

  130. lazy val packedBytesStringToDecimal: UserDefinedFunction
  131. lazy val packedBytesToDecimal: UserDefinedFunction
  132. lazy val raw_data_concat: UserDefinedFunction
  133. lazy val raw_data_substring: UserDefinedFunction
  134. lazy val re_get_match: UserDefinedFunction

    Returns the first string in a target string that matches a regular expression.

  135. lazy val re_get_match_with_index: UserDefinedFunction
  136. lazy val re_get_matches: UserDefinedFunction
  137. lazy val re_get_matches_with_offset: UserDefinedFunction
  138. lazy val re_index: UserDefinedFunction

    UDF wrapper over re_index function.

  139. lazy val re_index_with_offset: UserDefinedFunction

    Returns the first string in a target string that matches a regular expression.

  140. lazy val re_match_replace_all: UserDefinedFunction
  141. def re_replace(target: Column, pattern: String, replacement: String, offset: Int = 0): Column

    Replaces all substrings in a target string that match a specified regular expression.

    Replaces all substrings in a target string that match a specified regular expression.

    target

    A string that the function searches for a substring that matches pattern_expr.

    pattern

    regular expression

    replacement

    replacement string

    offset

    Number of characters, from the beginning of str, to skip before searching.

    returns

    a replaced string in which all substrings, which matches a specified regular expression, are replaced.

    Annotations
    @Py4JWhitelist()
  142. def re_replace_first(target: Column, pattern: String, replacement: String, offset: Column = lit(0)): Column

    Replaces only the first regex matching occurrence in the target string.

    Replaces only the first regex matching occurrence in the target string.

    target

    A string that the function searches for a substring that matches pattern_expr.

    pattern

    regular expression

    replacement

    replacement string

    returns

    a replaced string in which first substring, which matches a specified regular expression, is replaced.

    Annotations
    @Py4JWhitelist()
  143. lazy val re_split_no_empty: UserDefinedFunction

    UDF to split input string via pattern string and remove all empty subtrings.

  144. lazy val readBytesIntoInteger: UserDefinedFunction
  145. lazy val readBytesIntoLong: UserDefinedFunction
  146. lazy val readBytesStringIntoInteger: UserDefinedFunction
  147. lazy val readBytesStringIntoLong: UserDefinedFunction
  148. lazy val read_file: UserDefinedFunction
  149. lazy val record_info: UserDefinedFunction
  150. lazy val record_info_with_includes: UserDefinedFunction
  151. def registerAllUDFs(spark: SparkSession): Unit
    Annotations
    @Py4JWhitelist()
  152. def remove_non_digit(input: Column): Column

    Method removes any non-digit characters from the specified string column.

    Method removes any non-digit characters from the specified string column.

    input

    input String Column

    returns

    Cleaned string column or null

    Annotations
    @Py4JWhitelist()
  153. def replaceBlankColumnWithNull(input: Column): Column

    Method to replace String Columns with Empty value to Null.

    Method to replace String Columns with Empty value to Null.

    Annotations
    @Py4JWhitelist()
  154. def replaceNullWithDefaultValues(input: Column, schema: String): Column
    Annotations
    @Py4JWhitelist()
  155. def replace_null_with_blank(input: Column): Column
    Annotations
    @Py4JWhitelist()
  156. def scanf_double(format: Column, value: Column): Column
    Annotations
    @Py4JWhitelist()
  157. def scanf_long(format: Column, value: Column): Column
    Annotations
    @Py4JWhitelist()
  158. def schemaRowCompareResult(row1: StructType, row2: StructType): Column
    Annotations
    @Py4JWhitelist()
  159. def sign_explicit(c: Column): Column

    Adds an explicit sign to the number.

    Adds an explicit sign to the number. E.g. 2 -> +2; -004 -> -004; 0 -> +0

    Annotations
    @Py4JWhitelist()
  160. lazy val sign_explicit_Udf: UserDefinedFunction
  161. def sign_reserved(c: Column): Column
    Annotations
    @Py4JWhitelist()
  162. lazy val sign_reserved_Udf: UserDefinedFunction
  163. lazy val splitIntoMultipleColumnsUdf: UserDefinedFunction

    UDF to break input string into multiple string via delimiter.

    UDF to break input string into multiple string via delimiter. Number of strings after split are adjusted as per passed width parameter. If number of strings are less then empty strings are added otherwise in case of more number of strings, first width number of entries are picked and remaining are discarded.

  164. lazy val splitIntoNormalizedJSON: UserDefinedFunction
  165. def starts_with(input: Column, prefix: String): Column

    Returns true if string columns starts with given prefix

    Returns true if string columns starts with given prefix

    Annotations
    @Py4JWhitelist()
  166. def string_char(inputStr: Column, index: Int): Column

    Method to return character code of character at index position in inputStr string.

    Method to return character code of character at index position in inputStr string.

    inputStr

    input string

    index

    location of character to get code.

    returns

    integer column.

    Annotations
    @Py4JWhitelist()
  167. lazy val string_cleanse: UserDefinedFunction

    This implementation is incorrect.

  168. def string_compare(input1: Column, input2: Column): Column
    Annotations
    @Py4JWhitelist()
  169. lazy val string_concat_in_loop: UserDefinedFunction
  170. lazy val string_convert_explicit: UserDefinedFunction

    Converts a string from one character set to another, replacing inconvertible characters with a specified string.

  171. lazy val string_filter: UserDefinedFunction

    Method which returns string of characters present in both of the strings in the same order as appearing in first string

  172. lazy val string_filter_out: UserDefinedFunction

    Compares two input strings, then returns characters that appear in one string but not in the other.

  173. lazy val string_from_hex: UserDefinedFunction
  174. lazy val string_index: UserDefinedFunction

    UDF to find index of seekStr in inputStr.

    UDF to find index of seekStr in inputStr. Returned index will be 1 based index.

  175. lazy val string_index_with_offset: UserDefinedFunction

    UDF to find index of seekStr in inputStr from offset index onwards.

    UDF to find index of seekStr in inputStr from offset index onwards. Returned string position is 1 based position.

  176. def string_is_alphabetic(input: Column): Column

    Method which returns true if input string contains all alphabetic characters, or false otherwise.

    Method which returns true if input string contains all alphabetic characters, or false otherwise.

    Annotations
    @Py4JWhitelist()
  177. def string_is_numeric(input: Column): Column

    Method which returns true if input string contains all numeric characters, or false otherwise.

    Method which returns true if input string contains all numeric characters, or false otherwise.

    Annotations
    @Py4JWhitelist()
  178. def string_join(column: Column, delimiter: String): Column

    Concatenates the elements of column using the delimiter.

    Concatenates the elements of column using the delimiter.

    Annotations
    @Py4JWhitelist()
  179. def string_length(input: Column): Column
    Annotations
    @Py4JWhitelist()
  180. lazy val string_like: UserDefinedFunction

    Method to test whether a string matches a specified pattern.

    Method to test whether a string matches a specified pattern. This function returns 1 if the input string matches a specified pattern, and 0 if the string does not match the pattern.

    In abinitio version % character in pattern means to match zero or more characters and _ character means matches a single character.

  181. def string_lpad(input: Column, len: Int, pad_char: String = " "): Column

    Left-pad the input string column with pad_char to a length of len.

    Left-pad the input string column with pad_char to a length of len. If length of input column is more than len then returns input column unmodified.

    Annotations
    @Py4JWhitelist()
  182. def string_lrepad(input: Column, len: Int, char_to_pad_with: String = " "): Column

    function trims the string and then pad the string with given character upto given length.

    function trims the string and then pad the string with given character upto given length. if the length of trimmed string is equal to or greater than given length than it return input string

    input

    input string

    len

    length in number of characters.

    char_to_pad_with

    A character used to pad input string to length len.

    returns

    string of a specified length, trimmed of leading and trailing blanks and left-padded with a given character.

    Annotations
    @Py4JWhitelist()
  183. def string_pad(input: Column, len: Int, char_to_pad_with: String = " "): Column

    function pads input on the right with the character char_to_pad_with to make the string length len.

    function pads input on the right with the character char_to_pad_with to make the string length len. If str is already len or more characters long, the function returns input unmodified.

    Annotations
    @Py4JWhitelist()
  184. lazy val string_pad: UserDefinedFunction
  185. lazy val string_pad_with_char: UserDefinedFunction
  186. def string_prefix(input: Column, length: Column): Column
    Annotations
    @Py4JWhitelist()
  187. def string_repad(input: Column, len: Int, char_to_pad_with: String = " "): Column

    function trims the string and then pad the string on right side with given character upto given length.

    function trims the string and then pad the string on right side with given character upto given length. if the length of trimmed string is equal to or greater than given length than it return input string

    input

    input string

    len

    length in number of characters.

    char_to_pad_with

    A character used to pad input string to length len.

    returns

    string of a specified length, trimmed of leading and trailing blanks and left-padded with a given character.

    Annotations
    @Py4JWhitelist()
  188. def string_replace(input: Column, seekStr: Column, newStr: Column, offset: Column = lit(0)): Column

    Function to replace occurrence of seekStr with newStr string in input string after offset characters from first character.

    Function to replace occurrence of seekStr with newStr string in input string after offset characters from first character.

    input

    input string on which to perform replace operation.

    seekStr

    string to be replaced in input string.

    newStr

    string to be used instead of seekStr in input string.

    offset

    number of characters to skip from begining in input string before performing string_replace operation.

    returns

    modified string where seekStr is replaced with newStr in input string.

    Annotations
    @Py4JWhitelist()
  189. lazy val string_replace_first: UserDefinedFunction
  190. lazy val string_replace_first_in_loop: UserDefinedFunction
  191. lazy val string_replace_in_loop: UserDefinedFunction
  192. lazy val string_representation: UserDefinedFunction
  193. lazy val string_rindex: UserDefinedFunction

    Returns the index of the first character of the last occurrence of a seek string within another input string.

    Returns the index of the first character of the last occurrence of a seek string within another input string. Returned index is 1 based.

  194. lazy val string_rindex_with_offset: UserDefinedFunction

    UDF to find index of seekStr in inputStr from end of inputStr skipping offset number of characters from end.

    UDF to find index of seekStr in inputStr from end of inputStr skipping offset number of characters from end. Offset index is number of characters, from the end of str, to skip before searching. Returned string position is 1 based position.

  195. lazy val string_run_length_split: UserDefinedFunction
  196. lazy val string_split: UserDefinedFunction

    UDF to split input string via delimiter string.

  197. lazy val string_split_no_empty: UserDefinedFunction

    UDF to split input string via delimiter string and remove all empty subtrings.

  198. def string_substring(input: Column, start_position: Column, length: Column = lit(Int.MaxValue)): Column

    Method to find substring of input string.

    Method to find substring of input string.

    input

    string on which to find substring.

    start_position

    1 based starting position to find substring from.

    length

    total length of substring to be found.

    returns

    substring of input string

    Annotations
    @Py4JWhitelist()
  199. def string_suffix(input: Column, len: Int): Column
    Annotations
    @Py4JWhitelist()
  200. lazy val string_suffix: UserDefinedFunction
  201. lazy val string_to_hex: UserDefinedFunction
  202. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  203. lazy val testUDFF: UserDefinedFunction
  204. lazy val test_characters_all: UserDefinedFunction

    UDF to identify the number of characters in inputStr which are present in charFlag

  205. def timezone_to_utc(timezone: String, time: Column): Column

    Method to convert

    Method to convert

    Annotations
    @Py4JWhitelist()
  206. def toString(): String
    Definition Classes
    AnyRef → Any
  207. def today(): Column

    Method to return integer value representing number of days to today from “1-1-1990”.

    Method to return integer value representing number of days to today from “1-1-1990”.

    returns

    integer value

    Annotations
    @Py4JWhitelist()
  208. lazy val translate_bytes: UserDefinedFunction

    UDF to return a string in the native character set made up of bytes from the given map.

    UDF to return a string in the native character set made up of bytes from the given map. Each byte of the result is the value of map indexed by the character code of the corresponding byte of the input string str. The function returns NULL if any argument is NULL.

  209. lazy val truncateMicroSeconds: UserDefinedFunction

    UDF to truncate microseconds part of timestamp.

    UDF to truncate microseconds part of timestamp. This is needed as abinitio and spark has some incompatibility in microseconds part of timestamp format.

  210. lazy val type_info: UserDefinedFunction
  211. lazy val type_info_with_includes: UserDefinedFunction
  212. lazy val unique_identifier: UserDefinedFunction
  213. lazy val url_encode_escapes: UserDefinedFunction
  214. lazy val vector_avg: UserDefinedFunction
  215. lazy val vector_stdev: UserDefinedFunction
  216. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  217. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  218. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  219. def warning(errorMsg: String): Column
    Annotations
    @Py4JWhitelist()
  220. def windowSpec(partitionByExpr: Column = lit(1)): WindowSpec
    Annotations
    @Py4JWhitelist()
  221. def windowSpecPrevRow(partitionByExpr: Column = lit(1)): WindowSpec
    Annotations
    @Py4JWhitelist()
  222. lazy val writeIntegerToBytes: UserDefinedFunction
  223. lazy val writeLongToBytes: UserDefinedFunction
  224. def xmlStringToJsString(input: String): String
  225. lazy val xmlToJSON: UserDefinedFunction
  226. def xml_split(content: Column, dmlStr: String): Column
    Annotations
    @Py4JWhitelist()
  227. def yyyyMMdd_to_YYYYJJJ(in_date: Column): Column

    Converts yyyyyMMdd to YYYYJJJ

    Converts yyyyyMMdd to YYYYJJJ

    in_date

    date in yyyyMMdd format

    returns

    a date converted to YYYYJJJ

    Annotations
    @Py4JWhitelist()
  228. def zip_eventInfo_arrays(column1: Column, column2: Column): Column

    Method to zip two arrays with first one having event_type and second one having event_text

    Method to zip two arrays with first one having event_type and second one having event_text

    Annotations
    @Py4JWhitelist()
  229. object LongSequence
  230. object RecordIterator extends Serializable

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated @deprecated
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

Inherited from AnyRef

Inherited from Any

Ungrouped