DoricColumn
Related Docs:
object DoricColumn
| package doric
Boolean AND
Boolean AND
Similar to concat doric function, but only with two columns
Similar to concat doric function, but only with two columns
Type & null safe equals between Columns
Type & null safe equals between Columns
the column to compare
a reference to a Boolean DoricColumn with the comparation
Type safe distinct between Columns
Type safe distinct between Columns
the column to compare
a reference to a Boolean DoricColumn with the comparation
Type safe equals between Columns
Type safe equals between Columns
the column to compare
a reference to a Boolean DoricColumn with the comparation
Computes the absolute value of a numeric value.
Computes the absolute value of a numeric value.
Inverse cosine of column in radians, as if computed by java.lang.Math.acos
Inverse cosine of column in radians, as if computed by java.lang.Math.acos
Returns the date that is days days after date column
Returns the date that is days days after date column
A column of the number of days to add to date column, can be negative to subtract days
Timestamp columns will be truncated to Date column
Adds to the Date or Timestamp column the number of months
Adds to the Date or Timestamp column the number of months
the number of months to add, can be negative to subtract.
Date column after adding months
Timestamp columns will be truncated to Date column
Boolean AND
Boolean AND
Gives the column an alias.
Gives the column an alias.
the alias to set the column.
DoricColumn with the alias
Gives the column an alias.
Gives the column an alias.
the alias to set the column.
DoricColumn with the alias
Sorts a column in ascending order
Sorts a column in ascending order
A DoricColumn of the provided type
Sorts a column in ascending order with null values returned before non-null values
Sorts a column in ascending order with null values returned before non-null values
A DoricColumn of the provided type
Sorts a column in ascending order with null values returned after non-null values
Sorts a column in ascending order with null values returned after non-null values
A DoricColumn of the provided type
Computes the numeric value of the first character of the string column, and returns the result as an int column.
Computes the numeric value of the first character of the string column, and returns the result as an int column.
Inverse sine of the column in radians, as if computed by java.lang.Math.asin
Inverse sine of the column in radians, as if computed by java.lang.Math.asin
Inverse tangent of the column as if computed by java.lang.Math.atan
Inverse tangent of the column as if computed by java.lang.Math.atan
The theta component of the point (r, theta) in polar coordinates that corresponds to the point (x, y) in Cartesian coordinates, as if computed by java.lang.Math.atan2
The theta component of the point (r, theta) in polar coordinates that corresponds to the point (x, y) in Cartesian coordinates, as if computed by java.lang.Math.atan2
The column corresponds to yCoordinates
Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
decimal type
Returns the value of the column rounded to 0 decimal places with HALF_EVEN round mode
Returns the value of the column rounded to 0 decimal places with HALF_EVEN round mode
decimal type
Computes the BASE64 encoding of a binary column and returns it as a string column.
Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.
An expression that returns the string representation of the binary value of the given long column.
An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".
Computes bitwise NOT (~) of a number.
Computes bitwise NOT (~) of a number.
Cast the column.
Cast the column.
the type to cast to.
the DoricColumn of the provided type.
Computes the cube-root of the given value
Computes the cube-root of the given value
Computes the ceiling of the given value.
Computes the ceiling of the given value.
decimal type
Contains the other element.
Contains the other element. Returns a boolean column based on a string match.
Convert a number in a string column from one base to another.
Convert a number in a string column from one base to another.
from which base
to which base
Cosine of the angle, as if computed by java.lang.Math.cos
Cosine of the angle, as if computed by java.lang.Math.cos
Hyperbolic cosine of the angle, as if computed by java.lang.Math.cosh
Hyperbolic cosine of the angle, as if computed by java.lang.Math.cosh
Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a long column.
Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a long column.
Extracts the day of the month as an integer from a given date.
Extracts the day of the month as an integer from a given date.
Extracts the day of the week as an integer from a given date.
Extracts the day of the week as an integer from a given date. Ranges from 1 for a Sunday through to 7 for a Saturday
Extracts the day of the year as an integer from a given date.
Extracts the day of the year as an integer from a given date.
Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
Sorts a column in descending order
Sorts a column in descending order
A DoricColumn of the provided type
Sorts a column in descending order with null values returned before non-null values
Sorts a column in descending order with null values returned before non-null values
A DoricColumn of the provided type
Sorts a column in descending order with null values returned after non-null values
Sorts a column in descending order with null values returned after non-null values
A DoricColumn of the provided type
Returns the number of days from date column to dateCol.
Returns the number of days from date column to dateCol.
A Date or Timestamp column
Returns value for the given key in value.
Returns value for the given key in value.
Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Sets the moment to the last day of the same month.
Sets the moment to the last day of the same month.
String ends with.
String ends with. Returns a boolean column based on a string match.
Computes the exponential of the given column
Computes the exponential of the given column
Creates a new row for each element in the given map column.
Creates a new row for each element in the given map column.
ORIGINAL SPARK DORIC
+----------------+ +---+-----+ +------+
|col | |key|value| |col |
+----------------+ +---+-----+ +------+
|[a -> b, c -> d]| |a |b | |{a, b}|
|[] | |c |d | |{c, d}|
|null | +---+-----+ +------+
+----------------+WARNING: Unlike spark, doric returns a struct
Creates a new row for each element in the given map column.
Creates a new row for each element in the given map column.
ORIGINAL SPARK DORIC +---+----------------+ +---+----+-----+ +---+------+ |ix |col | |ix |key |value| |ix |col | +---+----------------+ +---+----+-----+ +---+------+ |1 |{a -> b, c -> d}| |1 |a |b | |1 |{a, b}| |2 |{} | |1 |c |d | |1 |{c, d}| |3 |null | |2 |null|null | |2 |null | +---+----------------+ |3 |null|null | |3 |null | +---+----+-----+ +---+------+
WARNING: Unlike spark, doric returns a struct
,Unlike explode, if the array is null or empty then null is produced.
Computes the exponential of the given value minus one.
Computes the exponential of the given value minus one.
Computes the factorial of the given value.
Computes the factorial of the given value.
Computes the floor of the given value
Computes the floor of the given value
decimal type
Converts a date to a value of string in the format specified by the date format given by the second argument.
Converts a date to a value of string in the format specified by the date format given by the second argument.
A pattern dd.MM.yyyy would return a string like 18.03.1993
java.lang.IllegalArgumentException if malformed format
Use specialized functions like 'year' whenever possible as they benefit from a specialized implementation.
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.
If d is 0, the result has no decimal point or fractional part. If d is less than 0, the result will be null.
Parses a column containing a JSON string into a StructType with the specified schema.
Parses a column containing a JSON string into a StructType with the specified schema.
scaladoc link (issue #135)
Returns null, in the case of an unparseable string
org.apache.spark.sql.functions.from_json(e:org\.apache\.spark\.sql\.Column,schema:org\.apache\.spark\.sql\.types\.DataType,options:scala\.collection\.immutable\.Map\[java\.lang\.String,java\.lang\.String\]):* org.apache.spark.sql.functions.from_json
Parses a column containing a JSON string into a StructType with the specified schema.
Parses a column containing a JSON string into a StructType with the specified schema.
Returns null, in the case of an unparseable string
Parses a column containing a JSON string into a StructType with the specified schema.
Parses a column containing a JSON string into a StructType with the specified schema.
scaladoc link (issue #135)
Returns null, in the case of an unparseable string
org.apache.spark.sql.functions.from_json(e:org\.apache\.spark\.sql\.Column,schema:org\.apache\.spark\.sql\.types\.StructType,options:scala\.collection\.immutable\.Map\[java\.lang\.String,java\.lang\.String\]):* org.apache.spark.sql.functions.from_json
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
java.lang.IllegalArgumentException if invalid format
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the yyyy-MM-dd HH:mm:ss format.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone. For example, 'GMT+1' would yield '2017-07-14 03:40:00.0'.
java.time.DateTimeException if invalid timeZone
scaladoc link (issue #135)
org.apache.spark.sql.functions.from_utc_timestamp
Returns the value if exist of the key
Returns the value if exist of the key
the required key
a DoricColumn of the type of value, can be null if the key doesn't exist.
Retrieves the child row of the Struct column
Retrieves the child row of the Struct column
the expected type of the child column.
the column name expected to find in the struct.
the location if an error is generated
a reference to the child column of the provided type.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
df.withColumn("test", colString("col1").getJsonObject[String](colString("col2"))) .show(false) +-----------------------------------------+----*----* |col1 |col2|test| +-----------------------------------------+----*----* |{"a": 1,"b": "a","date": "26/08/2015"} |$.a |1 | |{"a": 2,"b": "test","date": "26/08/2015"}|$.b |test| |{"a": 3} |$.b |null| +-----------------------------------------+----*----*
It will return null if the input json string is invalid.
Computes hex value of the given column
Computes hex value of the given column
group Numeric Type
Computes sqrt(a2 + b2) without intermediate overflow or underflow.
Computes sqrt(a2 + b2) without intermediate overflow or underflow.
Locate the position of the first occurrence of substr column in the given string.
Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null.
The position is not zero based, but 1 based index. Returns 0 if substr could not be found in str.
Returns a new string column by converting the first letter of each word to uppercase.
Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace.
For example, "hello world" will become "Hello World".
Checks if the element is equal to any of the provided literals.
Checks if the element is equal to any of the provided literals.
literals to compare to
Boolean DoricColumn with the comparation logic.
Checks if the value of the column is not a number
Checks if the value of the column is not a number
Boolean DoricColumn
Checks if the value of the column is not null
Checks if the value of the column is not null
Boolean DoricColumn
Checks if the value of the column is null
Checks if the value of the column is null
Boolean DoricColumn
Creates a new row for a json column according to the given field names.
Creates a new row for a json column according to the given field names.
It does not return one column but N columns
Returns an unordered array containing the keys of the map.
Returns an unordered array containing the keys of the map.
the DoricColumn of the Array of keys
Returns the last day of the month which the given date belongs to.
Returns the last day of the month which the given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015.
Computes the character length of a given string or number of bytes of a binary string.
Computes the character length of a given string or number of bytes of a binary string. The length of character strings include the trailing spaces. The length of binary strings includes binary zeros.
Computes the Levenshtein distance of the two given string columns.
Computes the Levenshtein distance of the two given string columns.
SQL like expression.
SQL like expression. Returns a boolean column based on a SQL LIKE match.
Transforms the original value to a literal.
Transforms the original value to a literal.
a literal with the same type.
Locate the position of the first occurrence of substr in a string column, after position pos.
Locate the position of the first occurrence of substr in a string column, after position pos.
scaladoc link (issue #135)
The position is not zero based, but 1 based index. returns 0 if substr could not be found in str.
org.apache.spark.sql.functions.locate
Computes the natural logarithm of the given value.
Computes the natural logarithm of the given value.
Computes the logarithm of the given value in base 10.
Computes the logarithm of the given value in base 10.
Computes the natural logarithm of the given value plus one.
Computes the natural logarithm of the given value plus one.
Computes the logarithm of the given value in base 2.
Computes the logarithm of the given value in base 2.
Converts a string column to lower case.
Converts a string column to lower case.
Left-pad the string column with pad to a length of len.
Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.
scaladoc link (issue #135)
org.apache.spark.sql.functions.lpad
Trim the specified character string from left end for the specified string column.
Trim the specified character string from left end for the specified string column.
Trim the spaces from left end for the specified string value.
Trim the spaces from left end for the specified string value.
Same as rLike doric function.
Same as rLike doric function.
SQL RLIKE expression (LIKE with Regex). Returns a boolean column based on a regex match.
Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
Extracts the month as an integer from a given date.
Extracts the month as an integer from a given date.
Returns number of months between dates dateCol and date column.
Returns number of months between dates dateCol and date column.
Date or Timestamp column
If roundOff is set to true, the result is rounded off to 8 digits;
it is not rounded otherwise.
Returns number of months between dates date column and dateCol.
Returns number of months between dates date column and dateCol.
A whole number is returned if both inputs have the same day of month or both are the last day of their respective months. Otherwise, the difference is calculated assuming 31 days per month.
For example:
Date or Timestamp column
Date("2017-11-14").monthsBetween(Date("2017-07-14")) // returns 4.0 Date("2017-01-01").monthsBetween(Date("2017-01-10")) // returns 0.29032258 Timestamp("2017-06-01 00:00:00").monthsBetween(Timestamp("2017-06-16 12:00:00")) // returns -0.5
Returns col1 if it is not NaN, or col2 if col1 is NaN.
Returns col1 if it is not NaN, or col2 if col1 is NaN.
Unary minus, i.e.
Unary minus, i.e. negate the expression.
// Select the amount column and negates all values. // Scala: df.select( -df("amount") )
DayTimeIntervalType & YearMonthIntervalType
Returns the first date which is later than the value of the date column that is on the
specified day of the week.
Returns the first date which is later than the value of the date column that is on the
specified day of the week.
Case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"
For example, Date("2015-07-27").nextDay("Sunday") returns Date("2015-08-02")
because that is the first Sunday after 2015-07-27.
scaladoc link (issue #135)
Timestamp columns will be truncated to Date column
org.apache.spark.sql.functions.next_day
Boolean OR
Boolean OR
Returns the positive value of dividend mod divisor.
Returns the positive value of dividend mod divisor.
Pipes the column with the provided transformation
Pipes the column with the provided transformation
the returning type
the function to apply to the column.
the DoricColumn reference with of the provided logic
Creates a new row for each element with position in the given map column.
Creates a new row for each element with position in the given map column.
ORIGINAL SPARK DORIC +----------------+ +---+---+-----+ +---------+ |col | |pos|key|value| |col | +----------------+ +---+---+-----+ +---------+ |[a -> b, c -> d]| |1 |a |b | |{1, a, b}| |[] | |2 |c |d | |{2, c, d}| |null | +---+---+-----+ +---------+ +----------------+
WARNING: Unlike spark, doric returns a struct
,Uses the default column name pos for position, and key and value for elements in the map.
Creates a new row for each element with position in the given map column.
Creates a new row for each element with position in the given map column. Unlike posexplode, if the map is null or empty then the row (null, null) is produced.
ORIGINAL SPARK DORIC +----------------+ +---+----+-----+ +---------+ |col | |pos|key |value| |col | +----------------+ +---+----+-----+ +---------+ |[a -> b, c -> d]| |1 |a |b | |{1, a, b}| |[] | |2 |c |d | |{2, c, d}| |null | |2 |null|null | |null | +----------------+ |3 |null|null | |null | +---+----+-----+ +---+-----+
WARNING: Unlike spark, doric returns a struct
,Uses the default column name pos for position, and key and value for elements in the map.
Returns the value of the first argument raised to the power of the second argument.
Returns the value of the first argument raised to the power of the second argument.
Extracts the quarter as an integer from a given date.
Extracts the quarter as an integer from a given date.
SQL RLIKE expression (LIKE with Regex).
SQL RLIKE expression (LIKE with Regex). Returns a boolean column based on a regex match.
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
Extract a specific group matched by a Java regex, from the specified string column.
Extract a specific group matched by a Java regex, from the specified string column. If the regex did not match, or the specified group did not match, an empty string is returned. if the specified group index exceeds the group count of regex, an IllegalArgumentException will be thrown.
java.lang.IllegalArgumentException if the specified group index exceeds the group count of regex
Replace all substrings of the specified string value that match regexp with replacement.
Replace all substrings of the specified string value that match regexp with replacement.
Repeats a string column n times, and returns it as a new string column.
Repeats a string column n times, and returns it as a new string column.
Creates an array containing the left argument repeated the number of times given by the right argument.
Creates an array containing the left argument repeated the number of times given by the right argument.
Returns a reversed string.
Returns a reversed string.
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode.
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode.
decimal type
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode
decimal type
Right-pad the string column with pad to a length of len.
Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.
scaladoc link (issue #135)
org.apache.spark.sql.functions.rpad
Trim the specified character string from right end for the specified string column.
Trim the specified character string from right end for the specified string column.
Trim the spaces from right end for the specified string value.
Trim the spaces from right end for the specified string value.
Parses a JSON string and infers its schema in DDL format.
Parses a JSON string and infers its schema in DDL format.
org.apache.spark.sql.AnalysisException if it is not a foldable string expression or null
Extracts the seconds as an integer from a given timestamp.
Extracts the seconds as an integer from a given timestamp.
Generate a sequence of integers from start to stop, incrementing by step.
Generate a sequence of integers from start to stop, incrementing by step.
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
Generate a sequence of integers from start to stop, incrementing by step.
Generate a sequence of integers from start to stop, incrementing by step.
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
java.lang.IllegalArgumentException if numBits is not in the permitted values
Shift the given value numBits left.
Shift the given value numBits left.
group Numeric Type
(Signed) shift the given value numBits right.
(Signed) shift the given value numBits right.
group Numeric Type
Unsigned shift the given value numBits right.
Unsigned shift the given value numBits right.
group Numeric Type
Computes the signum of the given value.
Computes the signum of the given value.
Sine of the angle, as if computed by java.lang.Math.sin
Sine of the angle, as if computed by java.lang.Math.sin
Hyperbolic sine of the given value, as if computed by java.lang.Math.sinh
Hyperbolic sine of the given value, as if computed by java.lang.Math.sinh
Returns length of map.
Returns length of map.
The function returns null for null input if spark.sql.legacy.sizeOfNull is set to false or spark.sql.ansi.enabled is set to true. Otherwise, the function returns -1 for null input. With the default settings, the function returns -1 for null input.
Returns the soundex code for the specified expression.
Returns the soundex code for the specified expression.
Splits str around matches of the given pattern.
Splits str around matches of the given pattern.
a string representing a regular expression. The regex string should be a Java regular expression.
scaladoc link (issue #135)
org.apache.spark.sql.functions.split
Computes the square root of the specified float value
Computes the square root of the specified float value
String starts with.
String starts with. Returns a boolean column based on a string match.
Returns the date that is days days before date column
Returns the date that is days days before date column
A column of the number of days to subtract from date column, can be negative to add days
Timestamp columns will be truncated to Date column
Substring starts at pos and is of length len when str is String type
or returns the slice of byte array that starts at pos in byte and is
of length len when str is Binary type
Substring starts at pos and is of length len when str is String type
or returns the slice of byte array that starts at pos in byte and is
of length len when str is Binary type
The position is not zero based, but 1 based index.
Returns the substring from string str before count occurrences of the delimiter delim.
Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.
Tangent of the given value, as if computed by java.lang.Math.tan
Tangent of the given value, as if computed by java.lang.Math.tan
Hyperbolic tangent of the given value, as if computed by java.lang.Math.tanh
Hyperbolic tangent of the given value, as if computed by java.lang.Math.tanh
DORIC EXCLUSIVE! Map to array conversion
DORIC EXCLUSIVE! Map to array conversion
Converts the column into a DateType with a specified format
Converts the column into a DateType with a specified format
See <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html"> Datetime Patterns for valid date and time format patterns
A date time pattern detailing the format of e when eis a string
A date, or null if e was a string that could not be cast to a date
or format was an invalid format
Safe casting to Date column
Safe casting to Date column
a Date Column without the hour
Transform date to Instant
Transform date to Instant
Safe casting to LocalDate column
Safe casting to LocalDate column
a LocalDate Column without the hour
Converts time string with the given pattern to timestamp.
Converts time string with the given pattern to timestamp.
See <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html"> Datetime Patterns for valid date and time format patterns
A date time pattern detailing the format of s when s is a string
A timestamp, or null if s was a string that could not be cast to a
timestamp or format was an invalid format
Transform date to timestamp
Transform date to timestamp
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
java.time.DateTimeException if invalid timeZone
scaladoc link (issue #135)
org.apache.spark.sql.functions.to_utc_timestamp
Translate any character in the src by a character in replaceString.
Translate any character in the src by a character in replaceString. The
characters in replaceString correspond to the characters in
matchingString. The translate will happen when any character in the
string matches the character in the matchingString.
Trim the specified character from both ends for the specified string column (literal).
Trim the specified character from both ends for the specified string column (literal).
Trim the spaces from both ends for the specified string column.
Trim the spaces from both ends for the specified string column.
Returns date truncated to the unit specified by the format.
Returns date truncated to the unit specified by the format.
For example, Timestamp("2018-11-19 12:01:19").trunc("year") returns Date("2018-01-01")
Timestamp columns will be truncated to Date column
Inverse of hex.
Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to the byte representation of number
Inversion of boolean expression, i.e.
Inversion of boolean expression, i.e. NOT.
Unary minus, i.e.
Unary minus, i.e. negate the expression.
Decodes a BASE64 encoded string column and returns it as a binary column.
Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64.
Converts date/timestamp with given pattern to Unix timestamp (in seconds).
Converts date/timestamp with given pattern to Unix timestamp (in seconds).
A long, or null if the input was a string not of the correct format
java.lang.IllegalArgumentException if invalid pattern
Allows to cast to posible wrong or with unexpected behaviour type, like casting String to Int, that can be resulted in null types.
Allows to cast to posible wrong or with unexpected behaviour type, like casting String to Int, that can be resulted in null types.
the type to cast to.
the DoricColumn of the provided type
Converts a string column to upper case.
Converts a string column to upper case.
Returns an unordered array containing the values of the map.
Returns an unordered array containing the values of the map.
the DoricColumn of the Array of values
Extracts the week number as an integer from a given date.
Extracts the week number as an integer from a given date.
A week is considered to start on a Monday and week 1 is the first week with more than 3 days, as defined by ISO 8601
Generates tumbling time windows given a timestamp specifying column.
Generates tumbling time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive.
A string specifying the width of the window, e.g. 10 minutes,
1 second. Check org.apache.spark.unsafe.types.CalendarInterval for
valid duration identifiers. Note that the duration is a fixed length of
time, and does not vary over time according to a calendar. For example,
1 day always means 86,400,000 milliseconds, not a calendar day.
A string specifying the sliding interval of the window, e.g. 1 minute.
A new window will be generated every slideDuration. Must be less than
or equal to the windowDuration. Check
org.apache.spark.unsafe.types.CalendarInterval for valid duration
identifiers. This duration is likewise absolute, and does not vary
according to a calendar.
The offset with respect to 1970-01-01 00:00:00 UTC with which to start
window intervals. For example, in order to have hourly tumbling windows that
start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... provide
startTime as 15 minutes.
Generates tumbling time windows given a timestamp specifying column.
Generates tumbling time windows given a timestamp specifying column. Window starts are inclusive but the window ends are exclusive.
A string specifying the width of the window, e.g. 10 minutes,
1 second. Check org.apache.spark.unsafe.types.CalendarInterval for
valid duration identifiers.
Extracts the year as an integer from a given date.
Extracts the year as an integer from a given date.
Boolean OR
Boolean OR
(doricColumn: doric.DateColumnLikeSyntax[T]).<(other)
(doricColumn: doric.NumericOperationsSyntax[T]).<(other)
(doricColumn: doric.DateColumnLikeSyntax[T]).<=(other)
(doricColumn: doric.NumericOperationsSyntax[T]).<=(other)
(doricColumn: doric.DateColumnLikeSyntax[T]).>(other)
(doricColumn: doric.NumericOperationsSyntax[T]).>(other)
(doricColumn: doric.DateColumnLikeSyntax[T]).>=(other)
(doricColumn: doric.NumericOperationsSyntax[T]).>=(other)
Converts a column containing a StructType into a JSON string with the specified schema.
Converts a column containing a StructType into a JSON string with the specified schema.
(doricColumn: doric.MapColumnOps[K, V]).toJson(options)
java.lang.IllegalArgumentException in the case of an unsupported type.
scaladoc link (issue #135)
org.apache.spark.sql.functions.to_json(e:org\.apache\.spark\.sql\.Column,options:scala\.collection\.immutable\.Map\[java\.lang\.String,java\.lang\.String\]):* org.apache.spark.sql.functions.to_csv
Converts a column containing a StructType into a JSON string with the specified schema.
Converts a column containing a StructType into a JSON string with the specified schema.
(doricColumn: doric.DStructOps[T]).toJson(options)
java.lang.IllegalArgumentException in the case of an unsupported type.
scaladoc link (issue #135)
org.apache.spark.sql.functions.to_json(e:org\.apache\.spark\.sql\.Column,options:scala\.collection\.immutable\.Map\[java\.lang\.String,java\.lang\.String\]):* org.apache.spark.sql.functions.to_csv
Converts date/timestamp to Unix timestamp (in seconds), using the default timezone and the default locale.
Converts date/timestamp to Unix timestamp (in seconds), using the default timezone and the default locale.
A long
(doricColumn: doric.StringOperationsSyntax).unixTimestamp
Converts date/timestamp to Unix timestamp (in seconds), using the default timezone and the default locale.
Converts date/timestamp to Unix timestamp (in seconds), using the default timezone and the default locale.
A long
(doricColumn: doric.DateColumnLikeSyntax[T]).unixTimestamp