object Utils

Various arbitrary helper functions

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Utils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val CERTIFIED_JDBC_VERSION: String

    The certified JDBC version to work with this spark connector version.

  5. val PROPERTY_NAME_OF_CONNECTOR_VERSION: String

    Important: Never change the value of PROPERTY_NAME_OF_CONNECTOR_VERSION.

    Important: Never change the value of PROPERTY_NAME_OF_CONNECTOR_VERSION. Changing it will cause spark connector doesn't work in some cases.

  6. val SNOWFLAKE_SOURCE_NAME: String

    Literal to be used with the Spark DataFrame's .format method

  7. val SNOWFLAKE_SOURCE_SHORT_NAME: String

    Short literal name of SNOWFLAKE_SOURCE_NAME

  8. val VERSION: String
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def checkFileSystem(uri: URI, hadoopConfig: Configuration): Unit

    Given a URI, verify that the Hadoop FileSystem for that URI is not the S3 block FileSystem.

    Given a URI, verify that the Hadoop FileSystem for that URI is not the S3 block FileSystem. spark-snowflakedb cannot use this FileSystem because the files written to it will not be readable by Snowflake (and vice versa).

  11. def checkThatBucketHasObjectLifecycleConfiguration(tempDir: String, tempDirStorageType: FSType, s3Client: AmazonS3Client): Unit

    Checks whether the S3 bucket for the given UI has an object lifecycle configuration to ensure cleanup of temporary files.

    Checks whether the S3 bucket for the given UI has an object lifecycle configuration to ensure cleanup of temporary files. If no applicable configuration is found, this method logs a helpful warning for the user.

  12. def classForName(className: String): Class[_]
  13. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native() @HotSpotIntrinsicCandidate()
  14. def containVariant(schema: StructType): Boolean

    Check whether the giving DataFrame contains variant type or not

  15. def ensureQuoted(name: String): String

    ensure a name wrapped with double quotes

  16. def ensureUnquoted(name: String): String

    Unquote a name

  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  19. def fixS3Url(url: String): String

    Snowflake COPY and UNLOAD commands don't support s3n or s3a, but users may wish to use them for data loads.

    Snowflake COPY and UNLOAD commands don't support s3n or s3a, but users may wish to use them for data loads. This function converts the URL back to the s3:// format.

  20. def fixUrlForCopyCommand(url: String): String

    Converts url for the copy command.

    Converts url for the copy command. For S3, convert s3a|s3n to s3. For Azure, convert the wasb: url to azure: url.

    url

    the url to be used in hadoop/spark

    returns

    the url to be used in Snowflake

  21. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  22. def getClientInfoJson(): ObjectNode
  23. def getClientInfoString(): String
  24. def getJDBCConnection(params: Map[String, String]): Connection
  25. def getJDBCConnection(params: Map[String, String]): Connection
  26. def getLastCopyLoad: String
  27. def getLastCopyLoadQueryId: String

    Get the query ID for the last Copy Into Table command.

  28. def getLastCopyUnload: String
  29. def getLastGetCommand: String
  30. def getLastPutCommand: String
  31. def getLastSelect: String
  32. def getLastSelectQueryId: String

    Get the query ID for the last query.

  33. def getSizeString(size: Long): String
  34. def getTimeString(milliSeconds: Long): String
  35. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  36. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  37. def isQuoted(name: String): Boolean

    check whether a name is quoted

  38. def joinUrls(a: String, b: String): String

    Joins prefix URL a to path suffix b, and appends a trailing /, in order to create a temp directory path for S3.

  39. def makeTempPath(tempRoot: String): String

    Creates a randomly named temp directory path for intermediate data

  40. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  41. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  42. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  43. def parseMap(source: String): Map[String, String]

    create a map from string for column mapping

  44. def printQuery(params: Map[String, String], query: String): Unit
  45. def quotedName(name: String): String

    wrap a name with double quotes

  46. def quotedNameIgnoreCase(name: String): String

    wrap a name with double quotes without capitalize letters

  47. def readMapFromFile(sc: SparkContext, file: String): Map[String, String]
  48. def readMapFromString(string: String): Map[String, String]

    Same as readMapFromFile, but accepts the file content as an argument

  49. def removeCredentialsFromURI(uri: URI): URI

    Returns a copy of the given URI with the user credentials removed.

  50. def runQuery(params: Map[String, String], query: String): ResultSet
  51. def runQuery(params: Map[String, String], query: String): ResultSet
  52. def sanitizeQueryText(q: String): String

    Removes (hopefully :)) sensitive content from a query string

  53. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  54. def toString(): String
    Definition Classes
    AnyRef → Any
  55. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  56. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  57. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped