(Since version 0.3.4) use the constructor with no spark session
(Since version 0.3.4) use the constructor with no spark session
(Since version 0.3.4) use the constructor with no spark session
(Since version 0.3.4) use the constructor with no spark session
(Since version 0.3.4) use the constructor with no spark session
Get the basePath of the current path.
Get the basePath of the current path. If the value path is a file path, then its basePath will be it's parent's path. Otherwise it will be the current path itself.
Delete the current file or directory
Delete the current file or directory
Get the boolean value of dropUserDefinedSuffix.
Get the boolean value of dropUserDefinedSuffix.
true if the column will be dropped, false otherwise
Set to true to drop the column containing user defined suffix (default name _user_defined_suffix)
Set to true to drop the column containing user defined suffix (default name _user_defined_suffix)
true to drop, false to keep
List files to be loaded.
List files to be loaded.
If the current connector has a non-empty filename pattern, then return a list of file paths that match the pattern.
When the filename pattern is not set: If the absolute path of this connector is a directory, return the path of the directory if detailed is set to false. Otherwise, return a list of file paths in the directory
true to return a list of file paths if the current absolute path is a directory
Get the current filesystem based on the path URI
Get the current filesystem based on the path URI
Get the sum of file size
Get the value of user defined suffix column name
Get the value of user defined suffix column name
List ALL the file paths (in format of string) of the current path of connector
List ALL the file paths (in format of string) of the current path of connector
List all the file path (in format of string) to be loaded.
List all the file path (in format of string) to be loaded.
If the current connector has a non-empty filename pattern, then return a list of file paths that match the pattern.
When the filename pattern is not set: If the absolute path of this connector is a directory, return the path of the directory if detailed is set to false. Otherwise, return a list of file paths in the directory
When the filename pattern IS set, a list of file paths will always be returned
true to list all file paths when the absolute path points to a directory otherwise return only the directory path.
List ALL the file paths of the current path of connector
List ALL the file paths of the current path of connector
Read a DataFrame from a file with the path defined during the instantiation.
Read a DataFrame from a file with the path defined during the instantiation.
DataFrame reader for the current path of connector
DataFrame reader for the current path of connector
Reset suffix to None
Reset suffix to None
set to true to ignore the validity check of suffix value
The current version of FileConnector doesn't support a mix of suffix and non-suffix write when the DataFrame is partitioned.
The current version of FileConnector doesn't support a mix of suffix and non-suffix write when the DataFrame is partitioned.
This method will detect, in the case of a partitioned table, if user try to use both suffix write and non-suffix write
an option of suffix in string format
Set the name of user defined suffix column (by default is _user_defined_suffix
Set the name of user defined suffix column (by default is _user_defined_suffix
name of the new key
Write a DataFrame into file
Write a DataFrame into file
dataframe to be written
optional, String, write the df in a sub-directory of the defined path
Write a DataFrame into the given path with the given save mode
Write a DataFrame into the given path with the given save mode
Initialize a DataFrame writer.
Initialize a DataFrame writer. A new writer will be initiate only if the hashcode of input DataFrame is different than the last written DataFrame.
ParquetConnector contains functionality for transforming DataFrame into parquet files