This case class contains the various parameters one can set when reading RDDs from S3 using Spark.
This case class contains the various parameters one can set when reading RDDs from S3 using Spark.
TODO: Add persistLevel option
Read all file with an extension contained in the given list.
Override CRS of the input files. If None, the reader will use the file's original CRS.
Name of tiff tag containing the timestamp for the tile.
Pattern for java.time.format.DateTimeFormatter to parse timeTag.
Maximum allowed size of each tiles in output RDD. May result in a one input GeoTiff being split amongst multiple records if it exceeds this size. If no maximum tile size is specific, then each file is broken into 256x256 tiles. If None, then the whole file will be read in. This option is incompatible with numPartitions and anything set to that parameter will be ignored.
How many partitions Spark should create when it repartitions the data.
Desired partition size in bytes, at least one item per partition will be assigned. If no size is specified, then partitions 128 Mb in size will be created by default. This option is incompatible with the numPartitions option. If both are set and maxTileSize isn't, then partitionBytes will be ignored in favor of numPartitions. However, if maxTileSize is set, then partitionBytes will be retained. If None and maxTileSize is defined, then the default partitionBytes' value will still be used. If maxTileSize is also None, then partitionBytes will remain None as well.
How many bytes should be read in at a time when reading a file. If None, then 65536 byte chunks will be read in at a time.
Delimiter to use for S3 objet listings. This provides a way to further define what files should be read. If None, then only the prefix will be used when determing which files to read.
A function to instantiate an S3Client.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
An instance of Options that contains any user defined or default settings.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Function to transform input key basing on the URI information.
An instance of Options that contains any user defined or default settings.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
Creates a RDD[(K, V)] whose K and V on the type of the GeoTiff that is going to be read in.
This function has two modes of operation: When options.maxTileSize is set windows will be read from GeoTiffs and their size and count will be balanced among partitions using partitionBytes option. Resulting partitions will be grouped in relation to GeoTiff segment layout.
When maxTileSize is None the GeoTiffs will be read fully and balanced among partitions using either numPartitions or partitionBytes option.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Function to transform input key basing on the URI information.
An instance of Options that contains any user defined or default settings.
An optional geometry to filter by. If this is provided, it is assumed that all GeoTiffs are in the same CRS, and that this geometry is in that CRS.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband GeoTiffs.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband GeoTiffs. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband GeoTiffs.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband GeoTiffs. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
An instance of Options that contains any user defined or default settings.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
An instance of Options that contains any user defined or default settings.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband GeoTiffs. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles. If a GeoTiff contains multiple bands, only the first will be read.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
An instance of Options that contains any user defined or default settings.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
An instance of Options that contains any user defined or default settings.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles. Will parse a timestamp from a tiff tags specified in options to associate with each tile.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
Options for the reading process. Including the timestamp tiff tag and its pattern.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles. Will parse a timestamp from a tiff tags specified in options to associate with each tile.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Options for the reading process. Including the timestamp tiff tag and its pattern.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as singleband tiles. Will parse a timestamp from the default tiff tags to associate with each file.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles. Will parse a timestamp from a tiff tags specified in options to associate with each tile.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
function to transform input key basing on the URI information.
Options for the reading process. Including the timestamp tiff tag and its pattern.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles. Will parse a timestamp from a tiff tags specified in options to associate with each tile.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
Options for the reading process. Including the timestamp tiff tag and its pattern.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles.
Creates RDD that will read all GeoTiffs in the given bucket and prefix as multiband tiles. Will parse a timestamp from a tiff tags specified in options to associate with each tile.
Name of the bucket on S3 where the files are kept.
Prefix of all of the keys on S3 that are to be read in.
The S3GeoTiffRDD object allows for the creation of whole or windowed RDD[(K, V)]s from files on S3.