io

Type Members

1. class CheckpointReader extends utilities.Closeable

   

   Helper class for reading checkpoint files.

   Helper class for reading checkpoint files.

   This class currently only interacts with single-slice (i.e., non-partitioned) variables.

2. sealed trait CompressionType extends AnyRef

   

3. case class DirectoryLoader[T](directory: Path, loaderFactory: (Path) ⇒ Loader[T], pathFilter: (Path) ⇒ Boolean = _ => true) extends Product with Serializable

   

   A directory loader that wraps a loader factory to load entries from a sequence of paths.

   A directory loader that wraps a loader factory to load entries from a sequence of paths.

   A Loader reads a path and produces an iterator over entries. An DirectoryLoader takes a directory, a factory for entry loaders, and optionally a path filter and watches all the paths inside that directory.

   This class is only valid under the assumption that only one path will be written to by the data source at a time and that once the source stops writing to a path, it will start writing to a new path that is lexicographically greater, and never come back. It uses some heuristics to check whether this is true based on tracking changes to the files' sizes, but the checks can yield false negatives. However, they should never yield false positives.

   directory

   Directory to watch.

   loaderFactory

   Factory for loaders.

   pathFilter

   Optional path filter to use.

4. case class FileIO(filePath: Path, mode: Mode, readBufferSize: Long = 1024 * 512) extends utilities.Closeable with Product with Serializable

   

   Helper class used for reading from and writing to a file.

   Helper class used for reading from and writing to a file.

   IMPORTANT: FileIO.close() needs to be called after being done with this object in order to prevent memory leaks. The memory will not be automatically freed, like its done for tensors and graphs.

   filePath

   Path to a file.

   mode

   Mode in which to open the file.

   readBufferSize

   Buffer size used when reading from the file.

5. trait Loader[T] extends AnyRef

   

   Simple trait for representing data loaders of arbitrary types.