org.apache.druid.java.util.common

Class FileUtils

java.lang.Object

org.apache.druid.java.util.common.FileUtils

public class FileUtils
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileUtils.FileCopyResult Keeps results of a file copy, including children and total size of the resultant files.
static interface	FileUtils.InputStreamSupplier Like ByteSource, but this is an interface, which allows use of lambdas.
static class	FileUtils.LinkOrCopyResult
static interface	FileUtils.OutputStreamConsumer<T>

Field Summary

Fields

Modifier and Type	Field and Description
static com.google.common.base.Predicate<Throwable>	IS_EXCEPTION Useful for retry functionality that doesn't want to stop Throwables, but does want to retry on Exceptions

Constructor Summary

Constructors

Constructor and Description
FileUtils()

Method Summary

Modifier and Type	Method and Description
static long	copyLarge(FileUtils.InputStreamSupplier inputSource, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry) Copy a potentially large amount of data from an input source to a file.
static <T> long	copyLarge(T object, ObjectOpenFunction<T> objectOpenFunction, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry) Copies data from the InputStream opened with objectOpenFunction to the given file.
static File	createTempDir() Creates a temporary directory inside the configured temporary space (java.io.tmpdir).
static File	createTempDir(String prefix) Creates a temporary directory inside the configured temporary space (java.io.tmpdir).
static void	deleteDirectory(File directory) Equivalent to FileUtils.deleteDirectory(File).
static long	getFileSize(File file) Computes the size of the file.
static FileUtils.LinkOrCopyResult	linkOrCopy(File src, File dest) Hard-link "src" as "dest", if possible.
static MappedByteBufferHandler	map(File file) Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).
static MappedByteBufferHandler	map(File file, long offset, long length) Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).
static MappedByteBufferHandler	map(RandomAccessFile randomAccessFile, long offset, long length) Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).
static void	mkdirp(File directory) Create "directory" and all intermediate directories as needed.
static FileUtils.FileCopyResult	retryCopy(com.google.common.io.ByteSource byteSource, File outFile, com.google.common.base.Predicate<Throwable> shouldRetry, int maxAttempts) Copy input byte source to outFile.
static <T> T	writeAtomically(File file, File tmpDir, FileUtils.OutputStreamConsumer<T> f) Write to a file atomically, by first writing to a temporary file in given tmpDir directory and then moving it to the target location.
static <T> T	writeAtomically(File file, FileUtils.OutputStreamConsumer<T> f) Write to a file atomically, by first writing to a temporary file in the same directory and then moving it to the target location.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

IS_EXCEPTION

public static final com.google.common.base.Predicate<Throwable> IS_EXCEPTION

Useful for retry functionality that doesn't want to stop Throwables, but does want to retry on Exceptions

Constructor Detail

FileUtils

public FileUtils()

Method Detail

retryCopy

public static FileUtils.FileCopyResult retryCopy(com.google.common.io.ByteSource byteSource,
                                                 File outFile,
                                                 com.google.common.base.Predicate<Throwable> shouldRetry,
                                                 int maxAttempts)

Copy input byte source to outFile. If outFile exists, it is attempted to be deleted.

Parameters:

byteSource - Supplier for an input stream that is to be copied. The resulting stream is closed each iteration

outFile - Where the file should be written to.

shouldRetry - Predicate indicating if an error is recoverable and should be retried.

maxAttempts - The maximum number of assumed recoverable attempts to try before completely failing.

Throws:

RuntimeException - wrapping the inner exception on failure.

map

public static MappedByteBufferHandler map(File file)
                                   throws IOException

Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).

Files are mapped from offset 0 to its length.

This only works for files <= Integer.MAX_VALUE bytes.

Similar to Files.map(File), but returns MappedByteBufferHandler, that makes it easier to unmap the buffer within try-with-resources pattern:

 try (MappedByteBufferHandler fileMappingHandler = FileUtils.map(file)) {
   ByteBuffer fileMapping = fileMappingHandler.get();
   // use mapped buffer
 }

Parameters:

file - the file to map

Returns:

a MappedByteBufferHandler, wrapping a read-only buffer reflecting file

Throws:

FileNotFoundException - if the file does not exist

IOException - if an I/O error occurs

IllegalArgumentException - if length is greater than Integer.MAX_VALUE

See Also:

FileChannel.map(FileChannel.MapMode, long, long)

map

public static MappedByteBufferHandler map(File file,
                                          long offset,
                                          long length)
                                   throws IOException

Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).

Parameters:

file - the file to map

offset - starting offset for the mmap

length - length for the mmap

Returns:

a MappedByteBufferHandler, wrapping a read-only buffer reflecting file

Throws:

FileNotFoundException - if the file does not exist

IOException - if an I/O error occurs

IllegalArgumentException - if length is greater than Integer.MAX_VALUE

See Also:

FileChannel.map(FileChannel.MapMode, long, long)

map

public static MappedByteBufferHandler map(RandomAccessFile randomAccessFile,
                                          long offset,
                                          long length)
                                   throws IOException

Fully maps a file read-only in to memory as per FileChannel.map(FileChannel.MapMode, long, long).

Parameters:

randomAccessFile - the file to map. The file will not be closed.

offset - starting offset for the mmap

length - length for the mmap

Returns:

a MappedByteBufferHandler, wrapping a read-only buffer reflecting randomAccessFile

Throws:

IOException - if an I/O error occurs

IllegalArgumentException - if length is greater than Integer.MAX_VALUE

See Also:

FileChannel.map(FileChannel.MapMode, long, long)

writeAtomically

public static <T> T writeAtomically(File file,
                                    FileUtils.OutputStreamConsumer<T> f)
                             throws IOException

Write to a file atomically, by first writing to a temporary file in the same directory and then moving it to the target location. More docs at writeAtomically(File, File, OutputStreamConsumer) .

Throws:

IOException

writeAtomically

public static <T> T writeAtomically(File file,
                                    File tmpDir,
                                    FileUtils.OutputStreamConsumer<T> f)
                             throws IOException

Write to a file atomically, by first writing to a temporary file in given tmpDir directory and then moving it to the target location. This function attempts to clean up its temporary files when possible, but they may stick around (for example, if the JVM crashes partway through executing the function). In any case, the target file should be unharmed. The OutputStream passed to the consumer is uncloseable; calling close on it will do nothing. This is to ensure that the stream stays open so we can fsync it here before closing. Hopefully, this doesn't cause any problems for callers. This method is not just thread-safe, but is also safe to use from multiple processes on the same machine.

Throws:

IOException

copyLarge

public static <T> long copyLarge(T object,
                                 ObjectOpenFunction<T> objectOpenFunction,
                                 File outFile,
                                 byte[] fetchBuffer,
                                 com.google.common.base.Predicate<Throwable> retryCondition,
                                 int numTries,
                                 String messageOnRetry)
                          throws IOException

Copies data from the InputStream opened with objectOpenFunction to the given file. This method is supposed to be used for copying large files. The output file is deleted automatically if copy fails.

Parameters:

object - object to open

objectOpenFunction - function to open the given object

outFile - file to write data

fetchBuffer - a buffer to copy data from the input stream to the file

retryCondition - condition which should be satisfied for retry

numTries - max number of retries

messageOnRetry - log message on retry

Returns:

the number of bytes copied

Throws:

IOException

copyLarge

public static long copyLarge(FileUtils.InputStreamSupplier inputSource,
                             File outFile,
                             byte[] fetchBuffer,
                             com.google.common.base.Predicate<Throwable> retryCondition,
                             int numTries,
                             String messageOnRetry)
                      throws IOException

Copy a potentially large amount of data from an input source to a file.

Throws:

IOException

getFileSize

public static long getFileSize(File file)

Computes the size of the file. If it is a directory, computes the size up to a depth of 1.

createTempDir

public static File createTempDir()

Creates a temporary directory inside the configured temporary space (java.io.tmpdir). Similar to the method Files.createTempDir() from Guava, but has nicer error messages.

Throws:

IllegalStateException - if the directory could not be created

createTempDir

public static File createTempDir(@Nullable
                                 String prefix)

Creates a temporary directory inside the configured temporary space (java.io.tmpdir). Similar to the method Files.createTempDir() from Guava, but has nicer error messages.

Parameters:

prefix - base directory name; if null/empty then this method will use "druid"

Throws:

IllegalStateException - if the directory could not be created

mkdirp

public static void mkdirp(File directory)
                   throws IOException

Create "directory" and all intermediate directories as needed. If the directory is successfully created, or already exists, returns quietly. Otherwise, throws an IOException. Simpler to use than File.mkdirs(), and more reliable since it is safe from races where two threads try to create the same directory at the same time. The name is inspired by UNIX mkdir -p, which has the same behavior.

Throws:

IOException

deleteDirectory

public static void deleteDirectory(File directory)
                            throws IOException

Equivalent to FileUtils.deleteDirectory(File). Exists here mostly so callers can avoid dealing with our FileUtils and the Commons FileUtils having the same name.

Throws:

IOException

linkOrCopy

public static FileUtils.LinkOrCopyResult linkOrCopy(File src,
                                                    File dest)
                                             throws IOException

Hard-link "src" as "dest", if possible. If not possible -- perhaps they are on separate filesystems -- then copy "src" to "dest".

Returns:

whether a link or copy was made. Can be safely ignored if you don't care.

Throws:

IOException - if something went wrong