Class FileUtils
- java.lang.Object
-
- org.apache.druid.java.util.common.FileUtils
-
public class FileUtils extends Object
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
FileUtils.FileCopyResult
Keeps results of a file copy, including children and total size of the resultant files.static interface
FileUtils.InputStreamSupplier
LikeByteSource
, but this is an interface, which allows use of lambdas.static class
FileUtils.LinkOrCopyResult
static interface
FileUtils.OutputStreamConsumer<T>
-
Field Summary
Fields Modifier and Type Field Description static com.google.common.base.Predicate<Throwable>
IS_EXCEPTION
Useful for retry functionality that doesn't want to stop Throwables, but does want to retry on Exceptions
-
Constructor Summary
Constructors Constructor Description FileUtils()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static long
copyLarge(FileUtils.InputStreamSupplier inputSource, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry)
Copy a potentially large amount of data from an input source to a file.static <T> long
copyLarge(T object, ObjectOpenFunction<T> objectOpenFunction, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry)
Copies data from the InputStream opened with objectOpenFunction to the given file.static File
createTempDir()
Creates a temporary directory inside the configured temporary space (java.io.tmpdir).static File
createTempDir(String prefix)
Creates a temporary directory inside the configured temporary space (java.io.tmpdir).static File
createTempDirInLocation(Path parentDirectory, String prefix)
static void
deleteDirectory(File directory)
Equivalent toFileUtils.deleteDirectory(File)
.static long
getFileSize(File file)
Computes the size of the file.static Path
getTempDir()
static FileUtils.LinkOrCopyResult
linkOrCopy(File src, File dest)
Hard-link "src" as "dest", if possible.static MappedByteBufferHandler
map(File file)
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.static MappedByteBufferHandler
map(File file, long offset, long length)
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.static MappedByteBufferHandler
map(RandomAccessFile randomAccessFile, long offset, long length)
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.static void
mkdirp(File directory)
Create "directory" and all intermediate directories as needed.static FileUtils.FileCopyResult
retryCopy(com.google.common.io.ByteSource byteSource, File outFile, com.google.common.base.Predicate<Throwable> shouldRetry, int maxAttempts)
Copy input byte source to outFile.static <T> T
writeAtomically(File file, File tmpDir, FileUtils.OutputStreamConsumer<T> f)
Write to a file atomically, by first writing to a temporary file in given tmpDir directory and then moving it to the target location.static <T> T
writeAtomically(File file, FileUtils.OutputStreamConsumer<T> f)
Write to a file atomically, by first writing to a temporary file in the same directory and then moving it to the target location.
-
-
-
Field Detail
-
IS_EXCEPTION
public static final com.google.common.base.Predicate<Throwable> IS_EXCEPTION
Useful for retry functionality that doesn't want to stop Throwables, but does want to retry on Exceptions
-
-
Method Detail
-
retryCopy
public static FileUtils.FileCopyResult retryCopy(com.google.common.io.ByteSource byteSource, File outFile, com.google.common.base.Predicate<Throwable> shouldRetry, int maxAttempts)
Copy input byte source to outFile. If outFile exists, it is attempted to be deleted.- Parameters:
byteSource
- Supplier for an input stream that is to be copied. The resulting stream is closed each iterationoutFile
- Where the file should be written to.shouldRetry
- Predicate indicating if an error is recoverable and should be retried.maxAttempts
- The maximum number of assumed recoverable attempts to try before completely failing.- Throws:
RuntimeException
- wrapping the inner exception on failure.
-
map
public static MappedByteBufferHandler map(File file) throws IOException
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.Files are mapped from offset 0 to its length.
This only works for files <=
Integer.MAX_VALUE
bytes.Similar to
Files.map(File)
, but returnsMappedByteBufferHandler
, that makes it easier to unmap the buffer within try-with-resources pattern:try (MappedByteBufferHandler fileMappingHandler = FileUtils.map(file)) { ByteBuffer fileMapping = fileMappingHandler.get(); // use mapped buffer }
- Parameters:
file
- the file to map- Returns:
- a
MappedByteBufferHandler
, wrapping a read-only buffer reflectingfile
- Throws:
FileNotFoundException
- if thefile
does not existIOException
- if an I/O error occursIllegalArgumentException
- if length is greater thanInteger.MAX_VALUE
- See Also:
FileChannel.map(FileChannel.MapMode, long, long)
-
map
public static MappedByteBufferHandler map(File file, long offset, long length) throws IOException
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.- Parameters:
file
- the file to mapoffset
- starting offset for the mmaplength
- length for the mmap- Returns:
- a
MappedByteBufferHandler
, wrapping a read-only buffer reflectingfile
- Throws:
FileNotFoundException
- if thefile
does not existIOException
- if an I/O error occursIllegalArgumentException
- if length is greater thanInteger.MAX_VALUE
- See Also:
FileChannel.map(FileChannel.MapMode, long, long)
-
map
public static MappedByteBufferHandler map(RandomAccessFile randomAccessFile, long offset, long length) throws IOException
Fully maps a file read-only in to memory as perFileChannel.map(FileChannel.MapMode, long, long)
.- Parameters:
randomAccessFile
- the file to map. The file will not be closed.offset
- starting offset for the mmaplength
- length for the mmap- Returns:
- a
MappedByteBufferHandler
, wrapping a read-only buffer reflectingrandomAccessFile
- Throws:
IOException
- if an I/O error occursIllegalArgumentException
- if length is greater thanInteger.MAX_VALUE
- See Also:
FileChannel.map(FileChannel.MapMode, long, long)
-
writeAtomically
public static <T> T writeAtomically(File file, FileUtils.OutputStreamConsumer<T> f) throws IOException
Write to a file atomically, by first writing to a temporary file in the same directory and then moving it to the target location. More docs atwriteAtomically(File, File, OutputStreamConsumer)
.- Throws:
IOException
-
writeAtomically
public static <T> T writeAtomically(File file, File tmpDir, FileUtils.OutputStreamConsumer<T> f) throws IOException
Write to a file atomically, by first writing to a temporary file in given tmpDir directory and then moving it to the target location. This function attempts to clean up its temporary files when possible, but they may stick around (for example, if the JVM crashes partway through executing the function). In any case, the target file should be unharmed. The OutputStream passed to the consumer is uncloseable; calling close on it will do nothing. This is to ensure that the stream stays open so we can fsync it here before closing. Hopefully, this doesn't cause any problems for callers. This method is not just thread-safe, but is also safe to use from multiple processes on the same machine.- Throws:
IOException
-
copyLarge
public static <T> long copyLarge(T object, ObjectOpenFunction<T> objectOpenFunction, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry) throws IOException
Copies data from the InputStream opened with objectOpenFunction to the given file. This method is supposed to be used for copying large files. The output file is deleted automatically if copy fails.- Parameters:
object
- object to openobjectOpenFunction
- function to open the given objectoutFile
- file to write datafetchBuffer
- a buffer to copy data from the input stream to the fileretryCondition
- condition which should be satisfied for retrynumTries
- max number of retriesmessageOnRetry
- log message on retry- Returns:
- the number of bytes copied
- Throws:
IOException
-
copyLarge
public static long copyLarge(FileUtils.InputStreamSupplier inputSource, File outFile, byte[] fetchBuffer, com.google.common.base.Predicate<Throwable> retryCondition, int numTries, String messageOnRetry) throws IOException
Copy a potentially large amount of data from an input source to a file.- Throws:
IOException
-
getFileSize
public static long getFileSize(File file)
Computes the size of the file. If it is a directory, computes the size up to a depth of 1.
-
createTempDir
public static File createTempDir()
Creates a temporary directory inside the configured temporary space (java.io.tmpdir). Similar to the methodFiles.createTempDir()
from Guava, but has nicer error messages.- Throws:
IllegalStateException
- if the directory could not be created
-
createTempDir
public static File createTempDir(@Nullable String prefix)
Creates a temporary directory inside the configured temporary space (java.io.tmpdir). Similar to the methodFiles.createTempDir()
from Guava, but has nicer error messages.- Parameters:
prefix
- base directory name; if null/empty then this method will use "druid"- Throws:
IllegalStateException
- if the directory could not be created
-
getTempDir
public static Path getTempDir()
-
createTempDirInLocation
public static File createTempDirInLocation(Path parentDirectory, @Nullable String prefix)
-
mkdirp
public static void mkdirp(File directory) throws IOException
Create "directory" and all intermediate directories as needed. If the directory is successfully created, or already exists, returns quietly. Otherwise, throws an IOException. Simpler to use thanFile.mkdirs()
, and more reliable since it is safe from races where two threads try to create the same directory at the same time. The name is inspired by UNIXmkdir -p
, which has the same behavior.- Throws:
IOException
-
deleteDirectory
public static void deleteDirectory(File directory) throws IOException
Equivalent toFileUtils.deleteDirectory(File)
. Exists here mostly so callers can avoid dealing with our FileUtils and the Commons FileUtils having the same name.- Throws:
IOException
-
linkOrCopy
public static FileUtils.LinkOrCopyResult linkOrCopy(File src, File dest) throws IOException
Hard-link "src" as "dest", if possible. If not possible -- perhaps they are on separate filesystems -- then copy "src" to "dest".- Returns:
- whether a link or copy was made. Can be safely ignored if you don't care.
- Throws:
IOException
- if something went wrong
-
-