Package alluxio.util
Class CommonUtils
- java.lang.Object
-
- alluxio.util.CommonUtils
-
@ThreadSafe public final class CommonUtils extends java.lang.Object
Common utilities shared by all components in Alluxio.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
CommonUtils.ProcessType
Alluxio process types.
-
Field Summary
Fields Modifier and Type Field Description static java.util.concurrent.atomic.AtomicReference<CommonUtils.ProcessType>
PROCESS_TYPE
Represents the type of Alluxio process running in this JVM.
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static <T> java.lang.String
argsToString(java.lang.String separator, T... args)
Converts varargs of objects to a string.static java.io.IOException
castToIOException(java.lang.Throwable e)
Casts aThrowable
to anIOException
.static java.lang.RuntimeException
closeAndRethrow(com.google.common.io.Closer closer, java.lang.Throwable t)
Closes the Closer and re-throws the Throwable.static java.lang.RuntimeException
closeAndRethrowRuntimeException(com.google.common.io.Closer closer, java.lang.Throwable t)
Similar tocloseAndRethrow(com.google.common.io.Closer, java.lang.Throwable)
but always return a RuntimeException.static void
closeChannel(io.netty.channel.Channel channel)
Closes the netty channel from outside the netty I/O thread.static java.util.Map<java.lang.String,byte[]>
convertFromByteString(java.util.Map<java.lang.String,com.google.protobuf.ByteString> input)
static java.lang.String
convertMsToClockTime(long millis)
Converts milliseconds to clock time.static java.lang.String
convertMsToDate(long millis, java.lang.String dateFormatPattern)
Converts a millisecond number to a formatted date String.static java.util.Map<java.lang.String,com.google.protobuf.ByteString>
convertToByteString(java.util.Map<java.lang.String,byte[]> input)
static <T> T
createNewClassInstance(java.lang.Class<T> cls, java.lang.Class<?>[] ctorClassArgs, java.lang.Object[] ctorArgs)
Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.static java.lang.Thread
createProgressThread(long intervalMs, java.io.PrintStream stream)
Creates a thread which will write "." to the given print stream at the given interval.static java.lang.Thread
createProgressThread(java.io.PrintStream stream)
Convenience method for callingcreateProgressThread(long, PrintStream)
with an interval of 2 seconds.static long
getCurrentMs()
static java.util.List<java.lang.String>
getGroups(java.lang.String userName, AlluxioConfiguration conf)
UsingCachedGroupMapping
to get the group list of a user.static int
getJavaVersion()
static java.lang.String
getPrimaryGroupName(java.lang.String userName, AlluxioConfiguration conf)
Gets the primary group name of a user.static java.lang.Throwable
getRootCause(java.lang.Throwable e)
Gets the root cause of an exception.static java.lang.String
getTmpDir(java.util.List<java.lang.String> tmpDirs)
static java.util.List<java.lang.String>
getUnixGroups(java.lang.String user)
Gets the current user's group list from Unix by running the command 'groups' NOTE.static java.lang.String
getValueFromStaticMapping(java.lang.String mapping, java.lang.String key)
Gets the value with a given key from a static key/value mapping in string format.static java.lang.String
getWorkerDataDirectory(java.lang.String storageDir, AlluxioConfiguration conf)
static <T> void
invokeAll(java.util.concurrent.ExecutorService service, java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs)
Executes the given callables, waiting for them to complete (or time out).static <T> void
invokeAll(java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs)
Executes the given callables, waiting for them to complete (or time out).static boolean
isAddressReachable(java.lang.String hostname, int port, int timeoutMs)
Validates whether a network address is reachable.static boolean
isCollection(java.lang.Object obj)
static boolean
isFatalError(java.lang.Throwable e)
static boolean
isLocalHost(WorkerNetAddress address, AlluxioConfiguration conf)
static <T> java.lang.String
listToString(java.util.List<T> list)
Converts a list of objects to a string.static <T> java.util.function.Supplier<T>
memoize(java.util.function.Supplier<T> original)
Memoize implementation for java.util.function.supplier.static int
parseMajorVersion(java.lang.String version)
static <T> java.util.List<java.util.List<T>>
partition(java.util.List<T> list, int numLists)
Partitions a list into numLists many lists each with around list.size() / numLists elements.static java.lang.String
randomAlphaNumString(int length)
Generates a random alphanumeric string of the given length.static byte[]
randomBytes(int length)
Generates a random byte array of the given length.static java.util.List<java.io.File>
recursiveListLocalDir(java.io.File dir)
Recursively lists a local dir and all its subdirs and return all the files.static <T> java.util.Iterator<T>
singleElementIterator(T element)
Returns an iterator that iterates on a single element.static void
sleepMs(long timeMs)
Sleeps for the given number of milliseconds.static void
sleepMs(org.slf4j.Logger logger, long timeMs)
Sleeps for the given number of milliseconds, reporting interruptions using the given logger.static java.lang.String
stripLeadingAndTrailingQuotes(java.lang.String str)
Strips the leading and trailing quotes from the given string.static java.lang.String
stripPrefixIfPresent(java.lang.String key, java.lang.String prefix)
Strips the prefix from the key if it is present.static java.lang.String
stripSuffixIfPresent(java.lang.String key, java.lang.String suffix)
Strips the suffix if it exists.static java.lang.String
summarizeCollection(java.lang.Object obj)
static java.lang.String[]
toStringArray(java.util.ArrayList<java.lang.String> src)
ParsesArrayList<String>
intoString[]
.static void
unwrapResponse(alluxio.proto.dataserver.Protocol.Response response)
Unwraps aProtocol.Response
.static void
unwrapResponseFrom(alluxio.proto.dataserver.Protocol.Response response, io.netty.channel.Channel channel)
Unwraps aProtocol.Response
associated with a channel.static void
waitFor(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition)
Waits for a condition to be satisfied.static void
waitFor(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition, WaitForOptions options)
Waits for a condition to be satisfied.static <T> T
waitForResult(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,java.lang.Boolean> condition)
Waits for the object to meet a certain condition.static <T> T
waitForResult(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,java.lang.Boolean> condition, WaitForOptions options)
Waits for the object to meet a certain condition.static void
warmUpLoop()
Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.
-
-
-
Field Detail
-
PROCESS_TYPE
public static final java.util.concurrent.atomic.AtomicReference<CommonUtils.ProcessType> PROCESS_TYPE
Represents the type of Alluxio process running in this JVM. NOTE: This will only be set by main methods of Alluxio processes. It will not be set properly for tests. Avoid using this field if at all possible.
-
-
Method Detail
-
createProgressThread
public static java.lang.Thread createProgressThread(java.io.PrintStream stream)
Convenience method for callingcreateProgressThread(long, PrintStream)
with an interval of 2 seconds.- Parameters:
stream
- the print stream to write to- Returns:
- the thread
-
createProgressThread
public static java.lang.Thread createProgressThread(long intervalMs, java.io.PrintStream stream)
Creates a thread which will write "." to the given print stream at the given interval. The created thread is not started by this method. The created thread will be a daemon thread and will halt when interrupted.- Parameters:
intervalMs
- the time interval in milliseconds between writesstream
- the print stream to write to- Returns:
- the thread
-
getCurrentMs
public static long getCurrentMs()
- Returns:
- current time in milliseconds
-
getJavaVersion
public static int getJavaVersion()
- Returns:
- the current jvm major version, 8 for java 1.8 or 11 for java 11
-
getTmpDir
public static java.lang.String getTmpDir(java.util.List<java.lang.String> tmpDirs)
- Parameters:
tmpDirs
- the list of possible temporary directories to pick from- Returns:
- a path to a temporary directory based on the user configuration
-
getWorkerDataDirectory
public static java.lang.String getWorkerDataDirectory(java.lang.String storageDir, AlluxioConfiguration conf)
- Parameters:
storageDir
- the root of a storage directory in tiered storageconf
- Alluxio's current configuration- Returns:
- the worker data folder path after each storage directory, the final path will be like
"/mnt/ramdisk/alluxioworker" for storage dir "/mnt/ramdisk" by appending
PropertyKey.WORKER_DATA_FOLDER
.
-
listToString
public static <T> java.lang.String listToString(java.util.List<T> list)
Converts a list of objects to a string.- Type Parameters:
T
- type of the objects- Parameters:
list
- list of objects- Returns:
- space-separated concatenation of the string representation returned by Object#toString of the individual objects
-
argsToString
public static <T> java.lang.String argsToString(java.lang.String separator, T... args)
Converts varargs of objects to a string.- Type Parameters:
T
- type of the objects- Parameters:
separator
- separator stringargs
- variable arguments- Returns:
- concatenation of the string representation returned by Object#toString of the individual objects
-
toStringArray
public static java.lang.String[] toStringArray(java.util.ArrayList<java.lang.String> src)
ParsesArrayList<String>
intoString[]
.- Parameters:
src
- is the ArrayList of strings- Returns:
- an array of strings
-
randomAlphaNumString
public static java.lang.String randomAlphaNumString(int length)
Generates a random alphanumeric string of the given length.- Parameters:
length
- the length- Returns:
- a random string
-
randomBytes
public static byte[] randomBytes(int length)
Generates a random byte array of the given length.- Parameters:
length
- the length- Returns:
- a random byte array
-
sleepMs
public static void sleepMs(long timeMs)
Sleeps for the given number of milliseconds.- Parameters:
timeMs
- sleep duration in milliseconds
-
sleepMs
public static void sleepMs(org.slf4j.Logger logger, long timeMs)
Sleeps for the given number of milliseconds, reporting interruptions using the given logger. Unlike Thread.sleep(), this method responds to interrupts by setting the thread interrupt status. This means that callers must check the interrupt status if they need to handle interrupts.- Parameters:
logger
- logger for reporting interruptions; no reporting is done if the logger is nulltimeMs
- sleep duration in milliseconds
-
warmUpLoop
public static void warmUpLoop()
Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.
-
createNewClassInstance
public static <T> T createNewClassInstance(java.lang.Class<T> cls, java.lang.Class<?>[] ctorClassArgs, java.lang.Object[] ctorArgs)
Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.- Type Parameters:
T
- type of the object- Parameters:
cls
- the class to createctorClassArgs
- parameters type list of the constructor to initiate, if null default constructor will be calledctorArgs
- the arguments to pass the constructor- Returns:
- new class object
- Throws:
java.lang.RuntimeException
- if the class cannot be instantiated
-
getUnixGroups
public static java.util.List<java.lang.String> getUnixGroups(java.lang.String user) throws java.io.IOException
Gets the current user's group list from Unix by running the command 'groups' NOTE. For non-existing user it will return EMPTY list. This method may return duplicate groups.- Parameters:
user
- user name- Returns:
- the groups list that the
user
belongs to. The primary group is returned first - Throws:
java.io.IOException
-
waitFor
public static void waitFor(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition) throws java.lang.InterruptedException, java.util.concurrent.TimeoutException
Waits for a condition to be satisfied.- Parameters:
description
- a description of what causes condition to evaluate to truecondition
- the condition to wait on- Throws:
java.util.concurrent.TimeoutException
- if the function times out while waiting for the condition to be truejava.lang.InterruptedException
-
waitFor
public static void waitFor(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition, WaitForOptions options) throws java.lang.InterruptedException, java.util.concurrent.TimeoutException
Waits for a condition to be satisfied.- Parameters:
description
- a description of what causes condition to evaluate to truecondition
- the condition to wait onoptions
- the options to use- Throws:
java.util.concurrent.TimeoutException
- if the function times out while waiting for the condition to be truejava.lang.InterruptedException
-
waitForResult
public static <T> T waitForResult(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,java.lang.Boolean> condition) throws java.util.concurrent.TimeoutException, java.lang.InterruptedException
Waits for the object to meet a certain condition.- Type Parameters:
T
- type of the object- Parameters:
description
- a description of what causes condition to be metobjectSupplier
- the object to check the condition forcondition
- the condition to wait on- Returns:
- the object
- Throws:
java.util.concurrent.TimeoutException
- if the function times out while waiting for the condition to be truejava.lang.InterruptedException
- if the thread was interrupted
-
waitForResult
public static <T> T waitForResult(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,java.lang.Boolean> condition, WaitForOptions options) throws java.util.concurrent.TimeoutException, java.lang.InterruptedException
Waits for the object to meet a certain condition.- Type Parameters:
T
- type of the object- Parameters:
description
- a description of what causes condition to be metobjectSupplier
- the object to check the condition forcondition
- the condition to wait onoptions
- the options to use- Returns:
- the object
- Throws:
java.util.concurrent.TimeoutException
- if the function times out while waiting for the condition to be truejava.lang.InterruptedException
- if the thread was interrupted
-
getPrimaryGroupName
public static java.lang.String getPrimaryGroupName(java.lang.String userName, AlluxioConfiguration conf) throws java.io.IOException
Gets the primary group name of a user.- Parameters:
userName
- Alluxio user nameconf
- Alluxio configuration- Returns:
- primary group name
- Throws:
java.io.IOException
-
getGroups
public static java.util.List<java.lang.String> getGroups(java.lang.String userName, AlluxioConfiguration conf) throws java.io.IOException
UsingCachedGroupMapping
to get the group list of a user.- Parameters:
userName
- Alluxio user nameconf
- Alluxio configuration- Returns:
- the group list of the user
- Throws:
java.io.IOException
-
stripSuffixIfPresent
public static java.lang.String stripSuffixIfPresent(java.lang.String key, java.lang.String suffix)
Strips the suffix if it exists. This method will leave keys without a suffix unaltered.- Parameters:
key
- the key to strip the suffix fromsuffix
- suffix to remove- Returns:
- the key with the suffix removed, or the key unaltered if the suffix is not present
-
stripPrefixIfPresent
public static java.lang.String stripPrefixIfPresent(java.lang.String key, java.lang.String prefix)
Strips the prefix from the key if it is present. For example, for input key ufs://my-bucket-name/my-key/file and prefix ufs://my-bucket-name/, the output would be my-key/file. This method will leave keys without a prefix unaltered, ie. my-key/file returns my-key/file.- Parameters:
key
- the key to stripprefix
- prefix to remove- Returns:
- the key without the prefix
-
stripLeadingAndTrailingQuotes
public static java.lang.String stripLeadingAndTrailingQuotes(java.lang.String str)
Strips the leading and trailing quotes from the given string. E.g. return 'alluxio' for input '"alluxio"'.- Parameters:
str
- The string to strip- Returns:
- The string without the leading and trailing quotes
-
getValueFromStaticMapping
@Nullable public static java.lang.String getValueFromStaticMapping(java.lang.String mapping, java.lang.String key)
Gets the value with a given key from a static key/value mapping in string format. E.g. with mapping "id1=user1;id2=user2", it returns "user1" with key "id1". It returns null if the given key does not exist in the mapping.- Parameters:
mapping
- the "key=value" mapping in string format separated by ";"key
- the key to query- Returns:
- the mapped value if the key exists, otherwise returns null
-
getRootCause
public static java.lang.Throwable getRootCause(java.lang.Throwable e)
Gets the root cause of an exception. It stops at encountering gRPC's StatusRuntimeException.- Parameters:
e
- the exception- Returns:
- the root cause
-
castToIOException
public static java.io.IOException castToIOException(java.lang.Throwable e)
Casts aThrowable
to anIOException
.- Parameters:
e
- the throwable- Returns:
- the IO exception
-
singleElementIterator
public static <T> java.util.Iterator<T> singleElementIterator(T element)
Returns an iterator that iterates on a single element.- Type Parameters:
T
- the type of the element- Parameters:
element
- the element- Returns:
- the iterator
-
invokeAll
public static <T> void invokeAll(java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs) throws java.util.concurrent.TimeoutException, java.util.concurrent.ExecutionException
Executes the given callables, waiting for them to complete (or time out). If a callable throws an exception, that exception will be re-thrown from this method.- Type Parameters:
T
- the return type of the callables- Parameters:
callables
- the callables to executetimeoutMs
- time to wait for the callables to complete, in milliseconds- Throws:
java.util.concurrent.TimeoutException
- if the callables don't complete before the timeoutjava.util.concurrent.ExecutionException
- if any of the callables throws an exception
-
invokeAll
public static <T> void invokeAll(java.util.concurrent.ExecutorService service, java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs) throws java.util.concurrent.TimeoutException, java.util.concurrent.ExecutionException
Executes the given callables, waiting for them to complete (or time out). If a callable throws an exception, that exception will be re-thrown from this method. If the current thread is interrupted, an Exception will be thrown.- Type Parameters:
T
- the return type of the callables- Parameters:
service
- the service to execute the callablescallables
- the callables to executetimeoutMs
- time to wait for the callables to complete, in milliseconds- Throws:
java.util.concurrent.TimeoutException
- if the callables don't complete before the timeoutjava.util.concurrent.ExecutionException
- if any of the callables throws an exception
-
closeAndRethrow
public static java.lang.RuntimeException closeAndRethrow(com.google.common.io.Closer closer, java.lang.Throwable t) throws java.io.IOException
Closes the Closer and re-throws the Throwable. Any exceptions thrown while closing the Closer will be added as suppressed exceptions to the Throwable. This method always throws the given Throwable, wrapping it in a RuntimeException if it's a non-IOException checked exception. Note: This method will wrap non-IOExceptions in RuntimeException. Do not use this method in methods which throw non-IOExceptions.Closer closer = new Closer(); try { Closeable c = closer.register(new Closeable()); } catch (Throwable t) { throw closeAndRethrow(closer, t); }
- Parameters:
closer
- the Closer to closet
- the Throwable to re-throw- Returns:
- this method never returns
- Throws:
java.io.IOException
-
closeAndRethrowRuntimeException
public static java.lang.RuntimeException closeAndRethrowRuntimeException(com.google.common.io.Closer closer, java.lang.Throwable t)
Similar tocloseAndRethrow(com.google.common.io.Closer, java.lang.Throwable)
but always return a RuntimeException.- Parameters:
closer
- the Closer to closet
- the Throwable to re-throw- Returns:
- this method never returns
-
unwrapResponse
public static void unwrapResponse(alluxio.proto.dataserver.Protocol.Response response) throws AlluxioStatusException
Unwraps aProtocol.Response
.- Parameters:
response
- the response- Throws:
AlluxioStatusException
-
unwrapResponseFrom
public static void unwrapResponseFrom(alluxio.proto.dataserver.Protocol.Response response, io.netty.channel.Channel channel) throws AlluxioStatusException
Unwraps aProtocol.Response
associated with a channel.- Parameters:
response
- the responsechannel
- the channel that receives this response- Throws:
AlluxioStatusException
-
isLocalHost
public static boolean isLocalHost(WorkerNetAddress address, AlluxioConfiguration conf)
- Parameters:
address
- the Alluxio worker network addressconf
- Alluxio configuration- Returns:
- true if the worker is local
-
closeChannel
public static void closeChannel(io.netty.channel.Channel channel)
Closes the netty channel from outside the netty I/O thread. NOTE: Be careful when holding any lock that can be acquired in the netty I/O thread when calling this function to avoid having deadlocks.- Parameters:
channel
- the netty channel
-
convertMsToDate
public static java.lang.String convertMsToDate(long millis, java.lang.String dateFormatPattern)
Converts a millisecond number to a formatted date String.- Parameters:
millis
- a long millisecond numberdateFormatPattern
- the date format to follow when converting. i.e. mm-dd-yyyy- Returns:
- formatted date String
-
convertMsToClockTime
public static java.lang.String convertMsToClockTime(long millis)
Converts milliseconds to clock time.- Parameters:
millis
- milliseconds- Returns:
- input encoded as clock time
-
convertToByteString
public static java.util.Map<java.lang.String,com.google.protobuf.ByteString> convertToByteString(java.util.Map<java.lang.String,byte[]> input)
- Parameters:
input
- the input map- Returns:
- a map using protobuf
ByteString
for values instead ofbyte[]
-
convertFromByteString
public static java.util.Map<java.lang.String,byte[]> convertFromByteString(java.util.Map<java.lang.String,com.google.protobuf.ByteString> input)
- Parameters:
input
- the input map- Returns:
- a map using
byte[]
for values instead of protobufByteString
-
memoize
public static <T> java.util.function.Supplier<T> memoize(java.util.function.Supplier<T> original)
Memoize implementation for java.util.function.supplier.- Type Parameters:
T
- the object type- Parameters:
original
- the original supplier- Returns:
- the supplier with memorization
-
partition
public static <T> java.util.List<java.util.List<T>> partition(java.util.List<T> list, int numLists)
Partitions a list into numLists many lists each with around list.size() / numLists elements.- Type Parameters:
T
- the object type- Parameters:
list
- the list to partitionnumLists
- number of lists to return- Returns:
- partitioned list
-
isAddressReachable
public static boolean isAddressReachable(java.lang.String hostname, int port, int timeoutMs)
Validates whether a network address is reachable.- Parameters:
hostname
- host name of the network addressport
- port of the network addresstimeoutMs
- duration to attempt connection before returning false- Returns:
- whether the network address is reachable
-
recursiveListLocalDir
public static java.util.List<java.io.File> recursiveListLocalDir(java.io.File dir)
Recursively lists a local dir and all its subdirs and return all the files.- Parameters:
dir
- the directory- Returns:
- a list of all the files
-
parseMajorVersion
public static int parseMajorVersion(java.lang.String version)
- Parameters:
version
- the version string of the JVMgi- Returns:
- the major version of the current JVM, 8 for 1.8, 11 for java 11 see https://www.oracle.com/java/technologies/javase/versioning-naming.html for reference
-
isCollection
public static boolean isCollection(java.lang.Object obj)
- Parameters:
obj
- a Java object- Returns:
- if this object is an collection
-
summarizeCollection
public static java.lang.String summarizeCollection(java.lang.Object obj)
- Parameters:
obj
- a Java object- Returns:
- a string to summarize the object if this is a collection or a map
-
isFatalError
public static boolean isFatalError(java.lang.Throwable e)
- Parameters:
e
- error- Returns:
- whether it's fatal error
-
-