Package alluxio.util

Class CommonUtils


  • @ThreadSafe
    public final class CommonUtils
    extends java.lang.Object
    Common utilities shared by all components in Alluxio.
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  CommonUtils.ProcessType
      Alluxio process types.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.util.concurrent.atomic.AtomicReference<CommonUtils.ProcessType> PROCESS_TYPE
      Represents the type of Alluxio process running in this JVM.
    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static <T> java.lang.String argsToString​(java.lang.String separator, T... args)
      Converts varargs of objects to a string.
      static java.io.IOException castToIOException​(java.lang.Throwable e)
      Casts a Throwable to an IOException.
      static java.lang.RuntimeException closeAndRethrow​(com.google.common.io.Closer closer, java.lang.Throwable t)
      Closes the Closer and re-throws the Throwable.
      static java.lang.RuntimeException closeAndRethrowRuntimeException​(com.google.common.io.Closer closer, java.lang.Throwable t)
      Similar to closeAndRethrow(com.google.common.io.Closer, java.lang.Throwable) but always return a RuntimeException.
      static void closeChannel​(io.netty.channel.Channel channel)
      Closes the netty channel from outside the netty I/O thread.
      static java.util.Map<java.lang.String,​byte[]> convertFromByteString​(java.util.Map<java.lang.String,​com.google.protobuf.ByteString> input)  
      static java.lang.String convertMsToClockTime​(long millis)
      Converts milliseconds to clock time.
      static java.lang.String convertMsToDate​(long millis, java.lang.String dateFormatPattern)
      Converts a millisecond number to a formatted date String.
      static java.util.Map<java.lang.String,​com.google.protobuf.ByteString> convertToByteString​(java.util.Map<java.lang.String,​byte[]> input)  
      static <T> T createNewClassInstance​(java.lang.Class<T> cls, java.lang.Class<?>[] ctorClassArgs, java.lang.Object[] ctorArgs)
      Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.
      static java.lang.Thread createProgressThread​(long intervalMs, java.io.PrintStream stream)
      Creates a thread which will write "." to the given print stream at the given interval.
      static java.lang.Thread createProgressThread​(java.io.PrintStream stream)
      Convenience method for calling createProgressThread(long, PrintStream) with an interval of 2 seconds.
      static long getCurrentMs()  
      static java.util.List<java.lang.String> getGroups​(java.lang.String userName, AlluxioConfiguration conf)
      Using CachedGroupMapping to get the group list of a user.
      static int getJavaVersion()  
      static java.lang.String getPrimaryGroupName​(java.lang.String userName, AlluxioConfiguration conf)
      Gets the primary group name of a user.
      static java.lang.Throwable getRootCause​(java.lang.Throwable e)
      Gets the root cause of an exception.
      static java.lang.String getTmpDir​(java.util.List<java.lang.String> tmpDirs)  
      static java.util.List<java.lang.String> getUnixGroups​(java.lang.String user)
      Gets the current user's group list from Unix by running the command 'groups' NOTE.
      static java.lang.String getValueFromStaticMapping​(java.lang.String mapping, java.lang.String key)
      Gets the value with a given key from a static key/value mapping in string format.
      static java.lang.String getWorkerDataDirectory​(java.lang.String storageDir, AlluxioConfiguration conf)  
      static <T> void invokeAll​(java.util.concurrent.ExecutorService service, java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs)
      Executes the given callables, waiting for them to complete (or time out).
      static <T> void invokeAll​(java.util.List<java.util.concurrent.Callable<T>> callables, long timeoutMs)
      Executes the given callables, waiting for them to complete (or time out).
      static boolean isAddressReachable​(java.lang.String hostname, int port, int timeoutMs)
      Validates whether a network address is reachable.
      static boolean isCollection​(java.lang.Object obj)  
      static boolean isFatalError​(java.lang.Throwable e)  
      static boolean isLocalHost​(WorkerNetAddress address, AlluxioConfiguration conf)  
      static <T> java.lang.String listToString​(java.util.List<T> list)
      Converts a list of objects to a string.
      static <T> java.util.function.Supplier<T> memoize​(java.util.function.Supplier<T> original)
      Memoize implementation for java.util.function.supplier.
      static int parseMajorVersion​(java.lang.String version)  
      static <T> java.util.List<java.util.List<T>> partition​(java.util.List<T> list, int numLists)
      Partitions a list into numLists many lists each with around list.size() / numLists elements.
      static java.lang.String randomAlphaNumString​(int length)
      Generates a random alphanumeric string of the given length.
      static byte[] randomBytes​(int length)
      Generates a random byte array of the given length.
      static java.util.List<java.io.File> recursiveListLocalDir​(java.io.File dir)
      Recursively lists a local dir and all its subdirs and return all the files.
      static <T> java.util.Iterator<T> singleElementIterator​(T element)
      Returns an iterator that iterates on a single element.
      static void sleepMs​(long timeMs)
      Sleeps for the given number of milliseconds.
      static void sleepMs​(org.slf4j.Logger logger, long timeMs)
      Sleeps for the given number of milliseconds, reporting interruptions using the given logger.
      static java.lang.String stripLeadingAndTrailingQuotes​(java.lang.String str)
      Strips the leading and trailing quotes from the given string.
      static java.lang.String stripPrefixIfPresent​(java.lang.String key, java.lang.String prefix)
      Strips the prefix from the key if it is present.
      static java.lang.String stripSuffixIfPresent​(java.lang.String key, java.lang.String suffix)
      Strips the suffix if it exists.
      static java.lang.String summarizeCollection​(java.lang.Object obj)  
      static java.lang.String[] toStringArray​(java.util.ArrayList<java.lang.String> src)
      Parses ArrayList<String> into String[].
      static void unwrapResponse​(alluxio.proto.dataserver.Protocol.Response response)
      Unwraps a Protocol.Response.
      static void unwrapResponseFrom​(alluxio.proto.dataserver.Protocol.Response response, io.netty.channel.Channel channel)
      Unwraps a Protocol.Response associated with a channel.
      static void waitFor​(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition)
      Waits for a condition to be satisfied.
      static void waitFor​(java.lang.String description, java.util.function.Supplier<java.lang.Boolean> condition, WaitForOptions options)
      Waits for a condition to be satisfied.
      static <T> T waitForResult​(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,​java.lang.Boolean> condition)
      Waits for the object to meet a certain condition.
      static <T> T waitForResult​(java.lang.String description, java.util.function.Supplier<T> objectSupplier, java.util.function.Function<T,​java.lang.Boolean> condition, WaitForOptions options)
      Waits for the object to meet a certain condition.
      static void warmUpLoop()
      Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • PROCESS_TYPE

        public static final java.util.concurrent.atomic.AtomicReference<CommonUtils.ProcessType> PROCESS_TYPE
        Represents the type of Alluxio process running in this JVM. NOTE: This will only be set by main methods of Alluxio processes. It will not be set properly for tests. Avoid using this field if at all possible.
    • Method Detail

      • createProgressThread

        public static java.lang.Thread createProgressThread​(java.io.PrintStream stream)
        Convenience method for calling createProgressThread(long, PrintStream) with an interval of 2 seconds.
        Parameters:
        stream - the print stream to write to
        Returns:
        the thread
      • createProgressThread

        public static java.lang.Thread createProgressThread​(long intervalMs,
                                                            java.io.PrintStream stream)
        Creates a thread which will write "." to the given print stream at the given interval. The created thread is not started by this method. The created thread will be a daemon thread and will halt when interrupted.
        Parameters:
        intervalMs - the time interval in milliseconds between writes
        stream - the print stream to write to
        Returns:
        the thread
      • getCurrentMs

        public static long getCurrentMs()
        Returns:
        current time in milliseconds
      • getJavaVersion

        public static int getJavaVersion()
        Returns:
        the current jvm major version, 8 for java 1.8 or 11 for java 11
      • getTmpDir

        public static java.lang.String getTmpDir​(java.util.List<java.lang.String> tmpDirs)
        Parameters:
        tmpDirs - the list of possible temporary directories to pick from
        Returns:
        a path to a temporary directory based on the user configuration
      • getWorkerDataDirectory

        public static java.lang.String getWorkerDataDirectory​(java.lang.String storageDir,
                                                              AlluxioConfiguration conf)
        Parameters:
        storageDir - the root of a storage directory in tiered storage
        conf - Alluxio's current configuration
        Returns:
        the worker data folder path after each storage directory, the final path will be like "/mnt/ramdisk/alluxioworker" for storage dir "/mnt/ramdisk" by appending PropertyKey.WORKER_DATA_FOLDER.
      • listToString

        public static <T> java.lang.String listToString​(java.util.List<T> list)
        Converts a list of objects to a string.
        Type Parameters:
        T - type of the objects
        Parameters:
        list - list of objects
        Returns:
        space-separated concatenation of the string representation returned by Object#toString of the individual objects
      • argsToString

        public static <T> java.lang.String argsToString​(java.lang.String separator,
                                                        T... args)
        Converts varargs of objects to a string.
        Type Parameters:
        T - type of the objects
        Parameters:
        separator - separator string
        args - variable arguments
        Returns:
        concatenation of the string representation returned by Object#toString of the individual objects
      • toStringArray

        public static java.lang.String[] toStringArray​(java.util.ArrayList<java.lang.String> src)
        Parses ArrayList<String> into String[].
        Parameters:
        src - is the ArrayList of strings
        Returns:
        an array of strings
      • randomAlphaNumString

        public static java.lang.String randomAlphaNumString​(int length)
        Generates a random alphanumeric string of the given length.
        Parameters:
        length - the length
        Returns:
        a random string
      • randomBytes

        public static byte[] randomBytes​(int length)
        Generates a random byte array of the given length.
        Parameters:
        length - the length
        Returns:
        a random byte array
      • sleepMs

        public static void sleepMs​(long timeMs)
        Sleeps for the given number of milliseconds.
        Parameters:
        timeMs - sleep duration in milliseconds
      • sleepMs

        public static void sleepMs​(org.slf4j.Logger logger,
                                   long timeMs)
        Sleeps for the given number of milliseconds, reporting interruptions using the given logger. Unlike Thread.sleep(), this method responds to interrupts by setting the thread interrupt status. This means that callers must check the interrupt status if they need to handle interrupts.
        Parameters:
        logger - logger for reporting interruptions; no reporting is done if the logger is null
        timeMs - sleep duration in milliseconds
      • warmUpLoop

        public static void warmUpLoop()
        Common empty loop utility that serves the purpose of warming up the JVM before performance microbenchmarks.
      • createNewClassInstance

        public static <T> T createNewClassInstance​(java.lang.Class<T> cls,
                                                   java.lang.Class<?>[] ctorClassArgs,
                                                   java.lang.Object[] ctorArgs)
        Creates new instance of a class by calling a constructor that receives ctorClassArgs arguments.
        Type Parameters:
        T - type of the object
        Parameters:
        cls - the class to create
        ctorClassArgs - parameters type list of the constructor to initiate, if null default constructor will be called
        ctorArgs - the arguments to pass the constructor
        Returns:
        new class object
        Throws:
        java.lang.RuntimeException - if the class cannot be instantiated
      • getUnixGroups

        public static java.util.List<java.lang.String> getUnixGroups​(java.lang.String user)
                                                              throws java.io.IOException
        Gets the current user's group list from Unix by running the command 'groups' NOTE. For non-existing user it will return EMPTY list. This method may return duplicate groups.
        Parameters:
        user - user name
        Returns:
        the groups list that the user belongs to. The primary group is returned first
        Throws:
        java.io.IOException
      • waitFor

        public static void waitFor​(java.lang.String description,
                                   java.util.function.Supplier<java.lang.Boolean> condition)
                            throws java.lang.InterruptedException,
                                   java.util.concurrent.TimeoutException
        Waits for a condition to be satisfied.
        Parameters:
        description - a description of what causes condition to evaluate to true
        condition - the condition to wait on
        Throws:
        java.util.concurrent.TimeoutException - if the function times out while waiting for the condition to be true
        java.lang.InterruptedException
      • waitFor

        public static void waitFor​(java.lang.String description,
                                   java.util.function.Supplier<java.lang.Boolean> condition,
                                   WaitForOptions options)
                            throws java.lang.InterruptedException,
                                   java.util.concurrent.TimeoutException
        Waits for a condition to be satisfied.
        Parameters:
        description - a description of what causes condition to evaluate to true
        condition - the condition to wait on
        options - the options to use
        Throws:
        java.util.concurrent.TimeoutException - if the function times out while waiting for the condition to be true
        java.lang.InterruptedException
      • waitForResult

        public static <T> T waitForResult​(java.lang.String description,
                                          java.util.function.Supplier<T> objectSupplier,
                                          java.util.function.Function<T,​java.lang.Boolean> condition)
                                   throws java.util.concurrent.TimeoutException,
                                          java.lang.InterruptedException
        Waits for the object to meet a certain condition.
        Type Parameters:
        T - type of the object
        Parameters:
        description - a description of what causes condition to be met
        objectSupplier - the object to check the condition for
        condition - the condition to wait on
        Returns:
        the object
        Throws:
        java.util.concurrent.TimeoutException - if the function times out while waiting for the condition to be true
        java.lang.InterruptedException - if the thread was interrupted
      • waitForResult

        public static <T> T waitForResult​(java.lang.String description,
                                          java.util.function.Supplier<T> objectSupplier,
                                          java.util.function.Function<T,​java.lang.Boolean> condition,
                                          WaitForOptions options)
                                   throws java.util.concurrent.TimeoutException,
                                          java.lang.InterruptedException
        Waits for the object to meet a certain condition.
        Type Parameters:
        T - type of the object
        Parameters:
        description - a description of what causes condition to be met
        objectSupplier - the object to check the condition for
        condition - the condition to wait on
        options - the options to use
        Returns:
        the object
        Throws:
        java.util.concurrent.TimeoutException - if the function times out while waiting for the condition to be true
        java.lang.InterruptedException - if the thread was interrupted
      • getPrimaryGroupName

        public static java.lang.String getPrimaryGroupName​(java.lang.String userName,
                                                           AlluxioConfiguration conf)
                                                    throws java.io.IOException
        Gets the primary group name of a user.
        Parameters:
        userName - Alluxio user name
        conf - Alluxio configuration
        Returns:
        primary group name
        Throws:
        java.io.IOException
      • getGroups

        public static java.util.List<java.lang.String> getGroups​(java.lang.String userName,
                                                                 AlluxioConfiguration conf)
                                                          throws java.io.IOException
        Using CachedGroupMapping to get the group list of a user.
        Parameters:
        userName - Alluxio user name
        conf - Alluxio configuration
        Returns:
        the group list of the user
        Throws:
        java.io.IOException
      • stripSuffixIfPresent

        public static java.lang.String stripSuffixIfPresent​(java.lang.String key,
                                                            java.lang.String suffix)
        Strips the suffix if it exists. This method will leave keys without a suffix unaltered.
        Parameters:
        key - the key to strip the suffix from
        suffix - suffix to remove
        Returns:
        the key with the suffix removed, or the key unaltered if the suffix is not present
      • stripPrefixIfPresent

        public static java.lang.String stripPrefixIfPresent​(java.lang.String key,
                                                            java.lang.String prefix)
        Strips the prefix from the key if it is present. For example, for input key ufs://my-bucket-name/my-key/file and prefix ufs://my-bucket-name/, the output would be my-key/file. This method will leave keys without a prefix unaltered, ie. my-key/file returns my-key/file.
        Parameters:
        key - the key to strip
        prefix - prefix to remove
        Returns:
        the key without the prefix
      • stripLeadingAndTrailingQuotes

        public static java.lang.String stripLeadingAndTrailingQuotes​(java.lang.String str)
        Strips the leading and trailing quotes from the given string. E.g. return 'alluxio' for input '"alluxio"'.
        Parameters:
        str - The string to strip
        Returns:
        The string without the leading and trailing quotes
      • getValueFromStaticMapping

        @Nullable
        public static java.lang.String getValueFromStaticMapping​(java.lang.String mapping,
                                                                 java.lang.String key)
        Gets the value with a given key from a static key/value mapping in string format. E.g. with mapping "id1=user1;id2=user2", it returns "user1" with key "id1". It returns null if the given key does not exist in the mapping.
        Parameters:
        mapping - the "key=value" mapping in string format separated by ";"
        key - the key to query
        Returns:
        the mapped value if the key exists, otherwise returns null
      • getRootCause

        public static java.lang.Throwable getRootCause​(java.lang.Throwable e)
        Gets the root cause of an exception. It stops at encountering gRPC's StatusRuntimeException.
        Parameters:
        e - the exception
        Returns:
        the root cause
      • castToIOException

        public static java.io.IOException castToIOException​(java.lang.Throwable e)
        Casts a Throwable to an IOException.
        Parameters:
        e - the throwable
        Returns:
        the IO exception
      • singleElementIterator

        public static <T> java.util.Iterator<T> singleElementIterator​(T element)
        Returns an iterator that iterates on a single element.
        Type Parameters:
        T - the type of the element
        Parameters:
        element - the element
        Returns:
        the iterator
      • invokeAll

        public static <T> void invokeAll​(java.util.List<java.util.concurrent.Callable<T>> callables,
                                         long timeoutMs)
                                  throws java.util.concurrent.TimeoutException,
                                         java.util.concurrent.ExecutionException
        Executes the given callables, waiting for them to complete (or time out). If a callable throws an exception, that exception will be re-thrown from this method.
        Type Parameters:
        T - the return type of the callables
        Parameters:
        callables - the callables to execute
        timeoutMs - time to wait for the callables to complete, in milliseconds
        Throws:
        java.util.concurrent.TimeoutException - if the callables don't complete before the timeout
        java.util.concurrent.ExecutionException - if any of the callables throws an exception
      • invokeAll

        public static <T> void invokeAll​(java.util.concurrent.ExecutorService service,
                                         java.util.List<java.util.concurrent.Callable<T>> callables,
                                         long timeoutMs)
                                  throws java.util.concurrent.TimeoutException,
                                         java.util.concurrent.ExecutionException
        Executes the given callables, waiting for them to complete (or time out). If a callable throws an exception, that exception will be re-thrown from this method. If the current thread is interrupted, an Exception will be thrown.
        Type Parameters:
        T - the return type of the callables
        Parameters:
        service - the service to execute the callables
        callables - the callables to execute
        timeoutMs - time to wait for the callables to complete, in milliseconds
        Throws:
        java.util.concurrent.TimeoutException - if the callables don't complete before the timeout
        java.util.concurrent.ExecutionException - if any of the callables throws an exception
      • closeAndRethrow

        public static java.lang.RuntimeException closeAndRethrow​(com.google.common.io.Closer closer,
                                                                 java.lang.Throwable t)
                                                          throws java.io.IOException
        Closes the Closer and re-throws the Throwable. Any exceptions thrown while closing the Closer will be added as suppressed exceptions to the Throwable. This method always throws the given Throwable, wrapping it in a RuntimeException if it's a non-IOException checked exception. Note: This method will wrap non-IOExceptions in RuntimeException. Do not use this method in methods which throw non-IOExceptions.
         Closer closer = new Closer();
         try {
           Closeable c = closer.register(new Closeable());
         } catch (Throwable t) {
           throw closeAndRethrow(closer, t);
         }
         
        Parameters:
        closer - the Closer to close
        t - the Throwable to re-throw
        Returns:
        this method never returns
        Throws:
        java.io.IOException
      • closeAndRethrowRuntimeException

        public static java.lang.RuntimeException closeAndRethrowRuntimeException​(com.google.common.io.Closer closer,
                                                                                 java.lang.Throwable t)
        Similar to closeAndRethrow(com.google.common.io.Closer, java.lang.Throwable) but always return a RuntimeException.
        Parameters:
        closer - the Closer to close
        t - the Throwable to re-throw
        Returns:
        this method never returns
      • unwrapResponse

        public static void unwrapResponse​(alluxio.proto.dataserver.Protocol.Response response)
                                   throws AlluxioStatusException
        Unwraps a Protocol.Response.
        Parameters:
        response - the response
        Throws:
        AlluxioStatusException
      • unwrapResponseFrom

        public static void unwrapResponseFrom​(alluxio.proto.dataserver.Protocol.Response response,
                                              io.netty.channel.Channel channel)
                                       throws AlluxioStatusException
        Unwraps a Protocol.Response associated with a channel.
        Parameters:
        response - the response
        channel - the channel that receives this response
        Throws:
        AlluxioStatusException
      • isLocalHost

        public static boolean isLocalHost​(WorkerNetAddress address,
                                          AlluxioConfiguration conf)
        Parameters:
        address - the Alluxio worker network address
        conf - Alluxio configuration
        Returns:
        true if the worker is local
      • closeChannel

        public static void closeChannel​(io.netty.channel.Channel channel)
        Closes the netty channel from outside the netty I/O thread. NOTE: Be careful when holding any lock that can be acquired in the netty I/O thread when calling this function to avoid having deadlocks.
        Parameters:
        channel - the netty channel
      • convertMsToDate

        public static java.lang.String convertMsToDate​(long millis,
                                                       java.lang.String dateFormatPattern)
        Converts a millisecond number to a formatted date String.
        Parameters:
        millis - a long millisecond number
        dateFormatPattern - the date format to follow when converting. i.e. mm-dd-yyyy
        Returns:
        formatted date String
      • convertMsToClockTime

        public static java.lang.String convertMsToClockTime​(long millis)
        Converts milliseconds to clock time.
        Parameters:
        millis - milliseconds
        Returns:
        input encoded as clock time
      • convertToByteString

        public static java.util.Map<java.lang.String,​com.google.protobuf.ByteString> convertToByteString​(java.util.Map<java.lang.String,​byte[]> input)
        Parameters:
        input - the input map
        Returns:
        a map using protobuf ByteString for values instead of byte[]
      • convertFromByteString

        public static java.util.Map<java.lang.String,​byte[]> convertFromByteString​(java.util.Map<java.lang.String,​com.google.protobuf.ByteString> input)
        Parameters:
        input - the input map
        Returns:
        a map using byte[] for values instead of protobuf ByteString
      • memoize

        public static <T> java.util.function.Supplier<T> memoize​(java.util.function.Supplier<T> original)
        Memoize implementation for java.util.function.supplier.
        Type Parameters:
        T - the object type
        Parameters:
        original - the original supplier
        Returns:
        the supplier with memorization
      • partition

        public static <T> java.util.List<java.util.List<T>> partition​(java.util.List<T> list,
                                                                      int numLists)
        Partitions a list into numLists many lists each with around list.size() / numLists elements.
        Type Parameters:
        T - the object type
        Parameters:
        list - the list to partition
        numLists - number of lists to return
        Returns:
        partitioned list
      • isAddressReachable

        public static boolean isAddressReachable​(java.lang.String hostname,
                                                 int port,
                                                 int timeoutMs)
        Validates whether a network address is reachable.
        Parameters:
        hostname - host name of the network address
        port - port of the network address
        timeoutMs - duration to attempt connection before returning false
        Returns:
        whether the network address is reachable
      • recursiveListLocalDir

        public static java.util.List<java.io.File> recursiveListLocalDir​(java.io.File dir)
        Recursively lists a local dir and all its subdirs and return all the files.
        Parameters:
        dir - the directory
        Returns:
        a list of all the files
      • parseMajorVersion

        public static int parseMajorVersion​(java.lang.String version)
        Parameters:
        version - the version string of the JVMgi
        Returns:
        the major version of the current JVM, 8 for 1.8, 11 for java 11 see https://www.oracle.com/java/technologies/javase/versioning-naming.html for reference
      • isCollection

        public static boolean isCollection​(java.lang.Object obj)
        Parameters:
        obj - a Java object
        Returns:
        if this object is an collection
      • summarizeCollection

        public static java.lang.String summarizeCollection​(java.lang.Object obj)
        Parameters:
        obj - a Java object
        Returns:
        a string to summarize the object if this is a collection or a map
      • isFatalError

        public static boolean isFatalError​(java.lang.Throwable e)
        Parameters:
        e - error
        Returns:
        whether it's fatal error