SparkHadoopUtil

Instance Constructors

new SparkHadoopUtil()

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def addCredentials(conf: JobConf): Unit

Add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster.
def addCurrentUserCredentials(creds: Credentials): Unit
def addSecretKeyToUserCredentials(key: String, secret: String): Unit
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val conf: Configuration
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getConfigurationFromJobContext(context: JobContext): Configuration

Using reflection to get the Configuration from JobContext/TaskAttemptContext.
Using reflection to get the Configuration from JobContext/TaskAttemptContext. If we directly call JobContext/TaskAttemptContext.getConfiguration, it will generate different byte codes for Hadoop 1.+ and Hadoop 2.+ because JobContext/TaskAttemptContext is class in Hadoop 1.+ while it's interface in Hadoop 2.+.
def getCurrentUserCredentials(): Credentials
def getSecretKeyFromUserCredentials(key: String): Array[Byte]
def getTaskAttemptIDFromTaskAttemptContext(context: TaskAttemptContext): TaskAttemptID

Using reflection to call getTaskAttemptID from TaskAttemptContext.
Using reflection to call getTaskAttemptID from TaskAttemptContext. If we directly call TaskAttemptContext.getTaskAttemptID, it will generate different byte codes for Hadoop 1.+ and Hadoop 2.+ because TaskAttemptContext is class in Hadoop 1.+ while it's interface in Hadoop 2.+.
def getTimeFromNowToRenewal(sparkConf: SparkConf, fraction: Double, credentials: Credentials): Long

How much time is remaining (in millis) from now to (fraction * renewal time for the token that is valid the latest)? This will return -ve (or 0) value if the fraction of validity has already expired.
def globPath(pattern: Path): Seq[Path]
def globPathIfNecessary(pattern: Path): Seq[Path]
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def isYarnMode(): Boolean
def listFilesSorted(remoteFs: FileSystem, dir: Path, prefix: String, exclusionSuffix: String): Array[FileStatus]

Lists all the files in a directory with the specified prefix, and does not end with the given suffix.
Lists all the files in a directory with the specified prefix, and does not end with the given suffix. The returned {{FileStatus}} instances are sorted by the modification times of the respective files.
def listLeafDirStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]
def listLeafDirStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]
def listLeafStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

Get FileStatus objects for all leaf children (files) under the given base path.
Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.
def listLeafStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

Get FileStatus objects for all leaf children (files) under the given base path.
Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def loginUserFromKeytab(principalName: String, keytabFilename: String): Unit
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def newConfiguration(conf: SparkConf): Configuration

Return an appropriate (subclass) of Configuration.
Return an appropriate (subclass) of Configuration. Creating config can initializes some Hadoop subsystems.
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def runAsSparkUser(func: () ⇒ Unit): Unit

Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.
Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.
IMPORTANT NOTE: If this function is going to be called repeated in the same process you need to look https://issues.apache.org/jira/browse/HDFS-3545 and possibly do a FileSystem.closeAllForUGI in order to avoid leaking Filesystems
def substituteHadoopVariables(text: String, hadoopConf: Configuration): String

Substitute variables by looking them up in Hadoop configs.
Substitute variables by looking them up in Hadoop configs. Only variables that match the ${hadoopconf- .. } pattern are substituted.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Deprecated Value Members

def newConfiguration(): Configuration

Annotations
@deprecated
Deprecated
(Since version 1.2.0) use newConfiguration with SparkConf argument

Related Docs: object SparkHadoopUtil | package deploy

class SparkHadoopUtil extends Logging

Instance Constructors

new SparkHadoopUtil()

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def addCredentials(conf: JobConf): Unit

def addCurrentUserCredentials(creds: Credentials): Unit

def addSecretKeyToUserCredentials(key: String, secret: String): Unit

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val conf: Configuration

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def getConfigurationFromJobContext(context: JobContext): Configuration

def getCurrentUserCredentials(): Credentials

def getSecretKeyFromUserCredentials(key: String): Array[Byte]

def getTaskAttemptIDFromTaskAttemptContext(context: TaskAttemptContext): TaskAttemptID

def getTimeFromNowToRenewal(sparkConf: SparkConf, fraction: Double, credentials: Credentials): Long

def globPath(pattern: Path): Seq[Path]

def globPathIfNecessary(pattern: Path): Seq[Path]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def isYarnMode(): Boolean

def listFilesSorted(remoteFs: FileSystem, dir: Path, prefix: String, exclusionSuffix: String): Array[FileStatus]

def listLeafDirStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

def listLeafDirStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

def listLeafStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

def listLeafStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def loginUserFromKeytab(principalName: String, keytabFilename: String): Unit

final def ne(arg0: AnyRef): Boolean

def newConfiguration(conf: SparkConf): Configuration

final def notify(): Unit

final def notifyAll(): Unit

def runAsSparkUser(func: () ⇒ Unit): Unit

def substituteHadoopVariables(text: String, hadoopConf: Configuration): String

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Deprecated Value Members

def newConfiguration(): Configuration

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped