org.apache.spark.scheduler.cluster.mesos

MesosSchedulerUtils

trait MesosSchedulerUtils extends Logging

Shared trait for implementing a Mesos Scheduler. This holds common state and helper methods and Mesos scheduler will use.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MesosSchedulerUtils
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def checkPorts(conf: SparkConf, ports: List[(Long, Long)]): Boolean

    Checks executor ports if they are within some range of the offered list of ports ranges,

    Checks executor ports if they are within some range of the offered list of ports ranges,

    conf

    the Spark Config

    ports

    the list of ports to check

    returns

    true if ports are within range false otherwise

    Attributes
    protected

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def createResource(resourceName: String, quantity: Double): Resource

    Build a Mesos resource protobuf object

    Build a Mesos resource protobuf object

    Attributes
    protected

  10. def createResource(name: String, amount: Double, role: Option[String] = None): Resource

  11. def createSchedulerDriver(masterUrl: String, scheduler: Scheduler, sparkUser: String, appName: String, conf: SparkConf, webuiUrl: Option[String] = None, checkpoint: Option[Boolean] = None, failoverTimeout: Option[Double] = None, frameworkId: Option[String] = None): SchedulerDriver

    Creates a new MesosSchedulerDriver that communicates to the Mesos master.

    Creates a new MesosSchedulerDriver that communicates to the Mesos master.

    masterUrl

    The url to connect to Mesos master

    scheduler

    the scheduler class to receive scheduler callbacks

    sparkUser

    User to impersonate with when running tasks

    appName

    The framework name to display on the Mesos UI

    conf

    Spark configuration

    webuiUrl

    The WebUI url to link from Mesos UI

    checkpoint

    Option to checkpoint tasks for failover

    failoverTimeout

    Duration Mesos master expect scheduler to reconnect on disconnect

    frameworkId

    The id of the new framework

    Attributes
    protected

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  14. def executorMemory(sc: SparkContext): Int

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    sc

    SparkContext to use to get spark.mesos.executor.memoryOverhead value

    returns

    memory requirement as (0.1 * <memoryOverhead>) or MEMORY_OVERHEAD_MINIMUM (whichever is larger)

  15. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  16. def getAttribute(attr: Attribute): (String, Set[String])

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Attributes
    protected

  17. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  18. def getRangeResource(res: List[Resource], name: String): List[(Long, Long)]

    Transforms a range resource to a list of ranges

    Transforms a range resource to a list of ranges

    res

    the mesos resource list

    name

    the name of the resource

    returns

    the list of ranges returned

    Attributes
    protected

  19. def getRejectOfferDurationForReachedMaxCores(sc: SparkContext): Long

    Attributes
    protected

  20. def getRejectOfferDurationForUnmetConstraints(sc: SparkContext): Long

    Attributes
    protected

  21. def getResource(res: List[Resource], name: String): Double

  22. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  23. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging

  24. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  25. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  26. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  27. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  28. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  29. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  31. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  32. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  34. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  35. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  36. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  37. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  38. val managedPortNames: List[String]

  39. def markErr(): Unit

    Attributes
    protected

  40. def markRegistered(): Unit

    Signal that the scheduler has registered with Mesos.

    Signal that the scheduler has registered with Mesos.

    Attributes
    protected

  41. def matchesAttributeRequirements(slaveOfferConstraints: Map[String, Set[String]], offerAttributes: Map[String, GeneratedMessage]): Boolean

    Match the requirements (if any) to the offer attributes.

    Match the requirements (if any) to the offer attributes. if attribute requirements are not specified - return true else if attribute is defined and no values are given, simple attribute presence is performed else if attribute name and value is specified, subset match is performed on slave attributes

  42. var mesosDriver: SchedulerDriver

    Attributes
    protected

  43. def mesosToTaskState(state: mesos.Protos.TaskState): TaskState.TaskState

  44. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  45. def nonZeroPortValuesFromConfig(conf: SparkConf): List[Long]

    The values of the non-zero ports to be used by the executor process.

    The values of the non-zero ports to be used by the executor process.

    conf

    the spark config to use

    returns

    the ono-zero values of the ports

  46. final def notify(): Unit

    Definition Classes
    AnyRef

  47. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  48. def parseConstraintString(constraintsVal: String): Map[String, Set[String]]

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match]] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'.

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match]] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'. The ':' implies equality (for singular values) and "is one of" for multiple values (comma separated). For example:

    parseConstraintString("os:centos7;zone:us-east-1a,us-east-1b")
// would result in
<code>
Map(
  "os" -> Set("centos7"),
  "zone":   -> Set("us-east-1a", "us-east-1b")
)

    Mesos documentation: http://mesos.apache.org/documentation/attributes-resources/ https://github.com/apache/mesos/blob/master/src/common/values.cpp https://github.com/apache/mesos/blob/master/src/common/attributes.cpp

    constraintsVal

    constaints string consisting of ';' separated key-value pairs (separated by ':')

    returns

    Map of constraints to match resources offers.

  49. def partitionPortResources(requestedPorts: List[Long], offeredResources: List[Resource]): (List[Resource], List[Resource])

    Partitions port resources.

    Partitions port resources.

    requestedPorts

    non-zero ports to assign

    offeredResources

    the resources offered

    returns

    resources left, port resources to be used.

  50. def partitionResources(resources: List[Resource], resourceName: String, amountToUse: Double): (List[Resource], List[Resource])

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    resources

    The full list of available resources

    resourceName

    The name of the resource to take from the available resources

    amountToUse

    The amount of resources to take from the available resources

    returns

    The remaining resources list and the used resources list.

  51. def setupUris(uris: String, builder: Builder, useFetcherCache: Boolean = false): Unit

  52. def startScheduler(newDriver: SchedulerDriver): Unit

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance.

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance. This driver is expected to not be running. This method returns only after the scheduler has registered with Mesos.

  53. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  54. def taskStateToMesos(state: TaskState.TaskState): mesos.Protos.TaskState

  55. def toAttributeMap(offerAttributes: List[Attribute]): Map[String, GeneratedMessage]

    Converts the attributes from the resource offer into a Map of name -> Attribute Value The attribute values are the mesos attribute types and they are

    Converts the attributes from the resource offer into a Map of name -> Attribute Value The attribute values are the mesos attribute types and they are

    offerAttributes

    the attributes offered

    returns

    Attributes
    protected

  56. def toString(): String

    Definition Classes
    AnyRef → Any

  57. def unsetFrameworkID(sc: SparkContext): Unit

    spark.

    spark.mesos.driver.frameworkId is set by the cluster dispatcher to correlate driver submissions with frameworkIDs. However, this causes issues when a driver process launches more than one framework (more than one SparkContext(, because they all try to register with the same frameworkID. To enforce that only the first driver registers with the configured framework ID, the driver calls this method after the first registration.

  58. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  59. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  60. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped