com.nicta.scoobi.testing

UploadedLibJars

trait UploadedLibJars extends SpecificationStructure with HadoopExamples with ScoobiUserArgs

This trait can be mixed in a Specification to automatically add a setup step uploading the library jars to the cluster

Linear Supertypes
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. UploadedLibJars
  2. HadoopExamples
  3. HadoopLogFactoryInitialisation
  4. CommandLineScoobiUserArgs
  5. CommandLineArguments
  6. DelayedInit
  7. Hadoop
  8. LibJars
  9. Cluster
  10. LocalHadoop
  11. InMemoryHadoop
  12. ScoobiUserArgs
  13. ScoobiArgs
  14. SpecificationStructure
  15. AnyRef
  16. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class ClusterHadoopContext extends HadoopContext

    Context for running examples on the cluster

  2. trait HadoopContext extends Fixture[ScoobiConfiguration]

    trait for creating contexts having ScoobiConfigurations

  3. class InMemoryHadoopContext extends HadoopContext

    Context for running examples in memory

  4. class LocalHadoopContext extends HadoopContext

    Context for running examples locally

  5. class SkippedHadoopContext extends HadoopContext

    Context for showing that an execution is skipped

Abstract Value Members

  1. abstract def fs: String

    returns

    the filesystem address

    Definition Classes
    Cluster
  2. abstract def is: Fragments

    Definition Classes
    SpecificationStructure
  3. abstract def jobTracker: String

    returns

    the jobtracker address

    Definition Classes
    Cluster

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. lazy val arguments: Arguments

    Definition Classes
    CommandLineArguments
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def categories: String

    returns

    the categories to show when logging, as a regular expression

    Definition Classes
    ScoobiUserArgsScoobiArgs
  9. def chain(contexts: Seq[HadoopContext]): HadoopContext { val configuration: com.nicta.scoobi.core.ScoobiConfiguration }

    returns

    a context chaining a sequence of contexts

    Definition Classes
    HadoopExamples
  10. def classDirs: Seq[String]

    returns

    the classes directories to include on a job classpath

    Definition Classes
    Hadoop
  11. def cleanup(c: ScoobiConfiguration): Unit

    cleanup temporary files after job execution

    cleanup temporary files after job execution

    Definition Classes
    HadoopExamples
  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  13. def cluster: HadoopContext

    context for cluster execution

    context for cluster execution

    Definition Classes
    HadoopExamples
  14. def configureArguments(implicit configuration: ScoobiConfiguration): Unit

    set command-line arguments on the configuration object

    set command-line arguments on the configuration object

    Attributes
    protected
    Definition Classes
    InMemoryHadoop
  15. def configureForCluster(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with cluster setup

    Definition Classes
    Hadoop
  16. def configureForInMemory(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with memory setup

    Definition Classes
    InMemoryHadoop
  17. def configureForLocal(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with local setup

    Definition Classes
    LocalHadoop
  18. def configureJars(implicit configuration: ScoobiConfiguration): Configuration

    returns

    a configuration where the appropriate properties are set-up for uploaded jars: distributed files + classpath

    Definition Classes
    LibJars
  19. def context: Fixture[ScoobiConfiguration]

    define the context to use: local, cluster, localThenCluster

    define the context to use: local, cluster, localThenCluster

    Definition Classes
    HadoopExamples
  20. def contexts: Seq[HadoopContext]

    all contexts to run

    all contexts to run

    Definition Classes
    HadoopExamples
  21. def delayedInit(x: ⇒ Unit): Unit

    Definition Classes
    CommandLineArguments → DelayedInit
  22. def deleteJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    delete the remote jars currently on the cluster

    Definition Classes
    LibJars
  23. def deleteLibJars: Boolean

    returns

    true if the libjars must be deleted before the Scoobi job runs

    Definition Classes
    ScoobiUserArgsScoobiArgs
  24. def displayTime(prefix: String): (SimpleTimer) ⇒ Unit

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    returns

    a function to display execution times. The default uses log messages

    Definition Classes
    HadoopExamplesInMemoryHadoop
  25. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  27. def executeInMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    InMemoryHadoop
  28. def executeOnCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    Definition Classes
    Hadoop
  29. def executeOnLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    LocalHadoop
  30. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  31. implicit def fixtureContext: Fixture[ScoobiConfiguration]

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    Attributes
    protected
    Definition Classes
    HadoopExamples
  32. lazy val fss: FileSystems

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  33. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  34. def hadoopArgs: Array[String]

    Definition Classes
    InMemoryHadoop
  35. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  36. def identification: SpecIdentification

    Definition Classes
    SpecificationStructure
  37. implicit def identifySpecificationStructure(s: SpecificationStructure): SpecIdentification

    Definition Classes
    SpecificationStructure
  38. def inMemory[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  39. def inMemory: HadoopContext

    context for in memory execution

    context for in memory execution

    Definition Classes
    HadoopExamples
  40. def inMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code in memory, using a collection backend, possibly showing execution times

    execute some code in memory, using a collection backend, possibly showing execution times

    Definition Classes
    InMemoryHadoop
  41. def includeLibJars: Boolean

    returns

    true if you want to include the library jars in the jar that is sent to the cluster for each job

    Definition Classes
    Hadoop
  42. def isCluster: Boolean

    Definition Classes
    ScoobiUserArgs
  43. def isClusterOnly: Boolean

    returns

    true if the cluster argument is specified and the local argument is not

    Definition Classes
    ScoobiUserArgs
  44. def isInMemory: Boolean

    tests are always in memory by default, unless !inmemory is passed

    tests are always in memory by default, unless !inmemory is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  45. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  46. def isLocal: Boolean

    tests are always local by default, unless !local is passed

    tests are always local by default, unless !local is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  47. def isLocalOnly: Boolean

    alias for locally

    alias for locally

    Definition Classes
    ScoobiUserArgs
  48. def jars(implicit configuration: ScoobiConfiguration): Seq[URL]

    returns

    the list of library jars to upload

    Definition Classes
    LibJars
  49. def keepFiles: Boolean

    returns

    false if temporary files and working directory must be cleaned-up after job execution

    Definition Classes
    ScoobiUserArgsScoobiArgs
  50. def level: Level

    returns

    the log level to use when logging

    Definition Classes
    ScoobiUserArgsScoobiArgs
  51. def libjarsDirectory(implicit configuration: ScoobiConfiguration): String

    returns

    the path of the directory to use when loading jars to the filesystem.

    Definition Classes
    LibJars
  52. def local: HadoopContext

    context for local execution

    context for local execution

    Definition Classes
    HadoopExamples
  53. def locally[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  54. def locally: Boolean

    returns

    true if the Scoobi job must be run locally

    Definition Classes
    ScoobiUserArgsScoobiArgs
  55. def map(fs: ⇒ Fragments): Fragments

    add a first step to upload the library jars before doing anything else

    add a first step to upload the library jars before doing anything else

    Definition Classes
    UploadedLibJars → SpecificationStructure
  56. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  57. def noLibJars: Boolean

    returns

    false if libjars are used

    Definition Classes
    ScoobiUserArgsScoobiArgs
  58. final def notify(): Unit

    Definition Classes
    AnyRef
  59. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  60. def onCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, possibly showing the execution time

    execute some code on the cluster, possibly showing the execution time

    Definition Classes
    Hadoop
  61. def onHadoop[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    Definition Classes
    Hadoop
  62. def onLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally, possibly showing execution times

    execute some code locally, possibly showing execution times

    Definition Classes
    LocalHadoop
  63. def quiet: Boolean

    returns

    true to suppress log messages

    Definition Classes
    ScoobiUserArgsScoobiArgs
  64. def remotely[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body on the cluster

    execute an example body on the cluster

    Definition Classes
    HadoopExamples
  65. def runInMemory[T](t: ⇒ T): T

    returns

    the result of the in-memory run

    Definition Classes
    InMemoryHadoop
  66. def runOnCluster[T](t: ⇒ T): T

    returns

    the cluster evaluation of t

    Definition Classes
    Hadoop
  67. def runOnLocal[T](t: ⇒ T): T

    returns

    the result of the local run

    Definition Classes
    LocalHadoop
  68. lazy val scoobiArgs: Seq[String]

    the scoobi arguments passed on the command line

    the scoobi arguments passed on the command line

    Definition Classes
    CommandLineScoobiUserArgsScoobiUserArgs
  69. def set(args: Arguments): Unit

    Definition Classes
    CommandLineArguments
  70. def setLogFactory(name: String = classOf[HadoopLogFactory].getName): Unit

    Static setup to use a testing log factory

    Static setup to use a testing log factory

    Definition Classes
    InMemoryHadoop
  71. def showComputationGraph: Boolean

    returns

    true if the debug logs must show the computation graph

    Definition Classes
    ScoobiUserArgsScoobiArgs
  72. def showTime[T](t: ⇒ T)(display: (SimpleTimer) ⇒ Unit): T

    measure the time taken by some executed code and display the time with a specific display function

    measure the time taken by some executed code and display the time with a specific display function

    Definition Classes
    InMemoryHadoop
  73. def showTimes: Boolean

    returns

    true to display execution times for each job

    Definition Classes
    ScoobiUserArgsScoobiArgs
  74. def skippedContext(name: String): HadoopContext

    context for showing a skipped execution

    context for showing a skipped execution

    Definition Classes
    HadoopExamples
  75. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  76. lazy val sysProps: SystemProperties

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  77. def toString(): String

    Definition Classes
    AnyRef → Any
  78. def upload: Boolean

    this variable controls if the upload must be done at all

    this variable controls if the upload must be done at all

    Definition Classes
    LibJars
  79. def uploadLibJarsFiles(deleteLibJarsFirst: Boolean = false)(implicit configuration: ScoobiConfiguration): Any

    upload the jars which don't exist yet in the library directory on the cluster

    upload the jars which don't exist yet in the library directory on the cluster

    Definition Classes
    LibJars
  80. def uploadStep: Step

    create a Step to upload the jars on the cluster

  81. def uploadedJars(implicit configuration: ScoobiConfiguration): Seq[Path]

    returns

    the remote jars currently on the cluster

    Definition Classes
    LibJars
  82. def useHadoopConfDir: Boolean

    returns

    true if cluster configuration must be loaded from Hadoop's configuration directory

    Definition Classes
    ScoobiUserArgsScoobiArgs
  83. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  84. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  85. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  86. def withTimer[T](t: ⇒ T): (T, SimpleTimer)

    returns

    the time for the execution of a piece of code

    Definition Classes
    InMemoryHadoop

Inherited from HadoopExamples

Inherited from CommandLineScoobiUserArgs

Inherited from CommandLineArguments

Inherited from DelayedInit

Inherited from Hadoop

Inherited from LibJars

Inherited from Cluster

Inherited from LocalHadoop

Inherited from InMemoryHadoop

Inherited from ScoobiUserArgs

Inherited from ScoobiArgs

Inherited from SpecificationStructure

Inherited from AnyRef

Inherited from Any

Ungrouped