Package org.cloudsimplus.testbeds

Class ExperimentRunner<T extends Experiment>

java.lang.Object
org.cloudsimplus.testbeds.AbstractExperiment
org.cloudsimplus.testbeds.ExperimentRunner<T>
Type Parameters:
T - the type of Experiment the runner will execute
All Implemented Interfaces:
Runnable
public abstract class ExperimentRunner<T extends Experiment>
extends AbstractExperiment
A base class to run a given experiment a defined number of times and collect statistics about the execution. The runner represents a testbed compounded of a set of experiments that it runs.
Since:
CloudSim Plus 1.0
Author:
Manoel Campos da Silva Filho

  • Field Details

    • CONFIDENCE_LEVEL

      public static final double CONFIDENCE_LEVEL
      The confidence level for computing confidence interval.
      See Also:
      Constant Field Values

  • Constructor Details

    • ExperimentRunner

      protected ExperimentRunner​(long baseSeed, int simulationRuns)
      Creates an experiment runner with a given base seed that runs sequentially.
      Parameters:
      baseSeed - the seed to be used as base for each experiment seed
      simulationRuns - the number of times the experiment will be executed

    • ExperimentRunner

      protected ExperimentRunner​(long baseSeed, int simulationRuns, boolean latexTableResultsGeneration, boolean parallel)
      Creates an experiment runner with a given base seed that runs sequentially.
      Parameters:
      baseSeed - the seed to be used as base for each experiment seed
      simulationRuns - the number of times the experiment will be executed
      latexTableResultsGeneration - Enables/disables the generation of a result table in Latex format for computed metrics.
      parallel - whether experiments will run in parallel or sequentially

    • ExperimentRunner

      protected ExperimentRunner​(long baseSeed, int simulationRuns, boolean antitheticVariatesTechnique)
      Creates an experiment runner with a given base seed that runs sequentially.
      Parameters:
      baseSeed - the seed to be used as base for each experiment seed
      simulationRuns - the number of times the experiment will be executed
      antitheticVariatesTechnique - indicates if it's to be applied the antithetic variates technique.

    • ExperimentRunner

      protected ExperimentRunner​(long baseSeed, int simulationRuns, int batchesNumber, boolean antitheticVariatesTechnique)
      Creates an experiment runner with a given base seed that runs sequentially.
      Parameters:
      baseSeed - the seed to be used as base for each experiment seed
      simulationRuns - the number of times the experiment will be executed
      batchesNumber - number of simulation run batches (zero disables the batch means method)
      antitheticVariatesTechnique - indicates if it's to be applied the antithetic variates technique.

    • ExperimentRunner

      protected ExperimentRunner​(long baseSeed, int simulationRuns, int batchesNumber, boolean antitheticVariatesTechnique, boolean parallel, boolean latexTableResultsGeneration)
      Creates an experiment runner with a given base seed.
      Parameters:
      baseSeed - the seed to be used as base for each experiment seed
      simulationRuns - the number of times the experiment will be executed
      batchesNumber - number of simulation run batches (zero disables the batch means method)
      antitheticVariatesTechnique - indicates if it's to be applied the antithetic variates technique.
      parallel - whether experiments will run in parallel or sequentially. It's just actually enabled when the simulation runs is larger than 1.
      latexTableResultsGeneration - Enables/disables the generation of a result table in Latex format for computed metrics.

  • Method Details

    • batchSizeCeil

      public int batchSizeCeil()
      Returns:
      the batch size rounded by the Math.ceil(double) method.

    • isApplyBatchMeansMethod

      public boolean isApplyBatchMeansMethod()
      Checks if the "Batch Means Method" is to be applied to reduce correlation between the results for different experiment runs. That happens if the number of simulation runs and the number of batches are compatible.
      Returns:

    • computeBatchMeans

      protected List<Double> computeBatchMeans​(List<Double> samples)
      Gets an list of samples and apply the "Batch Means Method" to reduce samples correlation, if the "Batch Means Method" is to be applied.
      Parameters:
      samples - the list with samples to apply the "Batch Means Method". Samples size is defined by the getSimulationRuns().
      Returns:
      the samples list after applying the "Batch Means Method", in case the method is enabled to be applied, which will reduce the array to the number of batches defined by getBatchesNumber() (each value in the returned array will be the mean of every sample batch). Otherwise, returns the same given array

    • isApplyAntitheticVariatesTechnique

      public boolean isApplyAntitheticVariatesTechnique()
      Checks if the "Antithetic Variates Technique" is to be applied to reduce results variance.
      Returns:
      See Also:
      Antithetic variates

    • getSimulationRuns

      public int getSimulationRuns()
      Gets the number of times the experiment will be executed in order to get values such as means and standard deviations. It has to be an even number if the "Antithetic Variates Technique" is to be used.
      Returns:

    • isSingleRun

      public boolean isSingleRun()
      Checks if the experiment will run a single time or not.
      Returns:
      true if the experiment will run a single time, false if there are multiple simulation runs.

    • getBatchesNumber

      public int getBatchesNumber()
      Gets the number of batches in which the simulation runs will be divided. If this number is greater than 1, the "Batch Means Method" is used to reduce the correlation between experiment runs.
      Returns:

    • getBaseSeed

      public long getBaseSeed()
      Gets the seed to be used for the first executed experiment. The seed for each subsequent experiment is this seed plus the index of the experiment.
      Returns:

    • createRandomGen

      public <T extends StatisticalDistribution> T createRandomGen​(int experimentIndex, Function<Long,​T> randomGenCreator)
      Uses the provided Function to create a pseudo random number generator (PRNG) for a experiment run. The kind and parameters for this PRNG is defined internally by the given Function. This method calls that Function just providing the seed to be used for the current experiment run. If it is to apply the "Antithetic Variates Technique" to reduce results variance, the second half of experiments will used the seeds from the first half.
      Parameters:
      experimentIndex - index of the experiment run to create a PRNG
      randomGenCreator - a Function that receives a seed generated by the runner and returns a new instance of some PRNG
      Returns:
      the created PRNG with the seed provided by the runner
      See Also:
      UniformDistr.isApplyAntitheticVariates(), createRandomGen(int, double, double)

    • createRandomGen

      public ContinuousDistribution createRandomGen​(int experimentIndex)
      Creates a pseudo random number generator (PRNG) for a experiment run that generates uniform values between [0 and 1[. If it is to apply the "Antithetic Variates Technique" to reduce results variance, the second half of experiments will used the seeds from the first half.
      Parameters:
      experimentIndex - index of the experiment run to create a PRNG
      Returns:
      the created PRNG
      See Also:
      UniformDistr.isApplyAntitheticVariates(), createRandomGen(int, double, double)

    • createRandomGen

      public ContinuousDistribution createRandomGen​(int experimentIndex, double minInclusive, double maxExclusive)
      Creates a pseudo random number generator (PRNG) for a experiment run that generates uniform values between [min and max[. If it is to apply the "Antithetic Variates Technique" to reduce results' variance, the second half of experiments will use the seeds from the first half.
      Parameters:
      experimentIndex - index of the experiment run to create a PRNG
      minInclusive - the minimum value the generator will return (inclusive)
      maxExclusive - the maximum value the generator will return (exclusive)
      Returns:
      the created PRNG
      See Also:
      UniformDistr.isApplyAntitheticVariates(), createRandomGen(int)

    • isToReuseSeedFromFirstHalfOfExperiments

      public boolean isToReuseSeedFromFirstHalfOfExperiments​(int currentExperimentIndex)

    • halfSimulationRuns

      public int halfSimulationRuns()
      Returns:
      the half of getSimulationRuns()

    • getExperimentsExecutionTimeSecs

      public long getExperimentsExecutionTimeSecs()
      Time in seconds the experiments took to finish.
      Returns:

    • getExperimentsStartTimeSecs

      public long getExperimentsStartTimeSecs()
      Time in seconds the experiments started.
      Returns:

    • run

      public void run()
      Setups and starts the execution of all experiments sequentially or in parallel.

    • createExperimentInternal

      protected abstract T createExperimentInternal​(int i)
      Creates an experiment to be run for the i'th time.
      Parameters:
      i - a number that identifies the experiment
      Returns:
      the created experiment
      See Also:
      createExperiment(int)

    • computeAndPrintFinalResults

      protected org.apache.commons.math3.stat.descriptive.SummaryStatistics computeAndPrintFinalResults​(Map.Entry<String,​List<Double>> metricEntry)
      Computes and prints final simulation results, including mean, standard deviations and confidence intervals for a given metric computed across all simulation runs.
      Parameters:
      metricEntry - a map entry represented by the name of the metric and its list of values across multiple simulation runs
      Returns:
      the computed SummaryStatistics from the provided values for the metric

    • computeFinalStatistics

      protected final org.apache.commons.math3.stat.descriptive.SummaryStatistics computeFinalStatistics​(List<Double> values)
      Creates a SummaryStatistics object from a list of Double values, allowing computation of statistics such as mean over these values. The method also checks if the Antithetic Variates and the Batch Means techniques are enabled and then apply them over the given list of Doubles. These techniques are used for variance reduction.
      Parameters:
      values - the List of values to add to the SummaryStatistics object
      Returns:
      the SummaryStatistics object containing the double values, after applying the the techniques for variance reduction.

    • confidenceErrorMargin

      protected double confidenceErrorMargin​(org.apache.commons.math3.stat.descriptive.SummaryStatistics stats)

      Computes the confidence interval error margin for a given set of samples in order to enable finding the interval lower and upper bound around a mean value. By this way, the confidence interval can be computed as [mean + errorMargin .. mean - errorMargin].

      To reduce the confidence interval by half, one have to execute the experiments 4 more times. This is called the "Replication Method" and just works when the samples are i.i.d. (independent and identically distributed). Thus, if you have correlation between samples of each simulation run, a different method such as a bias compensation, batch means or regenerative method has to be used.

      NOTE: How to compute the error margin is a little bit confusing. The Harry Perros' book states that if less than 30 samples are collected, the t-Distribution has to be used to that purpose. However, this article Wikipedia article says that if the standard deviation of the real population is known, it has to be used the z-value from the Standard Normal Distribution. Otherwise, it has to be used the t-value from the t-Distribution to calculate the critical value for defining the error margin (also called standard error). The book "Numeric Computation and Statistical Data Analysis on the Java Platform" confirms the last statement and such approach was followed.
      Parameters:
      stats - the statistic object with the values to compute the error margin of the confidence interval
      Returns:
      the error margin to compute the lower and upper bound of the confidence interval
      See Also:
      Critical Values of the Student's t Distribution, t-Distribution, Harry Perros, "Computer Simulation Techniques: The definitive introduction!," 2009, Numeric Computation and Statistical Data Analysis on the Java Platform

    • addMetricValue

      protected final void addMetricValue​(String metricName, double value)
      Add a value to a given metric inside the metricsMap.

      This method must be called for each metric inside the experiment finish listener. The listener can be set inside the runner's createExperimentInternal(int).

      See Also:
      Experiment.setAfterExperimentFinish(Consumer)

    • getMetricValues

      protected final List<Double> getMetricValues​(String metricName)

    • computeAntitheticMeans

      protected List<Double> computeAntitheticMeans​(List<Double> samples)

      Computes the antithetic means for the given samples if the "Antithetic Variates Technique" is to be applied. These values are the mean between the first half of samples with the second half. By this way, the resulting value is an array with half of the samples length.

      NOTE: To correctly compute the antithetic values the seeds from the first half of experiments must be used for the second half.

      Parameters:
      samples - the list of samples to compute the antithetic means from
      Returns:
      the computed antithetic means from the given samples if the "Antithetic Variates Technique" is to be applied, otherwise return the same given samples list.
      See Also:
      createRandomGen(int, double, double)

    • printSimulationParameters

      protected abstract void printSimulationParameters()

    • setFirstExperimentCreated

      public void setFirstExperimentCreated​(int firstExperimentCreated)

    • getFirstExperimentCreated

      public int getFirstExperimentCreated()

    • setDescription

      public ExperimentRunner setDescription​(String description)
      Sets a description for this experiment which is shown when it starts. It's also used to generate a caption for the Latex table.
      Parameters:
      description - the description to set
      See Also:
      enableLatexTableResultsGeneration()

    • setResultsTableId

      public ExperimentRunner setResultsTableId​(String resultsTableId)
      An id used to identify the experiment results table generated in formats such as Latex for computed metrics.
      Parameters:
      resultsTableId - the name to set
      See Also:
      latexTableResultsGeneration

    • enableLatexTableResultsGeneration

      public ExperimentRunner enableLatexTableResultsGeneration()
      Enables the generation of a result table in Latex format for computed metrics.
      Returns:

    • isShowProgress

      public boolean isShowProgress()
      Checks if a progress bar is to be printed to show when each experiment run finishes. It's just printed when the number of simulations is greater than 1 and experiments are not set as verbose. It's shown by default if those conditions are met.
      Returns:

    • setShowProgress

      public ExperimentRunner<T> setShowProgress​(boolean showProgress)
      Enable or disables a progress bar to show when each experiment run finishes. It's just printed when the number of simulations is greater than 1 and experiments are not set as verbose.
      Parameters:
      showProgress - true to enable the progress bar, false to disable
      Returns:

    • getFinishedRuns

      public int getFinishedRuns()

    • setProgressBarInNewLine

      public ExperimentRunner<T> setProgressBarInNewLine​(boolean progressBarInNewLine)

    • isVerbose

      public boolean isVerbose()
      Description copied from class: AbstractExperiment
      Indicates if simulation results of the experiment have to be output.
      Overrides:
      isVerbose in class AbstractExperiment
      Returns: