org.broadinstitute.hellbender.tools.examples

Class ExampleStreamingPythonExecutor

java.lang.Object

org.broadinstitute.hellbender.cmdline.CommandLineProgram

org.broadinstitute.hellbender.engine.GATKTool

org.broadinstitute.hellbender.engine.WalkerBase

org.broadinstitute.hellbender.engine.ReadWalker

org.broadinstitute.hellbender.tools.examples.ExampleStreamingPythonExecutor

All Implemented Interfaces:

org.broadinstitute.barclay.argparser.CommandLinePluginProvider

public class ExampleStreamingPythonExecutor
extends ReadWalker

Example ReadWalker program that uses a Python streaming executor to stream summary data from a BAM input file to a Python process through an asynchronous stream writer. Reads data is accumulated in a List until a batch size threshold is reached, at which point the batch is handed off to the asynchronous stream writer, which writes the batch to the stream on a background thread. The Python process in turn just writes the data to an output file.

1. Creates a StreamingPythonExecutor.
2. Initializes an AsynchronousWriterService, which is backed by a FIFO file, to allow streaming data to the Python process in batches on a background thread
3. Writes a string of attributes for each read to a List until the batchSize threshold is reached.
4. Starts a background write of the batch, using Python to read each attribute line from the underlying FIFO, and write it back to an output file.

See https://github.com/broadinstitute/gatk/wiki/Writing-GATK-Tools-that-use-Python for more information on using Python with GATK.

Field Summary

Fields inherited from class org.broadinstitute.hellbender.engine.ReadWalker

FEATURE_CACHE_LOOKAHEAD

Fields inherited from class org.broadinstitute.hellbender.engine.GATKTool

addOutputSAMProgramRecord, addOutputVCFCommandLine, cloudIndexPrefetchBuffer, cloudPrefetchBuffer, createOutputBamIndex, createOutputBamMD5, createOutputVariantIndex, createOutputVariantMD5, disableBamIndexCaching, features, intervalArgumentCollection, lenientVCFProcessing, outputSitesOnlyVCFs, progressMeter, readArguments, referenceArguments, SECONDS_BETWEEN_PROGRESS_UPDATES_NAME, seqValidationArguments

Fields inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

GATK_CONFIG_FILE, logger, NIO_MAX_REOPENS, NIO_PROJECT_FOR_REQUESTER_PAYS, QUIET, specialArgumentsCollection, tmpDir, useJdkDeflater, useJdkInflater, VERBOSITY

Constructor Summary

Constructors

Constructor and Description
ExampleStreamingPythonExecutor()

Method Summary

Modifier and Type	Method and Description
void	apply(GATKRead read, ReferenceContext referenceContext, FeatureContext featureContext) Process an individual read (with optional contextual information).
void	closeTool() This method is called by the GATK framework at the end of the GATKTool.doWork() template method.
void	onTraversalStart() Operations performed just prior to the start of traversal.
java.lang.Object	onTraversalSuccess() On traversal success, write the remaining batch.

Methods inherited from class org.broadinstitute.hellbender.engine.ReadWalker

getDefaultReadFilters, getProgressMeterRecordLabel, onShutdown, onStartup, requiresReads, resetReadsDataSource, traverse

Methods inherited from class org.broadinstitute.hellbender.engine.WalkerBase

directlyAccessEngineFeatureManager, directlyAccessEngineReadsDataSource, directlyAccessEngineReferenceDataSource

Methods inherited from class org.broadinstitute.hellbender.engine.GATKTool

addFeatureInputsAfterInitialization, createSAMWriter, createVCFWriter, createVCFWriter, createVCFWriter, doWork, getBestAvailableSequenceDictionary, getDefaultCloudIndexPrefetchBufferSize, getDefaultCloudPrefetchBufferSize, getDefaultToolVCFHeaderLines, getDefaultVariantAnnotationGroups, getDefaultVariantAnnotations, getGenomicsDBOptions, getHeaderForFeatures, getHeaderForReads, getHeaderForSAMWriter, getMasterSequenceDictionary, getPluginDescriptors, getReferenceDictionary, getSequenceDictionaryValidationArgumentCollection, getToolName, getTransformedReadStream, getTraversalIntervals, hasFeatures, hasReads, hasReference, hasUserSuppliedIntervals, initializeProgressMeter, makePostReadFilterTransformer, makePreReadFilterTransformer, makeReadFilter, makeVariantAnnotations, requiresFeatures, requiresIntervals, requiresReference, transformTraversalIntervals, useVariantAnnotations

Methods inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

customCommandLineValidation, getCommandLine, getCommandLineParser, getDefaultHeaders, getMetricsFile, getSupportInformation, getToolkitName, getToolkitShortName, getToolStatusWarning, getUsage, getVersion, instanceMain, instanceMainPostParseArgs, isBetaFeature, isExperimentalFeature, parseArgs, printLibraryVersions, printSettings, printStartupMessage, runTool, setDefaultHeaders, warnOnToolStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExampleStreamingPythonExecutor

public ExampleStreamingPythonExecutor()

Method Detail

onTraversalStart

public void onTraversalStart()

Description copied from class: GATKTool

Operations performed just prior to the start of traversal. Should be overridden by tool authors who need to process arguments local to their tool or perform other kinds of local initialization. Default implementation does nothing.

Overrides:

onTraversalStart in class GATKTool

apply

public void apply(GATKRead read,
                  ReferenceContext referenceContext,
                  FeatureContext featureContext)

Description copied from class: ReadWalker

Process an individual read (with optional contextual information). Must be implemented by tool authors. In general, tool authors should simply stream their output from apply(), and maintain as little internal state as possible. TODO: Determine whether and to what degree the GATK engine should provide a reduce operation TODO: to complement this operation. At a minimum, we should make apply() return a value to TODO: discourage statefulness in walkers, but how this value should be handled is TBD.

Specified by:

apply in class ReadWalker

Parameters:

read - current read

referenceContext - Reference bases spanning the current read. Will be an empty, but non-null, context object if there is no backing source of reference data (in which case all queries on it will return an empty array/iterator). Can request extra bases of context around the current read's interval by invoking ReferenceContext.setWindow(int, int) on this object before calling ReferenceContext.getBases()

featureContext - Features spanning the current read. Will be an empty, but non-null, context object if there is no backing source of Feature data (in which case all queries on it will return an empty List).

onTraversalSuccess

public java.lang.Object onTraversalSuccess()

On traversal success, write the remaining batch. Post traversal work would be done here.

Overrides:

onTraversalSuccess in class GATKTool

Returns:

Success indicator.

closeTool

public void closeTool()

Description copied from class: GATKTool

This method is called by the GATK framework at the end of the GATKTool.doWork() template method. It is called regardless of whether the GATKTool.traverse() has succeeded or not. It is called after the GATKTool.onTraversalSuccess() has completed (successfully or not) but before the GATKTool.doWork() method returns. In other words, on successful runs both GATKTool.onTraversalSuccess() and GATKTool.closeTool() will be called (in this order) while on failed runs (when GATKTool.traverse() causes an exception), only GATKTool.closeTool() will be called. The default implementation does nothing. Subclasses should override this method to close any resources that must be closed regardless of the success of traversal.

Overrides:

closeTool in class GATKTool