org.broadinstitute.hellbender.tools.walkers.filters

Class VariantFiltration

java.lang.Object

org.broadinstitute.hellbender.cmdline.CommandLineProgram

org.broadinstitute.hellbender.engine.GATKTool

org.broadinstitute.hellbender.engine.WalkerBase

org.broadinstitute.hellbender.engine.VariantWalkerBase

org.broadinstitute.hellbender.engine.VariantWalker

org.broadinstitute.hellbender.tools.walkers.filters.VariantFiltration

All Implemented Interfaces:

org.broadinstitute.barclay.argparser.CommandLinePluginProvider

@DocumentedFeature
public final class VariantFiltration
extends VariantWalker

Filter variant calls based on INFO and/or FORMAT annotations

This tool is designed for hard-filtering variant calls based on certain criteria. Records are hard-filtered by changing the value in the FILTER field to something other than PASS. Filtered records will be preserved in the output unless their removal is requested in the command line.

Inputs

• A VCF of variant calls to filter.
• One or more filtering expressions and corresponding filter names.

Output

A filtered VCF in which passing variants are annotated as PASS and failing variants are annotated with the name(s) of the filter(s) they failed.

Usage example

   gatk VariantFiltration \
   -R reference.fasta \
   -V input.vcf.gz \
   -O output.vcf.gz \
   --filter-name "my_filter1" \
   --filter-expression "AB < 0.2" \
   --filter-name "my_filter2" \
   --filter-expression "MQ0 > 50"
 

Note

Composing filtering expressions can range from very simple to extremely complicated depending on what you're trying to do.

Compound expressions (ones that specify multiple conditions connected by &&, AND, ||, or OR, and reference multiple attributes) require special consideration. By default, variants that are missing one or more of the attributes referenced in a compound expression are treated as PASS for the entire expression, even if the variant would satisfy the filter criteria for another part of the expression. This can lead to unexpected results if any of the attributes referenced in a compound expression are present for some variants, but missing for others.

It is strongly recommended that such expressions be provided as individual arguments, each referencing a single attribute and specifying a single criteria. This ensures that all of the individual expression are applied to each variant, even if a given variant is missing values for some of the expression conditions.

As an example, multiple individual expressions provided like this:

   gatk VariantFiltration \
   -R reference.fasta \
   -V input.vcf.gz \
   -O output.vcf.gz \
   --filter-name "my_filter1" \
   --filter-expression "AB < 0.2" \
   --filter-name "my_filter2" \
   --filter-expression "MQ0 > 50"
 

are preferable to a single compound expression such as this:

    gatk VariantFiltration \
    -R reference.fasta \
    -V input.vcf.gz \
    -O output.vcf.gz \
    --filter-name "my_filter" \
    --filter-expression "AB < 0.2 || MQ0 > 50"
  

See this article about using JEXL expressions for more information.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ALLELE_SPECIFIC_LONG_NAME
boolean	applyForAllele
static java.lang.String	CLUSTER_SIZE_LONG_NAME
static java.lang.String	CLUSTER_WINDOW_SIZE_LONG_NAME
static java.lang.String	CLUSTERED_SNP_FILTER_NAME
java.lang.Integer	clusterSize Works together with the --cluster-window-size argument.
java.lang.Integer	clusterWindow Works together with the --cluster-size argument.
java.lang.Boolean	failMissingValues By default, if JEXL cannot evaluate your expression for a particular record because one of the annotations is not present, the whole expression evaluates as PASSing.
static java.lang.String	FILTER_EXPRESSION_LONG_NAME
static java.lang.String	FILTER_NAME_LONG_NAME
static java.lang.String	FILTER_NOT_IN_MASK_LONG_NAME
java.util.List<java.lang.String>	filterExpressions VariantFiltration accepts any number of JEXL expressions (so you can have two named filters by using --filter-name One --filter-expression "X < 1" --filter-name Two --filter-expression "X > 2").
java.util.List<java.lang.String>	filterNames This name is put in the FILTER field for variants that get filtered.
boolean	filterRecordsNotInMask By default, if the --mask argument is used, any variant falling in a mask will be filtered.
static java.lang.String	GENOTYPE_FILTER_EXPRESSION_LONG_NAME
static java.lang.String	GENOTYPE_FILTER_NAME_LONG_NAME
java.util.List<java.lang.String>	genotypeFilterExpressions Similar to the INFO field based expressions, but used on the FORMAT (genotype) fields instead.
java.util.List<java.lang.String>	genotypeFilterNames Similar to the INFO field based expressions, but used on the FORMAT (genotype) fields instead.
static java.lang.String	INVERT_GT_LONG_NAME
static java.lang.String	INVERT_LONG_NAME
boolean	invertFilterExpression Invert the selection criteria for --filter-expression
boolean	invertGenotypeFilterExpression Invert the selection criteria for --genotype-filter-expression
FeatureInput<htsjdk.tribble.Feature>	mask Any variant which overlaps entries from the provided mask file will be filtered.
static java.lang.String	MASK_EXTENSION_LONG_NAME
static java.lang.String	MASK_NAME_LONG_NAME
java.lang.Integer	maskExtension
java.lang.String	maskName When using the --mask argument, the mask-name will be annotated in the variant record.
static java.lang.String	MISSING_VAL_LONG_NAME
static java.lang.String	NO_CALL_GTS_LONG_NAME
GATKPath	out
boolean	setFilteredGenotypesToNocall If this argument is provided, set filtered genotypes to no-call (./.).

Fields inherited from class org.broadinstitute.hellbender.engine.VariantWalker

drivingVariantFile

Fields inherited from class org.broadinstitute.hellbender.engine.VariantWalkerBase

DEFAULT_DRIVING_VARIANTS_LOOKAHEAD_BASES, genomicsDBOptions

Fields inherited from class org.broadinstitute.hellbender.engine.GATKTool

addOutputSAMProgramRecord, addOutputVCFCommandLine, cloudIndexPrefetchBuffer, cloudPrefetchBuffer, createOutputBamIndex, createOutputBamMD5, createOutputVariantIndex, createOutputVariantMD5, disableBamIndexCaching, features, intervalArgumentCollection, lenientVCFProcessing, outputSitesOnlyVCFs, progressMeter, readArguments, referenceArguments, SECONDS_BETWEEN_PROGRESS_UPDATES_NAME, seqValidationArguments

Fields inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

GATK_CONFIG_FILE, logger, NIO_MAX_REOPENS, NIO_PROJECT_FOR_REQUESTER_PAYS, QUIET, specialArgumentsCollection, tmpDir, useJdkDeflater, useJdkInflater, VERBOSITY

Constructor Summary

Constructors

Constructor and Description
VariantFiltration()

Method Summary

Modifier and Type	Method and Description
void	apply(htsjdk.variant.variantcontext.VariantContext variant, ReadsContext readsContext, ReferenceContext ref, FeatureContext featureContext) Process an individual variant.
void	closeTool() This method is called by the GATK framework at the end of the GATKTool.doWork() template method.
void	onTraversalStart() Operations performed just prior to the start of traversal.

Methods inherited from class org.broadinstitute.hellbender.engine.VariantWalker

getDrivingVariantsFeatureInput, getHeaderForVariants, getSequenceDictionaryForDrivingVariants, getSpliteratorForDrivingVariants, initializeDrivingVariants, onShutdown, onStartup, traverse

Methods inherited from class org.broadinstitute.hellbender.engine.VariantWalkerBase

getBestAvailableSequenceDictionary, getDrivingVariantCacheLookAheadBases, getGenomicsDBOptions, getProgressMeterRecordLabel, getTransformedVariantStream, getTransformedVariantStream, makePostVariantFilterTransformer, makePreVariantFilterTransformer, makeVariantFilter, requiresFeatures

Methods inherited from class org.broadinstitute.hellbender.engine.WalkerBase

directlyAccessEngineFeatureManager, directlyAccessEngineReadsDataSource, directlyAccessEngineReferenceDataSource

Methods inherited from class org.broadinstitute.hellbender.engine.GATKTool

addFeatureInputsAfterInitialization, createSAMWriter, createVCFWriter, createVCFWriter, createVCFWriter, doWork, getDefaultCloudIndexPrefetchBufferSize, getDefaultCloudPrefetchBufferSize, getDefaultReadFilters, getDefaultToolVCFHeaderLines, getDefaultVariantAnnotationGroups, getDefaultVariantAnnotations, getHeaderForFeatures, getHeaderForReads, getHeaderForSAMWriter, getMasterSequenceDictionary, getPluginDescriptors, getReferenceDictionary, getSequenceDictionaryValidationArgumentCollection, getToolName, getTransformedReadStream, getTraversalIntervals, hasFeatures, hasReads, hasReference, hasUserSuppliedIntervals, initializeProgressMeter, makePostReadFilterTransformer, makePreReadFilterTransformer, makeReadFilter, makeVariantAnnotations, onTraversalSuccess, requiresIntervals, requiresReads, requiresReference, transformTraversalIntervals, useVariantAnnotations

Methods inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

customCommandLineValidation, getCommandLine, getCommandLineParser, getDefaultHeaders, getMetricsFile, getSupportInformation, getToolkitName, getToolkitShortName, getToolStatusWarning, getUsage, getVersion, instanceMain, instanceMainPostParseArgs, isBetaFeature, isExperimentalFeature, parseArgs, printLibraryVersions, printSettings, printStartupMessage, runTool, setDefaultHeaders, warnOnToolStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FILTER_EXPRESSION_LONG_NAME

public static final java.lang.String FILTER_EXPRESSION_LONG_NAME

See Also:

Constant Field Values

FILTER_NAME_LONG_NAME

public static final java.lang.String FILTER_NAME_LONG_NAME

See Also:

Constant Field Values

GENOTYPE_FILTER_EXPRESSION_LONG_NAME

public static final java.lang.String GENOTYPE_FILTER_EXPRESSION_LONG_NAME

See Also:

Constant Field Values

GENOTYPE_FILTER_NAME_LONG_NAME

public static final java.lang.String GENOTYPE_FILTER_NAME_LONG_NAME

See Also:

Constant Field Values

CLUSTER_SIZE_LONG_NAME

public static final java.lang.String CLUSTER_SIZE_LONG_NAME

See Also:

Constant Field Values

CLUSTER_WINDOW_SIZE_LONG_NAME

public static final java.lang.String CLUSTER_WINDOW_SIZE_LONG_NAME

See Also:

Constant Field Values

MASK_EXTENSION_LONG_NAME

public static final java.lang.String MASK_EXTENSION_LONG_NAME

See Also:

Constant Field Values

MASK_NAME_LONG_NAME

public static final java.lang.String MASK_NAME_LONG_NAME

See Also:

Constant Field Values

FILTER_NOT_IN_MASK_LONG_NAME

public static final java.lang.String FILTER_NOT_IN_MASK_LONG_NAME

See Also:

Constant Field Values

MISSING_VAL_LONG_NAME

public static final java.lang.String MISSING_VAL_LONG_NAME

See Also:

Constant Field Values

INVERT_LONG_NAME

public static final java.lang.String INVERT_LONG_NAME

See Also:

Constant Field Values

INVERT_GT_LONG_NAME

public static final java.lang.String INVERT_GT_LONG_NAME

See Also:

Constant Field Values

NO_CALL_GTS_LONG_NAME

public static final java.lang.String NO_CALL_GTS_LONG_NAME

See Also:

Constant Field Values

ALLELE_SPECIFIC_LONG_NAME

public static final java.lang.String ALLELE_SPECIFIC_LONG_NAME

See Also:

Constant Field Values

mask

@Argument(fullName="mask",
          shortName="mask",
          doc="Input mask",
          optional=true)
public FeatureInput<htsjdk.tribble.Feature> mask

Any variant which overlaps entries from the provided mask file will be filtered. If the user wants logic to be reversed, i.e. filter variants that do not overlap with provided mask, then argument --filter-not-in-mask can be used. Note that it is up to the user to adapt the name of the mask to make it clear that the reverse logic was used (e.g. if masking against Hapmap, use --mask-name=hapmap for the normal masking and --mask-name=not_hapmap for the reverse masking).

out

@Argument(doc="File to which variants should be written",
          fullName="output",
          shortName="O",
          optional=false)
public GATKPath out

filterExpressions

@Argument(fullName="filter-expression",
          shortName="filter",
          doc="One or more expressions used with INFO fields to filter",
          optional=true)
public java.util.List<java.lang.String> filterExpressions

VariantFiltration accepts any number of JEXL expressions (so you can have two named filters by using --filter-name One --filter-expression "X < 1" --filter-name Two --filter-expression "X > 2"). It is preferable to use multiple expressions, each specifying an individual filter criteria, to a single compound expression that specifies multiple filter criteria.

filterNames

@Argument(fullName="filter-name",
          doc="Names to use for the list of filters",
          optional=true)
public java.util.List<java.lang.String> filterNames

This name is put in the FILTER field for variants that get filtered. Note that there must be a 1-to-1 mapping between filter expressions and filter names.

genotypeFilterExpressions

@Argument(fullName="genotype-filter-expression",
          shortName="G-filter",
          doc="One or more expressions used with FORMAT (sample/genotype-level) fields to filter (see documentation guide for more info)",
          optional=true)
public java.util.List<java.lang.String> genotypeFilterExpressions

Similar to the INFO field based expressions, but used on the FORMAT (genotype) fields instead. VariantFiltration will add the sample-level FT tag to the FORMAT field of filtered samples (this does not affect the record's FILTER tag). One can filter normally based on most fields (e.g. "GQ < 5.0"), but the GT (genotype) field is an exception. We have put in convenience methods so that one can now filter out hets ("isHet == 1"), refs ("isHomRef == 1"), or homs ("isHomVar == 1"). Also available are expressions isCalled, isNoCall, isMixed, and isAvailable, in accordance with the methods of the Genotype object. It is preferable to use multiple expressions, each specifying an individual filter criteria, to a single compound expression that specifies multiple filter criteria.

genotypeFilterNames

@Argument(fullName="genotype-filter-name",
          shortName="G-filter-name",
          doc="Names to use for the list of sample/genotype filters (must be a 1-to-1 mapping); this name is put in the FILTER field for variants that get filtered",
          optional=true)
public java.util.List<java.lang.String> genotypeFilterNames

Similar to the INFO field based expressions, but used on the FORMAT (genotype) fields instead.

clusterSize

@Argument(fullName="cluster-size",
          shortName="cluster",
          doc="The number of SNPs which make up a cluster. Must be at least 2",
          optional=true)
public java.lang.Integer clusterSize

Works together with the --cluster-window-size argument.

clusterWindow

@Argument(fullName="cluster-window-size",
          shortName="window",
          doc="The window size (in bases) in which to evaluate clustered SNPs",
          optional=true)
public java.lang.Integer clusterWindow

Works together with the --cluster-size argument. To disable the clustered SNP filter, set this value to less than 1.

maskExtension

@Argument(fullName="mask-extension",
          doc="How many bases beyond records from a provided \'mask\' should variants be filtered",
          optional=true)
public java.lang.Integer maskExtension

maskName

@Argument(fullName="mask-name",
          doc="The text to put in the FILTER field if a \'mask\' is provided and overlaps with a variant call",
          optional=true)
public java.lang.String maskName

When using the --mask argument, the mask-name will be annotated in the variant record. Note that when using the --filter-not-in-mask argument to reverse the masking logic, it is up to the user to adapt the name of the mask to make it clear that the reverse logic was used (e.g. if masking against Hapmap, use --mask-name=hapmap for the normal masking and --mask-name=not_hapmap for the reverse masking).

filterRecordsNotInMask

@Argument(fullName="filter-not-in-mask",
          doc="Filter records NOT in given input mask.",
          optional=true)
public boolean filterRecordsNotInMask

By default, if the --mask argument is used, any variant falling in a mask will be filtered. If this argument is used, logic is reversed, and variants falling outside a given mask will be filtered. Use case is, for example, if we have an interval list or BED file with "good" sites. Note that it is up to the user to adapt the name of the mask to make it clear that the reverse logic was used (e.g. if masking against Hapmap, use --mask-name=hapmap for the normal masking and --mask-name=not_hapmap for the reverse masking).

failMissingValues

@Argument(fullName="missing-values-evaluate-as-failing",
          doc="When evaluating the JEXL expressions, missing values should be considered failing the expression",
          optional=true)
public java.lang.Boolean failMissingValues

By default, if JEXL cannot evaluate your expression for a particular record because one of the annotations is not present, the whole expression evaluates as PASSing. Use this argument to have it evaluate as failing filters instead for these cases.

invertFilterExpression

@Argument(fullName="invert-filter-expression",
          shortName="invfilter",
          doc="Invert the selection criteria for --filter-expression",
          optional=true)
public boolean invertFilterExpression

Invert the selection criteria for --filter-expression

invertGenotypeFilterExpression

@Argument(fullName="invert-genotype-filter-expression",
          shortName="invG-filter",
          doc="Invert the selection criteria for --genotype-filter-expression",
          optional=true)
public boolean invertGenotypeFilterExpression

Invert the selection criteria for --genotype-filter-expression

setFilteredGenotypesToNocall

@Argument(fullName="set-filtered-genotype-to-no-call",
          optional=true,
          doc="Set filtered genotypes to no-call")
public boolean setFilteredGenotypesToNocall

If this argument is provided, set filtered genotypes to no-call (./.).

applyForAllele

@Argument(fullName="apply-allele-specific-filters",
          optional=true,
          doc="Set mask at the allele level. This option is not compatible with clustering.")
public boolean applyForAllele

CLUSTERED_SNP_FILTER_NAME

public static final java.lang.String CLUSTERED_SNP_FILTER_NAME

See Also:

Constant Field Values

Constructor Detail

VariantFiltration

public VariantFiltration()

Method Detail

onTraversalStart

public void onTraversalStart()

Description copied from class: GATKTool

Operations performed just prior to the start of traversal. Should be overridden by tool authors who need to process arguments local to their tool or perform other kinds of local initialization. Default implementation does nothing.

Overrides:

onTraversalStart in class GATKTool

apply

public void apply(htsjdk.variant.variantcontext.VariantContext variant,
                  ReadsContext readsContext,
                  ReferenceContext ref,
                  FeatureContext featureContext)

Description copied from class: VariantWalker

Process an individual variant. Must be implemented by tool authors. In general, tool authors should simply stream their output from apply(), and maintain as little internal state as possible.

Specified by:

apply in class VariantWalker

Parameters:

variant - Current variant being processed.

readsContext - Reads overlapping the current variant. Will be an empty, but non-null, context object if there is no backing source of reads data (in which case all queries on it will return an empty array/iterator)

ref - Reference bases spanning the current variant. Will be an empty, but non-null, context object if there is no backing source of reference data (in which case all queries on it will return an empty array/iterator). Can request extra bases of context around the current variant's interval by invoking ReferenceContext.setWindow(int, int) on this object before calling ReferenceContext.getBases()

featureContext - Features spanning the current variant. Will be an empty, but non-null, context object if there is no backing source of Feature data (in which case all queries on it will return an empty List).

closeTool

public void closeTool()

Description copied from class: GATKTool

This method is called by the GATK framework at the end of the GATKTool.doWork() template method. It is called regardless of whether the GATKTool.traverse() has succeeded or not. It is called after the GATKTool.onTraversalSuccess() has completed (successfully or not) but before the GATKTool.doWork() method returns. In other words, on successful runs both GATKTool.onTraversalSuccess() and GATKTool.closeTool() will be called (in this order) while on failed runs (when GATKTool.traverse() causes an exception), only GATKTool.closeTool() will be called. The default implementation does nothing. Subclasses should override this method to close any resources that must be closed regardless of the success of traversal.

Overrides:

closeTool in class GATKTool