org.broadinstitute.hellbender.engine

Class FeatureDataSource<T extends htsjdk.tribble.Feature>

java.lang.Object

org.broadinstitute.hellbender.engine.FeatureDataSource<T>

Type Parameters:

T - The type of Feature returned by this data source

All Implemented Interfaces:

java.lang.AutoCloseable, java.lang.Iterable<T>, GATKDataSource<T>

public final class FeatureDataSource<T extends htsjdk.tribble.Feature>
extends java.lang.Object
implements GATKDataSource<T>, java.lang.AutoCloseable

Enables traversals and queries over sources of Features, which are metadata associated with a location on the genome in a format supported by our file parsing framework, Tribble. Examples of Features are VCF records and hapmap records.

Two basic operations are available on this data source:

-Iteration over all Features in this data source, optionally restricted to Features overlapping a set of intervals if intervals are provided via setIntervalsForTraversal(List). Traversal by a set of intervals requires the file to have been indexed using the bundled tool IndexFeatureFile. The set of intervals provided MUST be non-overlapping and sorted in increasing order of start position.

-Targeted queries by one interval at a time. This also requires the file to have been indexed using the bundled tool IndexFeatureFile. Targeted queries by one interval at a time are unaffected by any intervals for full traversal set via setIntervalsForTraversal(List).

To improve performance in the case of targeted queries by one interval at a time, this class uses a caching scheme that is optimized for the common access pattern of multiple separate queries over intervals with gradually increasing start positions. It optimizes for this use case by pre-fetching records immediately following each interval during a query and caching them. Performance will suffer if the access pattern is random, involves queries over intervals with DECREASING start positions instead of INCREASING start positions, or involves lots of very large jumps forward on the genome or lots of contig switches. Query caching can be disabled, if desired.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_QUERY_LOOKAHEAD_BASES Default value for queryLookaheadBases, if none is specified.

Constructor Summary

Constructors

Constructor and Description
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer, GenomicsDBOptions genomicsDBOptions) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer, GenomicsDBOptions genomicsDBOptions, boolean setNameOnCodec) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer, java.nio.file.Path reference) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(FeatureInput<T> featureInput, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer, java.nio.file.Path reference, boolean setNameOnCodec) Creates a FeatureDataSource backed by the provided FeatureInput.
FeatureDataSource(java.io.File featureFile) Creates a FeatureDataSource backed by the provided File.
FeatureDataSource(java.io.File featureFile, java.lang.String name) Creates a FeatureDataSource backed by the provided File and assigns this data source the specified logical name.
FeatureDataSource(java.io.File featureFile, java.lang.String name, int queryLookaheadBases) Creates a FeatureDataSource backed by the provided File and assigns this data source the specified logical name.
FeatureDataSource(java.lang.String featurePath) Creates a FeatureDataSource backed by the provided path.
FeatureDataSource(java.lang.String featurePath, java.lang.String name, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType) Creates a FeatureDataSource backed by the resource at the provided path.
FeatureDataSource(java.lang.String featurePath, java.lang.String name, int queryLookaheadBases, java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType, int cloudPrefetchBuffer, int cloudIndexPrefetchBuffer) Creates a FeatureDataSource backed by the resource at the provided path.

Method Summary

Modifier and Type	Method and Description
void	close() Permanently close this data source, invalidating any open iteration over it, and making it invalid for future iterations and queries.
protected static htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>	getGenomicsDBFeatureReader(GATKPath path, java.io.File reference, GenomicsDBOptions genomicsDBOptions)
java.lang.Object	getHeader() Gets the header associated with this data source
java.lang.String	getName() Get the logical name of this data source.
htsjdk.samtools.SAMSequenceDictionary	getSequenceDictionary() Returns the sequence dictionary for this source of Features.
java.util.Iterator<T>	iterator() Gets an iterator over all Features in this data source, restricting traversal to Features overlapping our intervals if intervals were provided via setIntervalsForTraversal(List)
java.util.Iterator<T>	query(SimpleInterval interval) Gets an iterator over all Features in this data source that overlap the provided interval.
java.util.List<T>	queryAndPrefetch(htsjdk.samtools.util.Locatable interval) Returns a List of all Features in this data source that overlap the provided interval.
void	setIntervalsForTraversal(java.util.List<SimpleInterval> intervals) Restricts traversals of this data source via iterator() to only return Features that overlap the provided intervals.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

DEFAULT_QUERY_LOOKAHEAD_BASES

public static final int DEFAULT_QUERY_LOOKAHEAD_BASES

Default value for queryLookaheadBases, if none is specified. This is designed to be large enough so that in typical usage (ie., query intervals with gradually increasing start locations) there will be a substantial number of cache hits between cache misses, reducing the number of times we need to repopulate the cache from disk.

See Also:

Constant Field Values

Constructor Detail

FeatureDataSource

public FeatureDataSource(java.io.File featureFile)

Creates a FeatureDataSource backed by the provided File. The data source will have an automatically generated name, and will look ahead the default number of bases (DEFAULT_QUERY_LOOKAHEAD_BASES) during queries that produce cache misses.

Parameters:

featureFile - file containing Features

FeatureDataSource

public FeatureDataSource(java.lang.String featurePath)

Creates a FeatureDataSource backed by the provided path. The data source will have an automatically generated name, and will look ahead the default number of bases (DEFAULT_QUERY_LOOKAHEAD_BASES) during queries that produce cache misses.

Parameters:

featurePath - path or URI to source of Features

FeatureDataSource

public FeatureDataSource(java.io.File featureFile,
                         java.lang.String name)

Creates a FeatureDataSource backed by the provided File and assigns this data source the specified logical name. We will look ahead the default number of bases (DEFAULT_QUERY_LOOKAHEAD_BASES) during queries that produce cache misses.

Parameters:

featureFile - file containing Features

name - logical name for this data source (may be null)

FeatureDataSource

public FeatureDataSource(java.io.File featureFile,
                         java.lang.String name,
                         int queryLookaheadBases)

Creates a FeatureDataSource backed by the provided File and assigns this data source the specified logical name. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureFile - file containing Features

name - logical name for this data source (may be null)

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

FeatureDataSource

public FeatureDataSource(java.lang.String featurePath,
                         java.lang.String name,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType)

Creates a FeatureDataSource backed by the resource at the provided path.

Parameters:

featurePath - path to file or GenomicsDB url containing features

name - logical name for this data source (may be null)

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

FeatureDataSource

public FeatureDataSource(java.lang.String featurePath,
                         java.lang.String name,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer)

Creates a FeatureDataSource backed by the resource at the provided path.

Parameters:

featurePath - path to file or GenomicsDB url containing features

name - logical name for this data source (may be null)

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer,
                         java.nio.file.Path reference)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

reference - the reference genome corresponding to the data to be read

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer,
                         java.nio.file.Path reference,
                         boolean setNameOnCodec)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

reference - the reference genome corresponding to the data to be read

setNameOnCodec - If true, and if this FeatureDataSource uses a NameAwareCodec, the name of the FeatureInput will be used to set the codec's name. This exists as a mechanism to store the FeatureInput name in the source field of VariantContexts

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer,
                         GenomicsDBOptions genomicsDBOptions)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

genomicsDBOptions - options and info for reading from a GenomicsDB; may be null

FeatureDataSource

public FeatureDataSource(FeatureInput<T> featureInput,
                         int queryLookaheadBases,
                         java.lang.Class<? extends htsjdk.tribble.Feature> targetFeatureType,
                         int cloudPrefetchBuffer,
                         int cloudIndexPrefetchBuffer,
                         GenomicsDBOptions genomicsDBOptions,
                         boolean setNameOnCodec)

Creates a FeatureDataSource backed by the provided FeatureInput. We will look ahead the specified number of bases during queries that produce cache misses.

Parameters:

featureInput - a FeatureInput specifying a source of Features

queryLookaheadBases - look ahead this many bases during queries that produce cache misses

targetFeatureType - When searching for a FeatureCodec for this data source, restrict the search to codecs that produce this type of Feature. May be null, which results in an unrestricted search.

cloudPrefetchBuffer - MB size of caching/prefetching wrapper for the data, if on Google Cloud (0 to disable).

cloudIndexPrefetchBuffer - MB size of caching/prefetching wrapper for the index, if on Google Cloud (0 to disable).

genomicsDBOptions - options and info for reading from a GenomicsDB; may be null

setNameOnCodec - If true, and if this FeatureDataSource uses a NameAwareCodec, the name of the FeatureInput will be used to set the codec's name. This exists as a mechanism to store the FeatureInput name in the source field of VariantContexts

Method Detail

getGenomicsDBFeatureReader

protected static htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext> getGenomicsDBFeatureReader(GATKPath path,
                                                                                                                       java.io.File reference,
                                                                                                                       GenomicsDBOptions genomicsDBOptions)

getSequenceDictionary

public htsjdk.samtools.SAMSequenceDictionary getSequenceDictionary()

Returns the sequence dictionary for this source of Features. Uses the dictionary from the VCF header (if present) for variant inputs, otherwise attempts to create a sequence dictionary from the index file (if present). Returns null if no dictionary could be created from either the header or the index.

setIntervalsForTraversal

public void setIntervalsForTraversal(java.util.List<SimpleInterval> intervals)

Restricts traversals of this data source via iterator() to only return Features that overlap the provided intervals. Calls to query(SimpleInterval) and/or queryAndPrefetch(Locatable) are not affected by these intervals.

Intervals MUST be non-overlapping and sorted in order of increasing start position, otherwise traversal results will be incorrect.

Passing in a null or empty interval List clears the intervals for traversal, making future iterations over this data source unrestricted by intervals.

Parameters:

intervals - Our next full traversal will return only Features overlapping these intervals

iterator

public java.util.Iterator<T> iterator()

Gets an iterator over all Features in this data source, restricting traversal to Features overlapping our intervals if intervals were provided via setIntervalsForTraversal(List)

Calling this method invalidates (closes) any previous iterator obtained from this method.

Specified by:

iterator in interface java.lang.Iterable<T extends htsjdk.tribble.Feature>

Returns:

an iterator over all Features in this data source, limited to Features that overlap the intervals supplied via setIntervalsForTraversal(List) (if intervals were provided)

query

public java.util.Iterator<T> query(SimpleInterval interval)

Gets an iterator over all Features in this data source that overlap the provided interval.

This operation is not affected by intervals provided via setIntervalsForTraversal(List).

Requires the backing file to have been indexed using the IndexFeatureFile tool, and to be sorted in increasing order of start position for each contig.

Query results are cached to improve the performance of future queries during typical access patterns. See notes to the class as a whole for a description of the caching strategy.

Calling this method potentially invalidates (closes) any other open iterator obtained from this data source via a call to iterator()

Specified by:

query in interface GATKDataSource<T extends htsjdk.tribble.Feature>

Parameters:

interval - retrieve all Features overlapping this interval

Returns:

an iterator over all Features in this data source that overlap the provided interval

queryAndPrefetch

public java.util.List<T> queryAndPrefetch(htsjdk.samtools.util.Locatable interval)

Returns a List of all Features in this data source that overlap the provided interval.

This operation is not affected by intervals provided via setIntervalsForTraversal(List).

Requires the backing file to have been indexed using the IndexFeatureFile tool, and to be sorted in increasing order of start position for each contig.

Query results are cached to improve the performance of future queries during typical access patterns. See notes to the class as a whole for a description of the caching strategy.

Calling this method potentially invalidates (closes) any other open iterator obtained from this data source via a call to iterator()

Parameters:

interval - retrieve all Features overlapping this interval

Returns:

a List of all Features in this data source that overlap the provided interval

getName

public java.lang.String getName()

Get the logical name of this data source.

Returns:

the logical name of this data source

getHeader

public java.lang.Object getHeader()

Gets the header associated with this data source

Returns:

header associated with this data source as an Object

close

public void close()

Permanently close this data source, invalidating any open iteration over it, and making it invalid for future iterations and queries.

Specified by:

close in interface java.lang.AutoCloseable