org.broadinstitute.hellbender.tools.genomicsdb

Class GenomicsDBImport

java.lang.Object

org.broadinstitute.hellbender.cmdline.CommandLineProgram

org.broadinstitute.hellbender.engine.GATKTool

org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport

All Implemented Interfaces:

org.broadinstitute.barclay.argparser.CommandLinePluginProvider

@DocumentedFeature
public final class GenomicsDBImport
extends GATKTool

Import single-sample GVCFs into GenomicsDB before joint genotyping.

The GATK4 Best Practice Workflow for SNP and Indel calling uses GenomicsDBImport to merge GVCFs from multiple samples. GenomicsDBImport offers the same functionality as CombineGVCFs and initially came from the Intel-Broad Center for Genomics. The datastore transposes sample-centric variant information across genomic loci to make data more accessible to tools.

To query the contents of the GenomicsDB datastore, use SelectVariants. See Tutorial#11813 to get started.

Details on GenomicsDB are at https://github.com/GenomicsDB/GenomicsDB/wiki. In brief, GenomicsDB utilises a data storage system optimized for storing/querying sparse arrays. Genomics data is typically sparse in that each sample has few variants with respect to the entire reference genome. GenomicsDB contains specialized code for genomics applications, such as VCF parsing and INFO field annotation calculation.

Input

One or more GVCFs produced by in HaplotypeCaller with the `-ERC GVCF` or `-ERC BP_RESOLUTION` settings, containing the samples to joint-genotype.

Output

A GenomicsDB workspace

Usage examples

Provide each sample GVCF separately.

    gatk --java-options "-Xmx4g -Xms4g" GenomicsDBImport \
      -V data/gvcfs/mother.g.vcf.gz \
      -V data/gvcfs/father.g.vcf.gz \
      -V data/gvcfs/son.g.vcf.gz \
      --genomicsdb-workspace-path my_database \
      --tmp-dir=/path/to/large/tmp \
      -L 20
  

Provide sample GVCFs in a map file.

    gatk --java-options "-Xmx4g -Xms4g" \
       GenomicsDBImport \
       --genomicsdb-workspace-path my_database \
       --batch-size 50 \
       -L chr1:1000-10000 \
       --sample-name-map cohort.sample_map \
       --tmp-dir=/path/to/large/tmp \
       --reader-threads 5
  

The sample map is a tab-delimited text file with sample_name--tab--path_to_sample_vcf per line. Using a sample map saves the tool from having to download the GVCF headers in order to determine the sample names. Sample names in the sample name map file may have non-tab whitespace, but may not begin or end with whitespace.

  sample1      sample1.vcf.gz
  sample2      sample2.vcf.gz
  sample3      sample3.vcf.gz
  

Add new samples to an existing genomicsdb workspace.

    gatk --java-options "-Xmx4g -Xms4g" GenomicsDBImport \
      -V data/gvcfs/mother.g.vcf.gz \
      -V data/gvcfs/father.g.vcf.gz \
      -V data/gvcfs/son.g.vcf.gz \
      --genomicsdb-update-workspace-path my_database \
      --tmp-dir=/path/to/large/tmp \
  

In the incremental import case, no intervals are specified in the command because the tool will use the same intervals used in the initial import. Sample map is also supported for incremental import Get Picard-style interval_list from existing workspace

    gatk --java-options "-Xmx4g -Xms4g" GenomicsDBImport \
      --genomicsdb-update-workspace-path my_database \
      --output-interval-list-to-file /output/path/to/file
  

The interval_list for the specified/existing workspace will be written to /output/path/to/file. This will output a Picard-style interval_list (with a sequence dictionary header)

Caveats

• IMPORTANT: The -Xmx value the tool is run with should be less than the total amount of physical memory available by at least a few GB, as the native TileDB library requires additional memory on top of the Java memory. Failure to leave enough memory for the native code can result in confusing error messages!
• At least one interval must be provided, unless incrementally importing new samples in which case specified intervals are ignored in favor of intervals specified in the existing workspace
• Input GVCFs cannot contain multiple entries for a single genomic position
• The --genomicsdb-workspace-path must point to a non-existent or empty directory
• The --genomicsdb-update-workspace-path must point to a existing genomicsdb workspace
• Either --genomicsdb-workspace-path or genomicsdb-update-workspace-path must be specified
• GenomicsDBImport uses temporary disk storage during import. The amount of temporary disk storage required can exceed the space available, especially when specifying a large number of intervals. The command line argument `--tmp-dir` can be used to specify an alternate temporary storage location with sufficient space..
• It is recommended that users backup existing genomicsdb workspaces before adding new samples using --genomicsdb-update-workspace-path. If the tool fails during incremental import for any reason, the workspace may be in an inconsistent/corrupted state

Developer Note

To read data from GenomicsDB, use the query interface GenomicsDBFeatureReader

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BATCHSIZE_ARG_LONG_NAME
static java.lang.String	CONSOLIDATE_ARG_NAME
static java.lang.String	INCREMENTAL_WORKSPACE_ARG_LONG_NAME
static java.lang.String	INTERVAL_LIST_LONG_NAME
static int	INTERVAL_LIST_SIZE_WARNING_THRESHOLD
static java.lang.String	MAX_NUM_INTERVALS_TO_IMPORT_IN_PARALLEL
static java.lang.String	MERGE_INPUT_INTERVALS_LONG_NAME
static java.lang.String	OVERWRITE_WORKSPACE_LONG_NAME
static java.lang.String	SAMPLE_NAME_MAP_LONG_NAME
static java.lang.String	SEGMENT_SIZE_ARG_LONG_NAME
static java.lang.String	VALIDATE_SAMPLE_MAP_LONG_NAME
static java.lang.String	VCF_BUFFER_SIZE_ARG_NAME
static java.lang.String	VCF_INITIALIZER_THREADS_LONG_NAME
static java.lang.String	WORKSPACE_ARG_LONG_NAME

Fields inherited from class org.broadinstitute.hellbender.engine.GATKTool

addOutputSAMProgramRecord, addOutputVCFCommandLine, cloudIndexPrefetchBuffer, cloudPrefetchBuffer, createOutputBamIndex, createOutputBamMD5, createOutputVariantIndex, createOutputVariantMD5, disableBamIndexCaching, features, intervalArgumentCollection, lenientVCFProcessing, outputSitesOnlyVCFs, progressMeter, readArguments, referenceArguments, SECONDS_BETWEEN_PROGRESS_UPDATES_NAME, seqValidationArguments

Fields inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

GATK_CONFIG_FILE, logger, NIO_MAX_REOPENS, NIO_PROJECT_FOR_REQUESTER_PAYS, QUIET, specialArgumentsCollection, tmpDir, useJdkDeflater, useJdkInflater, VERBOSITY

Constructor Summary

Constructors

Constructor and Description
GenomicsDBImport()

Method Summary

Modifier and Type	Method and Description
htsjdk.samtools.SAMSequenceDictionary	getBestAvailableSequenceDictionary() Overriding getBestAvailableSequenceDictionary() to prefer the mergedVCFHeader's sequence directory, if present, over any other dictionaries
int	getDefaultCloudIndexPrefetchBufferSize()
int	getDefaultCloudPrefetchBufferSize()
java.lang.String	getProgressMeterRecordLabel()
protected void	initializeIntervals() Loads our intervals using the best available sequence dictionary (as returned by getBestAvailableSequenceDictionary()) to parse/verify them.
static java.util.LinkedHashMap<java.lang.String,java.net.URI>	loadSampleNameMapFile(java.nio.file.Path sampleToFileMapPath) load a tab delimited new line separated file of sample name to URI mapping: this maintains the keys in the same order that they appeared in the file this tool should only call loadSampleNameMapFileInSortedOrder(Path), this version is exposed for the benefit of FixCallSetSampleOrdering ex: Sample1\tpathToSample1.vcf\n Sample2\tpathTosample2.vcf\n ...
static java.util.SortedMap<java.lang.String,java.net.URI>	loadSampleNameMapFileInSortedOrder(java.nio.file.Path sampleToFileMapPath) load a tab delimited new line separated file of sample name to URI mapping: ex: Sample1\tpathToSample1.vcf\n Sample2\tpathTosample2.vcf\n ...
void	onShutdown() Close all data sources on shutdown.
void	onStartup() Before traversal starts, create the feature readers for all the input GVCFs, create the merged header and initialize the interval
void	onTraversalStart() Before traversal, fix configuration parameters and initialize GenomicsDB.
java.lang.Object	onTraversalSuccess() Operations performed immediately after a successful traversal (ie when no uncaught exceptions were thrown during the traversal).
protected java.util.List<SimpleInterval>	transformTraversalIntervals(java.util.List<SimpleInterval> getIntervals, htsjdk.samtools.SAMSequenceDictionary sequenceDictionary) Get the largest interval per contig that contains the intervals specified on the command line.
void	traverse() A complete traversal from start to finish.

Methods inherited from class org.broadinstitute.hellbender.engine.GATKTool

addFeatureInputsAfterInitialization, closeTool, createSAMWriter, createSAMWriter, createVCFWriter, createVCFWriter, directlyAccessEngineFeatureManager, directlyAccessEngineReadsDataSource, directlyAccessEngineReferenceDataSource, doWork, getDefaultReadFilters, getDefaultToolVCFHeaderLines, getDefaultVariantAnnotationGroups, getDefaultVariantAnnotations, getGenomicsDBOptions, getHeaderForFeatures, getHeaderForReads, getHeaderForSAMWriter, getMasterSequenceDictionary, getPluginDescriptors, getReferenceDictionary, getSequenceDictionaryValidationArgumentCollection, getToolName, getTransformedReadStream, getTraversalIntervals, hasFeatures, hasReads, hasReference, hasUserSuppliedIntervals, makePostReadFilterTransformer, makePreReadFilterTransformer, makeReadFilter, makeVariantAnnotations, requiresFeatures, requiresIntervals, requiresReads, requiresReference, useVariantAnnotations

Methods inherited from class org.broadinstitute.hellbender.cmdline.CommandLineProgram

customCommandLineValidation, getCommandLine, getCommandLineParser, getDefaultHeaders, getMetricsFile, getSupportInformation, getToolkitName, getToolkitShortName, getToolStatusWarning, getUsage, getVersion, instanceMain, instanceMainPostParseArgs, isBetaFeature, isExperimentalFeature, parseArgs, printLibraryVersions, printSettings, printStartupMessage, runTool, setDefaultHeaders, warnOnToolStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WORKSPACE_ARG_LONG_NAME

public static final java.lang.String WORKSPACE_ARG_LONG_NAME

See Also:

Constant Field Values

INCREMENTAL_WORKSPACE_ARG_LONG_NAME

public static final java.lang.String INCREMENTAL_WORKSPACE_ARG_LONG_NAME

See Also:

Constant Field Values

SEGMENT_SIZE_ARG_LONG_NAME

public static final java.lang.String SEGMENT_SIZE_ARG_LONG_NAME

See Also:

Constant Field Values

OVERWRITE_WORKSPACE_LONG_NAME

public static final java.lang.String OVERWRITE_WORKSPACE_LONG_NAME

See Also:

Constant Field Values

INTERVAL_LIST_LONG_NAME

public static final java.lang.String INTERVAL_LIST_LONG_NAME

See Also:

Constant Field Values

VCF_BUFFER_SIZE_ARG_NAME

public static final java.lang.String VCF_BUFFER_SIZE_ARG_NAME

See Also:

Constant Field Values

BATCHSIZE_ARG_LONG_NAME

public static final java.lang.String BATCHSIZE_ARG_LONG_NAME

See Also:

Constant Field Values

CONSOLIDATE_ARG_NAME

public static final java.lang.String CONSOLIDATE_ARG_NAME

See Also:

Constant Field Values

SAMPLE_NAME_MAP_LONG_NAME

public static final java.lang.String SAMPLE_NAME_MAP_LONG_NAME

See Also:

Constant Field Values

VALIDATE_SAMPLE_MAP_LONG_NAME

public static final java.lang.String VALIDATE_SAMPLE_MAP_LONG_NAME

See Also:

Constant Field Values

MERGE_INPUT_INTERVALS_LONG_NAME

public static final java.lang.String MERGE_INPUT_INTERVALS_LONG_NAME

See Also:

Constant Field Values

VCF_INITIALIZER_THREADS_LONG_NAME

public static final java.lang.String VCF_INITIALIZER_THREADS_LONG_NAME

See Also:

Constant Field Values

MAX_NUM_INTERVALS_TO_IMPORT_IN_PARALLEL

public static final java.lang.String MAX_NUM_INTERVALS_TO_IMPORT_IN_PARALLEL

See Also:

Constant Field Values

INTERVAL_LIST_SIZE_WARNING_THRESHOLD

public static final int INTERVAL_LIST_SIZE_WARNING_THRESHOLD

See Also:

Constant Field Values

Constructor Detail

GenomicsDBImport

public GenomicsDBImport()

Method Detail

transformTraversalIntervals

protected java.util.List<SimpleInterval> transformTraversalIntervals(java.util.List<SimpleInterval> getIntervals,
                                                                     htsjdk.samtools.SAMSequenceDictionary sequenceDictionary)

Get the largest interval per contig that contains the intervals specified on the command line.

Overrides:

transformTraversalIntervals in class GATKTool

Parameters:

getIntervals - intervals to be transformed

sequenceDictionary - used to validate intervals

Returns:

a list of one interval per contig spanning the input intervals after processing and validation

getDefaultCloudPrefetchBufferSize

public int getDefaultCloudPrefetchBufferSize()

Overrides:

getDefaultCloudPrefetchBufferSize in class GATKTool

Returns:

Default size in MB of the cloud prefetch buffer. May be overridden by individual tools. The default implementation returns a value (40 MB) that is suitable for tools with a small number of large cloud inputs. Tools with large numbers of cloud inputs will likely want to override to specify a smaller size. This value is maintained in the GATKConfig file.

getDefaultCloudIndexPrefetchBufferSize

public int getDefaultCloudIndexPrefetchBufferSize()

Overrides:

getDefaultCloudIndexPrefetchBufferSize in class GATKTool

Returns:

Default size in MB of the cloud index prefetch buffer. May be overridden by individual tools. A return value of -1 means to use the same value as returned by GATKTool.getDefaultCloudPrefetchBufferSize(). The default implementation returns -1. This value is maintained in the GATKConfig file.

getProgressMeterRecordLabel

public java.lang.String getProgressMeterRecordLabel()

Overrides:

getProgressMeterRecordLabel in class GATKTool

Returns:

String label to use for records in progress meter output. Defaults to ProgressMeter.DEFAULT_RECORD_LABEL, but tools may override to provide a more appropriate label (like "reads" or "regions")

onStartup

public void onStartup()

Before traversal starts, create the feature readers for all the input GVCFs, create the merged header and initialize the interval

Overrides:

onStartup in class GATKTool

loadSampleNameMapFile

public static java.util.LinkedHashMap<java.lang.String,java.net.URI> loadSampleNameMapFile(java.nio.file.Path sampleToFileMapPath)

load a tab delimited new line separated file of sample name to URI mapping: this maintains the keys in the same order that they appeared in the file this tool should only call loadSampleNameMapFileInSortedOrder(Path), this version is exposed for the benefit of FixCallSetSampleOrdering ex: Sample1\tpathToSample1.vcf\n Sample2\tpathTosample2.vcf\n ... The sample names must be unique.

Parameters:

sampleToFileMapPath - path to the mapping file

Returns:

map of sample name to corresponding file, the map will be ordered according to the order in the input file

loadSampleNameMapFileInSortedOrder

public static java.util.SortedMap<java.lang.String,java.net.URI> loadSampleNameMapFileInSortedOrder(java.nio.file.Path sampleToFileMapPath)

load a tab delimited new line separated file of sample name to URI mapping: ex: Sample1\tpathToSample1.vcf\n Sample2\tpathTosample2.vcf\n ... The sample names must be unique.

Parameters:

sampleToFileMapPath - path to the mapping file

Returns:

map of sample name to corresponding file, sorted by sample name

onTraversalStart

public void onTraversalStart()

Before traversal, fix configuration parameters and initialize GenomicsDB. Hard-coded to handle only VCF files and headers

Overrides:

onTraversalStart in class GATKTool

traverse

public void traverse()

A complete traversal from start to finish. This method will import all samples specified in the input GVCF files.

Specified by:

traverse in class GATKTool

onTraversalSuccess

public java.lang.Object onTraversalSuccess()

Description copied from class: GATKTool

Operations performed immediately after a successful traversal (ie when no uncaught exceptions were thrown during the traversal). Should be overridden by tool authors who need to close local resources, etc., after traversal. Also allows tools to return a value representing the traversal result, which is printed by the engine. Default implementation does nothing and returns null.

Overrides:

onTraversalSuccess in class GATKTool

Returns:

Object representing the traversal result, or null if a tool does not return a value

initializeIntervals

protected void initializeIntervals()

Loads our intervals using the best available sequence dictionary (as returned by getBestAvailableSequenceDictionary()) to parse/verify them. Does nothing if no intervals were specified.

onShutdown

public void onShutdown()

Description copied from class: GATKTool

Close all data sources on shutdown. Subclasses may extend (ie they must call super).

Overrides:

onShutdown in class GATKTool

getBestAvailableSequenceDictionary

public htsjdk.samtools.SAMSequenceDictionary getBestAvailableSequenceDictionary()

Overriding getBestAvailableSequenceDictionary() to prefer the mergedVCFHeader's sequence directory, if present, over any other dictionaries

Overrides:

getBestAvailableSequenceDictionary in class GATKTool

Returns:

Sequence directory from merged header, or super.getBestAvailableSequenceDictionary() if none