PositionBasedDownsampleSam (Picard)

Summary

Class to downsample a SAM/BAM file based on the position of the read in a flowcell. As with DownsampleSam, all the reads with the same queryname are either kept or dropped as a unit.

Details

The downsampling is _not_ random (and there is no random seed). It is deterministically determined by the position of each read within its tile. Specifically, it draws an ellipse that covers a FRACTION of the total tile's area and of all the edges of the tile. It uses this area to determine whether to keep or drop the record. Since reads with the same name have the same position (mates, secondary and supplemental alignments), the decision will be the same for all of them. The main concern of this downsampling method is that due to "optical duplicates" downsampling randomly can create a result that has a different optical duplicate rate, and therefore a different estimated library size (when running MarkDuplicates). This method keeps (physically) close read together, so that (except for reads near the boundary of the circle) optical duplicates are kept or dropped as a group. By default the program expects the read names to have 5 or 7 fields separated by colons (:) and it takes the last two to indicate the x and y coordinates of the reads within the tile whence it was sequenced. See DEFAULT_READ_NAME_REGEX for more detail. The program traverses the INPUT twice: first to find out the size of each of the tiles, and next to perform the downsampling. Downsampling invalidates the duplicate flag because duplicate reads before downsampling may not all remain duplicated after downsampling. Thus, the default setting also removes the duplicate information. Example java -jar picard.jar PositionBasedDownsampleSam \ I=input.bam \ O=downsampled.bam \ FRACTION=0.1 Caveats Note 1: This method is technology and read-name dependent. If the read-names do not have coordinate information embedded in them, or if your BAM contains reads from multiple technologies (flowcell versions, sequencing machines) this will not work properly. It has been designed to work with Illumina technology and reads-names. Consider modifying {@link #READ_NAME_REGEX} in other cases. Note 2: The code has been designed to simulate, as accurately as possible, sequencing less, not for getting an exact downsampled fraction (Use {@link DownsampleSam} for that.) In particular, since the reads may be distributed non-evenly within the lanes/tiles, the resulting downsampling percentage will not be accurately determined by the input argument FRACTION. Note 3:Consider running {@link MarkDuplicates} after downsampling in order to "expose" the duplicates whose representative has been downsampled away. Note 4:The downsampling assumes a uniform distribution of reads in the flowcell. Input already downsampled with PositionBasedDownsampleSam violates this assumption. To guard against such input, PositionBasedDownsampleSam always places a PG record in the header of its output, and aborts whenever it finds such a PG record in its input.

Category Read Data Manipulation

Overview

Summary

Class to downsample a SAM/BAM file based on the position of the read in a flowcell. As with DownsampleSam, all the reads with the same queryname are either kept or dropped as a unit.

Details

The downsampling is not random (and there is no random seed). It is deterministically determined by the position of each read within its tile. Specifically, it draws an ellipse that covers a #FRACTION of the total tile's area and of all the edges of the tile. It uses this area to determine whether to keep or drop the record. Since reads with the same name have the same position (mates, secondary and supplemental alignments), the decision will be the same for all of them. The main concern of this downsampling method is that due to "optical duplicates" downsampling randomly can create a result that has a different optical duplicate rate, and therefore a different estimated library size (when running MarkDuplicates). This method keeps (physically) close read together, so that (except for reads near the boundary of the circle) optical duplicates are kept or dropped as a group. By default the program expects the read names to have 5 or 7 fields separated by colons (:), and it takes the last two to indicate the x and y coordinates of the reads within the tile whence it was sequenced. See ReadNameParser#DEFAULT_READ_NAME_REGEX for more detail. The program traverses the #INPUT twice: first to find out the size of each of the tiles, and next to perform the downsampling. Downsampling invalidates the duplicate flag because duplicate reads before downsampling may not all remain duplicated after downsampling. Thus, the default setting also removes the duplicate information.

Example

 java -jar picard.jar PositionBasedDownsampleSam \
       I=input.bam \
       O=downsampled.bam \
       FRACTION=0.1

Caveats

This method is technology and read-name dependent. If the read-names do not have coordinate information embedded in them, or if your BAM contains reads from multiple technologies (flowcell versions, sequencing machines). this will not work properly. It has been designed to work with Illumina technology and reads-names. Consider modifying #READ_NAME_REGEX in other cases.
The code has been designed to simulate, as accurately as possible, sequencing less, not for getting an exact downsampled fraction (Use DownsampleSam for that.) In particular, since the reads may be distributed non-evenly within the lanes/tiles, the resulting downsampling percentage will not be accurately determined by the input argument #FRACTION.
Consider running MarkDuplicates after downsampling in order to "expose" the duplicates whose representative has been downsampled away.
The downsampling assumes a uniform distribution of reads in the flowcell. Input already downsampled with PositionBasedDownsampleSam violates this assumption. To guard against such input, PositionBasedDownsampleSam always places a PG record in the header of its output, and aborts whenever it finds such a PG record in its input.

PositionBasedDownsampleSam (Picard) specific arguments

This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.

Argument name(s)	Default value	Summary
Required Arguments
--FRACTION -F	null	The (approximate) fraction of reads to be kept, between 0 and 1.
--INPUT -I	null	The input SAM or BAM file to downsample.
--OUTPUT -O	null	The output, downsampled, SAM or BAM file.
Optional Tool Arguments
--ALLOW_MULTIPLE_DOWNSAMPLING_DESPITE_WARNINGS	false	Allow downsampling again despite this being a bad idea with possibly unexpected results.
--arguments_file	[]	read one or more arguments files and add them to the command line
--help -h	false	display the help message
--READ_NAME_REGEX		Use these regular expressions to parse read names in the input SAM file. Read names are parsed to extract three variables: tile/region, x coordinate and y coordinate. The x and y coordinates are used to determine the downsample decision. Set this option to null to disable optical duplicate detection, e.g. for RNA-seq The regular expression should contain three capture groups for the three variables, in order. It must match the entire read name. Note that if the default regex is specified, a regex match is not actually done, but instead the read name is split on colons (:). For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.
--REMOVE_DUPLICATE_INFORMATION	true	Determines whether the duplicate tag should be reset since the downsampling requires re-marking duplicates.
--STOP_AFTER	null	Stop after processing N reads, mainly for debugging.
--version	false	display the version number for this tool
Optional Common Arguments
--COMPRESSION_LEVEL	5	Compression level for all compressed files created (e.g. BAM and VCF).
--CREATE_INDEX	false	Whether to create a BAM index when writing a coordinate-sorted BAM file.
--CREATE_MD5_FILE	false	Whether to create an MD5 digest for any BAM or FASTQ files created.
--GA4GH_CLIENT_SECRETS	client_secrets.json	Google Genomics API client_secrets.json file path.
--MAX_RECORDS_IN_RAM	500000	When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed.
--QUIET	false	Whether to suppress job-summary info on System.err.
--REFERENCE_SEQUENCE -R	null	Reference sequence file.
--TMP_DIR	[]	One or more directories with space available to be used by this program for temporary storage of working files
--USE_JDK_DEFLATER -use_jdk_deflater	false	Use the JDK Deflater instead of the Intel Deflater for writing compressed output
--USE_JDK_INFLATER -use_jdk_inflater	false	Use the JDK Inflater instead of the Intel Inflater for reading compressed input
--VALIDATION_STRINGENCY	STRICT	Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
--VERBOSITY	INFO	Control verbosity of logging.
Advanced Arguments
--showHidden	false	display hidden arguments

Argument details

Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.

--ALLOW_MULTIPLE_DOWNSAMPLING_DESPITE_WARNINGS / NA

Allow downsampling again despite this being a bad idea with possibly unexpected results.

boolean false

--arguments_file / NA

read one or more arguments files and add them to the command line

List[File] []

--COMPRESSION_LEVEL / NA

Compression level for all compressed files created (e.g. BAM and VCF).

int 5 [ [ -∞ ∞ ] ]

--CREATE_INDEX / NA

Whether to create a BAM index when writing a coordinate-sorted BAM file.

Boolean false

--CREATE_MD5_FILE / NA

Whether to create an MD5 digest for any BAM or FASTQ files created.

boolean false

--FRACTION / -F

The (approximate) fraction of reads to be kept, between 0 and 1.

R Double null

--GA4GH_CLIENT_SECRETS / NA

Google Genomics API client_secrets.json file path.

String client_secrets.json

--help / -h

display the help message

boolean false

--INPUT / -I

The input SAM or BAM file to downsample.

R File null

--MAX_RECORDS_IN_RAM / NA

When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed.

Integer 500000 [ [ -∞ ∞ ] ]

--OUTPUT / -O

The output, downsampled, SAM or BAM file.

R File null

--QUIET / NA

Whether to suppress job-summary info on System.err.

Boolean false

--READ_NAME_REGEX / NA

Use these regular expressions to parse read names in the input SAM file. Read names are parsed to extract three variables: tile/region, x coordinate and y coordinate. The x and y coordinates are used to determine the downsample decision. Set this option to null to disable optical duplicate detection, e.g. for RNA-seq The regular expression should contain three capture groups for the three variables, in order. It must match the entire read name. Note that if the default regex is specified, a regex match is not actually done, but instead the read name is split on colons (:). For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.

String

--REFERENCE_SEQUENCE / -R

Reference sequence file.

File null

--REMOVE_DUPLICATE_INFORMATION / NA

Determines whether the duplicate tag should be reset since the downsampling requires re-marking duplicates.

boolean true

--showHidden / -showHidden

display hidden arguments

boolean false

--STOP_AFTER / NA

Stop after processing N reads, mainly for debugging.

Long null

--TMP_DIR / NA

One or more directories with space available to be used by this program for temporary storage of working files

List[File] []

--USE_JDK_DEFLATER / -use_jdk_deflater

Use the JDK Deflater instead of the Intel Deflater for writing compressed output

Boolean false

--USE_JDK_INFLATER / -use_jdk_inflater

Use the JDK Inflater instead of the Intel Inflater for reading compressed input

Boolean false

--VALIDATION_STRINGENCY / NA

Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.

The --VALIDATION_STRINGENCY argument is an enumerated type (ValidationStringency), which can have one of the following values:

STRICT
LENIENT
SILENT

ValidationStringency STRICT

--VERBOSITY / NA

Control verbosity of logging.

The --VERBOSITY argument is an enumerated type (LogLevel), which can have one of the following values:

ERROR
WARNING
INFO
DEBUG

LogLevel INFO

--version / NA

display the version number for this tool

boolean false

Return to top

GATK version 4.0.11.0 built at 23-11-2018 02:11:49.