1000_genome_HG02759
Google Cloud Platform (GCP), 4 vCPUs, 15 GB
Google Cloud Platform (GCP), 24 vCPUs, 32 GB
Continuing in background, pid 25770.
Continuing in background, pid 25776.
Continuing in background, pid 25788.
Continuing in background, pid 25790.
-rw-rw-r– 1 b0d2647 b0d2647 5595686029 Apr 22 02:03 ERR251295_1.fastq.gz
-rw-rw-r– 1 b0d2647 b0d2647 5719449352 Apr 22 02:03 ERR251295_2.fastq.gz
-rw-rw-r– 1 b0d2647 b0d2647 5653494965 Apr 21 22:30 ERR251296_1.fastq.gz
-rw-rw-r– 1 b0d2647 b0d2647 5772390726 Apr 21 22:19 ERR251296_2.fastq.gz
Copying gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta…
Copying gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.dict…
Copying gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta.fai…
-rw-rw-r– 1 b0d2647 b0d2647 581712 Apr 20 13:42 Homo_sapiens_assembly38.dict
-rw-rw-r– 1 b0d2647 b0d2647 3249912778 Apr 20 13:44 Homo_sapiens_assembly38.fasta
-rw-rw-r– 1 b0d2647 b0d2647 160928 Apr 20 16:47 Homo_sapiens_assembly38.fasta.fai
[bwa_index] Pack FASTA… 30.70 sec
[bwa_index] Construct BWT for the packed sequence…
[BWTIncCreate] textLength=6434693834, availableWord=464768632
[BWTIncConstructFromPacked] 10 iterations done. 99999994 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 199999994 characters processed.
………….
[BWTIncConstructFromPacked] 700 iterations done. 6411146922 characters processed.
[BWTIncConstructFromPacked] 710 iterations done. 6432978554 characters processed.
[bwt_gen] Finished constructing BWT in 711 iterations.
[bwa_index] 4734.94 seconds elapse.
[bwa_index] Update BWT… 23.36 sec
[bwa_index] Pack forward-only FASTA… 22.12 sec
[bwa_index] Construct SA from BWT and Occ… 2665.77 sec
[main] Version: 0.7.17-r1188
[main] CMD: bwa index -a bwtsw Homo_sapiens_assembly38.fasta
[main] Real time: 7558.780 sec; CPU: 7476.896 sec
-rw-rw-r– 1 b0d2647 b0d2647 487553 Apr 21 22:49 Homo_sapiens_assembly38.fasta.64.alt
-rw-rw-r– 1 b0d2647 b0d2647 20199 Apr 21 22:49 Homo_sapiens_assembly38.fasta.64.amb
-rw-rw-r– 1 b0d2647 b0d2647 455474 Apr 21 22:49 Homo_sapiens_assembly38.fasta.64.ann
-rw-rw-r– 1 b0d2647 b0d2647 3217347004 Apr 21 22:49 Homo_sapiens_assembly38.fasta.64.bwt
-rw-rw-r– 1 b0d2647 b0d2647 804336731 Apr 21 22:49 Homo_sapiens_assembly38.fasta.64.pac
-rw-rw-r– 1 b0d2647 b0d2647 1608673512 Apr 21 22:50 Homo_sapiens_assembly38.fasta.64.sa
Note: It is important to assign read groups (RG) to the sequences
@ERR251295.1 FCC1H7WACXX:1:1101:1133:2181/1 TGCTNAAAATACAGAAGACAAATTTTCTCATTTAGGGAAATCAATTTTATTAGTGCTTGGTCAGAAAAGACTAAAAATTTCATGTTGAATACGTACTAAT + CCCF#2ADHHHHGIGGHGIIIGHIJJJIIHEIIGGIJJJJIJJCGGHIIHJHIGHIEIJJBGIIIJJGIGIIIJIEEEEHFFFFCDEEEEEEDCCBBEDD
@ERR251295.2 FCC1H7WACXX:1:1101:1086:2200/1 TGACNATTCCATTCAATTCTGTTCAATGATTCCCTTAGATTCCATTTGATGATGATTCCATTCGATTCCATTTGATGATGATTCCATGCGATTCCATTAG + BC@F#2ADDHHHHJJIHIIJIJJJJJJIGIJJJJJJGHIJJIJBHIJIGHIFIIHIJJIJIJJJIJIIH>DHIIHEGHHGGHGJIJJJJJIGHHHHHFFF
@ERR251295.3 FCC1H7WACXX:1:1101:1094:2225/1 GCTGNGAGCCTCGTTGCCATGTAGGCAACTGTATTGAAAGAGTCCAGGCTCCTCCTCAGATTAGGGACTCATTTCCTCTCCCCTCTACAAGCCCGGTGTT
@ERR251295.1 FCC1H7WACXX:1:1101:1133:2181/2 TTGTCTAGTTACTTGAACAATAAGAGAAATTCTATCTTTGAGCTTTCATAATTGGCTTTCTGGACACCACTGAGTTTATTAGATAATCTTTAAAAATATT
…………….
[main] Version: 0.7.17-r1188
[main] Real time: 5776.739 sec; CPU: 81456.772 sec
[bam_sort_core] merging from 49 files and 1 in-memory blocks…
129330360 + 0 in total (QC-passed reads + QC-failed reads)
1128310 + 0 secondary
832800 + 0 supplementary
0 + 0 duplicates
128944247 + 0 mapped (99.70% : N/A)
127369250 + 0 paired in sequencing
63684625 + 0 read1
63684625 + 0 read2
122823900 + 0 properly paired (96.43% : N/A)
126601922 + 0 with itself and mate mapped
381215 + 0 singletons (0.30% : N/A)
2253238 + 0 with mate mapped to a different chr
1052000 + 0 with mate mapped to a different chr (mapQ>=5)
-rw-rw-r– 1 b0d2647 b0d2647 12241966896 Apr 26 22:08 ERR251295_1.bam
-rw-rw-r– 1 b0d2647 b0d2647 6665728 Apr 26 22:12 ERR251295_1.bam.bai
Using GATK jar /home/b0d2647/miniconda3/share/gatk4-4.1.2.0-0/gatk-package-4.1.2.0-local.jar
…………………
INFO 2019-04-27 23:39:51 SamFileValidator Validated Read 10,000,000 records. Elapsed time: 00:01:41s. Time for last 10,000,000: 100s. Last read position: chr1:221,207,964
INFO 2019-04-27 23:41:24 SamFileValidator Validated Read 20,000,000 records. Elapsed time: 00:03:14s. Time for last 10,000,000:93s. Last read position: chr2:211,520,747
…………………….
INFO 2019-04-27 23:58:03 SamFileValidator Validated Read 120,000,000 records. Elapsed time: 00:19:53s. Time for last 10,000,000: 104s. Last read position: chr22:47,204,986
[Sun Apr 28 00:06:18 UTC 2019] picard.sam.ValidateSamFile done. Elapsed time: 28.17 minutes.
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Tool returned: 3
## HISTOGRAM java.lang.String
Error Type Count
ERROR: INVALID_TAG_NM 16816
….
[bam_fillmd1] different MD for read ‘ERR251295.60210117’: ‘33G0T19C0C1C4C11C23’ -> ‘0N32G0T19C0C1C4C11C23’
[bam_fillmd1] different NM for read ‘ERR251295.59668755’: 0 -> 4
[bam_fillmd1] different MD for read ‘ERR251295.59668755’: ‘54’ -> ‘0N0N0N0N50’
[bam_fillmd1] different NM for read ‘ERR251295.47869477’: 0 -> 1
[bam_fillmd1] different MD for read ‘ERR251295.47869477’: ‘73’ -> ‘72N0’
-rw-rw-r– 1 b0d2647 b0d2647 13319762424 Apr 28 01:54 ERR251295_1.calmd.bam
-rw-rw-r– 1 b0d2647 b0d2647 3525 Apr 28 04:53 ERR251295_1.calmd.dupMetrics.txt
-rw-rw-r– 1 b0d2647 b0d2647 17998947047 Apr 28 04:53 ERR251295_1.calmd.mdup.bam
-rw-rw-r– 1 b0d2647 b0d2647 8635016 Apr 28 05:06 ERR251295_1.calmd.mdup.bam.bai
05:10:13.535 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
05:10:23.556 INFO ProgressMeter - chr20:16090633 0.2 1270 7604.8
05:10:23.995 INFO HaplotypeCaller -
167 read(s) filtered by: ((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter) AND WellformedReadFilter)
167 read(s) filtered by: (((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter)
167 read(s) filtered by: ((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter)
167 read(s) filtered by: (((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter)
167 read(s) filtered by: ((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter)
63 read(s) filtered by: (((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter)
56 read(s) filtered by: ((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter)
56 read(s) filtered by: (MappingQualityReadFilter AND MappingQualityAvailableReadFilter)
56 read(s) filtered by: MappingQualityReadFilter
7 read(s) filtered by: NotSecondaryAlignmentReadFilter
104 read(s) filtered by: NotDuplicateReadFilter
05:10:24.008 INFO ProgressMeter - chr20:16098307 0.2 1317 7545.1
05:10:24.009 INFO ProgressMeter - Traversal complete. Processed 1317 total regions in 0.2 minutes.
-rw-rw-r– 1 b0d2647 b0d2647 212003 Apr 28 05:10 ERR251295_1.calmd.mdup.vcf
-rw-rw-r– 1 b0d2647 b0d2647 115099 Apr 28 05:10 ERR251295_1.calmd.mdup.vcf.idx
# This file was produced by bcftools stats (1.9+htslib-1.9) and can be plotted using plot-vcfstats.
# SN [2]id [3]key [4]value
SN 0 number of samples: 1
SN 0 number of records: 200
SN 0 number of no-ALTs: 0
SN 0 number of SNPs: 196
SN 0 number of MNPs: 0
SN 0 number of indels: 4
SN 0 number of others: 0
SN 0 number of multiallelic sites: 0
SN 0 number of multiallelic SNP sites: 0
@ERR251296.1 FCC1H7WACXX:2:1101:1495:2145/1
NTCAGAAACTTGTTTGTGATGTGTGCCCTCTACTGACTGAGTTGAACCTTTCTTTTCATAGAGCAGTTTTGAAACACTCTTTTTGTAGAATCTGCAAGAG
@ERR251296.1 FCC1H7WACXX:2:1101:1495:2145/2
GCAAATATCCTCTTGCAGATTCCAGAAAAAGAGTGTTTCAAAACTGCTCCTTCAAAACGGTGGTTCAATTCTCTTAGTTGAGTACACACATCTCAAATAA
…………
[main] Version: 0.7.17-r1188
……………………
[main] Real time: 3407.936 sec; CPU: 72679.793 sec
[bam_sort_core] merging from 49 files and 1 in-memory blocks…
129958650 + 0 in total (QC-passed reads + QC-failed reads)
1139152 + 0 secondary
838006 + 0 supplementary
0 + 0 duplicates
129602327 + 0 mapped (99.73% : N/A)
127981492 + 0 paired in sequencing
63990746 + 0 read1
63990746 + 0 read2
123468430 + 0 properly paired (96.47% : N/A)
127273926 + 0 with itself and mate mapped
351243 + 0 singletons (0.27% : N/A)
2269832 + 0 with mate mapped to a different chr
1057580 + 0 with mate mapped to a different chr (mapQ>=5)
Using GATK jar /home/b0d2647/miniconda3/share/gatk4-4.1.2.0-0/gatk-package-4.1.2.0-local.jar
Tool returned: 3
## HISTOGRAM java.lang.String
Error Type Count
ERROR:INVALID_TAG_NM 17041
……………..
INFO 2019-04-28 04:31:40 SortingCollection Creating merging iterator from 6 files
INFO 2019-04-28 04:32:30 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2019-04-28 04:32:30 SortingCollection Creating merging iterator from 11 files
INFO 2019-04-28 04:33:42 MarkDuplicates Sorting list of duplicate records.
INFO 2019-04-28 04:33:42 MarkDuplicates After generateDuplicateIndexes freeMemory: 2790920768; totalMemory: 3755474944; maxMemory: 3755474944
INFO 2019-04-28 04:33:42 MarkDuplicates Marking 1927552 records as duplicates.
INFO 2019-04-28 04:33:42 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2019-04-28 04:33:43 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2019-04-28 04:35:48 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:02:05s. Time for last 10,000,000:125s. Last read position: chr1:220,296,254
INFO 2019-04-28 04:38:00 MarkDuplicates Written 20,000,000 records. Elapsed time: 00:04:17s. Time for last 10,000,000:131s. Last read position: chr2:209,223,795
………………….
INFO 2019-04-28 04:53:37 MarkDuplicates Written 120,000,000 records. Elapsed time: 00:19:54s. Time for last 10,000,000:87s. Last read position: chr22:35,225,173
………………………
[Sun Apr 28 04:54:47 UTC 2019] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 50.44 minutes.
Tool returned:0
05:12:24.446 INFO HaplotypeCaller - 158 read(s) filtered by: ((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadF ilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentRe adFilter) AND GoodCigarReadFilter) AND WellformedReadFilter)
……………………..
05:12:24.448 INFO ProgressMeter - chr20:16098789 0.1 1317 9246.4
05:12:24.448 INFO ProgressMeter - Traversal complete. Processed 1317 total regions in 0.1 minutes.
# This file was produced by bcftools stats (1.9+htslib-1.9) and can be plotted using plot-vcfstats.
.............
# SN [2]id [3]key [4]value
SN 0 number of samples: 1
SN 0 number of records: 207
SN 0 number of no-ALTs: 0
SN 0 number of SNPs: 202
SN 0 number of MNPs: 0
SN 0 number of indels: 5
SN 0 number of others: 0
SN 0 number of multiallelic sites: 0
SN 0 number of multiallelic SNP sites: 0
-rw-rw-r– 1 b0d2647 b0d2647 12359531427 Apr 28 00:39 ERR251296_1.bam
-rw-rw-r– 1 b0d2647 b0d2647 6765576 Apr 28 00:41 ERR251296_1.bam.bai
-rw-rw-r– 1 b0d2647 b0d2647 13441427070 Apr 28 02:59 ERR251296_1.calmd.bam
-rw-rw-r– 1 b0d2647 b0d2647 3531 Apr 28 04:54 ERR251296_1.calmd.dupMetrics.txt
-rw-rw-r– 1 b0d2647 b0d2647 18167621403 Apr 28 04:54 ERR251296_1.calmd.mdup.bam
-rw-rw-r– 1 b0d2647 b0d2647 8673408 Apr 28 05:07 ERR251296_1.calmd.mdup.bam.bai
-rw-rw-r– 1 b0d2647 b0d2647 213523 Apr 28 05:12 ERR251296_1.calmd.mdup.vcf
-rw-rw-r– 1 b0d2647 b0d2647 115099 Apr 28 05:12 ERR251296_1.calmd.mdup.vcf.idx
………………….
INFO 2019-04-28 03:50:00 MergeSamFiles Processed 259,000,000 records. Elapsed time: 00:19:20s. Time for last 1,000,000: 7s.Last read position:chrUn_JTFH01001899v1_decoy:273
INFO 2019-04-28 03:50:02 MergeSamFiles Finished reading inputs.
[Sun Apr 28 03:50:02 UTC 2019] picard.sam.MergeSamFiles done. Elapsed time: 19.39 minutes.
INFO 2019-04-28 04:44:56 SortingCollection Creating merging iterator from 10 files
INFO 2019-04-28 04:46:32 MarkDuplicates Traversing fragment information and detecting duplicates.
INFO 2019-04-28 04:46:32 SortingCollection Creating merging iterator from 21 files
INFO 2019-04-28 04:48:49 MarkDuplicates Sorting list of duplicate records.
INFO 2019-04-28 04:48:51 MarkDuplicates After generateDuplicateIndexes freeMemory: 2927996376; totalMemory: 3935305728; maxMemory: 3935305 728 INFO 2019-04-28 04:48:51 MarkDuplicates Marking 6604590 records as duplicates.
INFO 2019-04-28 04:48:51 MarkDuplicates Found 0 optical duplicate clusters.
INFO 2019-04-28 04:48:51 MarkDuplicates Reads are assumed to be ordered by: coordinate
INFO 2019-04-28 04:50:23 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:01:31s. Time for last 10,000,000: 91s. Last read position: chr1:114,332,129
………………….
INFO 2019-04-28 05:11:03 MarkDuplicates Written 250,000,000 records. Elapsed time: 00:22:11s. Time for last 10,000,000: 49s. Last read position: chrY:56,737,140
……………………
[Sun Apr 28 05:11:50 UTC 2019] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 76.69 minutes.
05:29:40.276 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
05:29:50.098 INFO HaplotypeCaller -
500 read(s) filtered by: ((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter) AND WellformedReadFilter)
500 read(s) filtered by: (((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter) AND GoodCigarReadFilter)
500 read(s) filtered by: ((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonZeroReferenceLengthAlignmentReadFilter)
500 read(s) filtered by: (((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter)
500 read(s) filtered by: ((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter)
133 read(s) filtered by: (((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter) AND NotSecondaryAlignmentReadFilter)
122 read(s) filtered by: ((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappedReadFilter)
122 read(s) filtered by: (MappingQualityReadFilter AND MappingQualityAvailableReadFilter)
122 read(s) filtered by: MappingQualityReadFilter
11 read(s) filtered by: NotSecondaryAlignmentReadFilter
367 read(s) filtered by: NotDuplicateReadFilter
05:29:50.098 INFO ProgressMeter - chr20:16099540 0.2 1373 8387.3
05:29:50.099 INFO ProgressMeter - Traversal complete. Processed 1373 total regions in 0.2 minutes.
# This file was produced by bcftools stats (1.9+htslib-1.9) and can be plotted using plot-vcfstats.
.............
# SN [2]id [3]key [4]value
SN 0 number of samples: 1
SN 0 number of records: 286
SN 0 number of no-ALTs: 0
SN 0 number of SNPs: 276
SN 0 number of MNPs: 0
SN 0 number of indels: 10
SN 0 number of others: 0
SN 0 number of multiallelic sites: 0
SN 0 number of multiallelic SNP sites: 0
-rw-rw-r– 1 b0d2647 b0d2647 34307229848 Apr 28 03:50 ERR25129596.bam
-rw-rw-r– 1 b0d2647 b0d2647 3569 Apr 28 05:11 ERR25129596.dupMetrics.txt
-rw-rw-r– 1 b0d2647 b0d2647 34564180275 Apr 28 05:11 ERR25129596.mdup.bam
-rw-rw-r– 1 b0d2647 b0d2647 9185888 Apr 28 05:24 ERR25129596.mdup.bam.bai
-rw-rw-r– 1 b0d2647 b0d2647 230777 Apr 28 05:29 ERR25129596.mdup.vcf
-rw-rw-r– 1 b0d2647 b0d2647 116101 Apr 28 05:29 ERR25129596.mdup.vcf.idx