BigDataScript report: atac.bds

Script file /users/pangwei/bds_atac/atac.bds
Program ID atac.bds.20160516_120116_681
Start time 2016-05-16 12:01:16
Run time 07:08:18.611
Tasks executed 5
Tasks failed 0
Tasks failed names
 
Arguments* [-out_dir, /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14, -num_rep, 2, -fastq1_1, /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair1.fastq.gz, -fastq1_2, /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair2.fastq.gz, -fastq2_1, /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair1.fastq.gz, -fastq2_2, /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair2.fastq.gz, -subsample, 17500000, -true_rep, -species, hg19]
System* local
Cpus* 1
Exit value 0
* Values in global scope when program finished execution.

Timeline

Task Graph

Parallel (threads) details

Thread ID Parent Tasks
thread_Root None
  
thread_20 thread_Root
 atac.bds.20160516_120116_681_parallel_20/task.align_etc.trim_adapters_PE_rep1.line_57.id_6
atac.bds.20160516_120116_681_parallel_20/task.align_bowtie2.bowtie2_PE_rep1.line_133.id_9
 
thread_21 thread_Root
 atac.bds.20160516_120116_681_parallel_21/task.align_etc.trim_adapters_PE_rep2.line_57.id_7
atac.bds.20160516_120116_681_parallel_21/task.align_bowtie2.bowtie2_PE_rep2.line_133.id_8
atac.bds.20160516_120116_681_parallel_21/task.postalign_bam.dedup_bam_PE_rep2.line_215.id_10
 

Task details

Task Execution Time Dependencies Task program, Errors, StdOut / StdErr
Num 1
ID task.align_etc.trim_adapters_PE_rep2.line_57.id_7
Name trim_adapters_PE rep2
Thread thread_21
PID 109907
OK true
Exit Code 0
Retries
State FINISHED
Dep. OK
Cpus 1
Mem
Start 2016-05-16 12:01:18
End 2016-05-16 13:14:13
Elapsed 01:12:55
Timeout 00:00:-1
Wall Timeout 100 days
Input files /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair1.fastq.gz /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair2.fastq.gz
Output files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.fastq.gz /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair2.trim.fastq.gz
Dependencies
 
# SYS command. line 59

 if [[ -f $(which activate) ]]; then source activate bds_atac; fi;  export PATH=/users/pangwei/bds_atac/.:/users/pangwei/bds_atac/modules:/users/pangwei/bds_atac/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds;

# SYS command. line 61

 cd /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2

# SYS command. line 62

 $(which trimAdapters.py) -a /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair1.fastq.gz -b /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC14/ATAC14-trimmed-pair2.fastq.gz

# SYS command. line 63

 gzip /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.fastq

# SYS command. line 64

 gzip /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair2.trim.fastq
 
   
--------------------Stdout--------------------
ATAC14-trimmed-pair1.trim.fastq
ATAC14-trimmed-pair2.trim.fastq

 
--------------------Stderr--------------------
discarding /users/pangwei/anaconda3/bin from PATH
prepending /users/pangwei/anaconda3/envs/bds_atac/bin to PATH
45671279 sequences total
20535613 sequences trimmed with 0 mismatches
2983759 sequences trimmed with 1 mismatch
23 mean number of bases trimmed for reads requiring trimming

 
Num 2
ID task.align_etc.trim_adapters_PE_rep1.line_57.id_6
Name trim_adapters_PE rep1
Thread thread_20
PID 109916
OK true
Exit Code 0
Retries
State FINISHED
Dep. OK
Cpus 1
Mem
Start 2016-05-16 12:01:18
End 2016-05-16 13:20:51
Elapsed 01:19:33
Timeout 00:00:-1
Wall Timeout 100 days
Input files /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair1.fastq.gz /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair2.fastq.gz
Output files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.fastq.gz /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair2.trim.fastq.gz
Dependencies
 
# SYS command. line 59

 if [[ -f $(which activate) ]]; then source activate bds_atac; fi;  export PATH=/users/pangwei/bds_atac/.:/users/pangwei/bds_atac/modules:/users/pangwei/bds_atac/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds;

# SYS command. line 61

 cd /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1

# SYS command. line 62

 $(which trimAdapters.py) -a /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair1.fastq.gz -b /srv/scratch/pangwei/mesoderm/data/ATACseq/ATAC13/ATAC13-trimmed-pair2.fastq.gz

# SYS command. line 63

 gzip /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.fastq

# SYS command. line 64

 gzip /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair2.trim.fastq
 
   
--------------------Stdout--------------------
ATAC13-trimmed-pair1.trim.fastq
ATAC13-trimmed-pair2.trim.fastq

 
--------------------Stderr--------------------
discarding /users/pangwei/anaconda3/bin from PATH
prepending /users/pangwei/anaconda3/envs/bds_atac/bin to PATH
48514285 sequences total
21219623 sequences trimmed with 0 mismatches
2842422 sequences trimmed with 1 mismatch
23 mean number of bases trimmed for reads requiring trimming

 
Num 3
ID task.align_bowtie2.bowtie2_PE_rep2.line_133.id_8
Name bowtie2_PE rep2
Thread thread_21
PID 111251
OK true
Exit Code 0
Retries
State FINISHED
Dep. OK
Cpus 2
Mem
Start 2016-05-16 13:14:14
End 2016-05-16 19:06:39
Elapsed 05:52:24
Timeout 00:00:-1
Wall Timeout 100 days
Input files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.fastq.gz /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair2.trim.fastq.gz
Output files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.align.log
Dependencies
 
# SYS command. line 135

 if [[ -f $(which activate) ]]; then source activate bds_atac; fi;  export PATH=/users/pangwei/bds_atac/.:/users/pangwei/bds_atac/modules:/users/pangwei/bds_atac/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds;

# SYS command. line 139

 bowtie2  -X2000 --mm --threads 2 -x /mnt/data/annotations/indexes/bowtie2_indexes/bowtie2/ENCODEHg19_male \
			-1 /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.fastq.gz -2 /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair2.trim.fastq.gz 2>/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.align.log | samtools view -bS - | samtools sort - /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE

# SYS command. line 142

 samtools index /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.bam
 
    
--------------------Stderr--------------------
discarding /users/pangwei/anaconda3/bin from PATH
prepending /users/pangwei/anaconda3/envs/bds_atac/bin to PATH
[bam_sort_core] merging from 38 files...

 
Num 4
ID task.align_bowtie2.bowtie2_PE_rep1.line_133.id_9
Name bowtie2_PE rep1
Thread thread_20
PID 111491
OK false
Exit Code 0
Retries
State RUNNING
Dep. WAIT
Cpus 2
Mem
Start 2016-05-16 13:20:52
End 2016-05-16 13:20:52
Elapsed 00:00:00
Timeout 00:00:-1
Wall Timeout 100 days
Input files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.fastq.gz /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair2.trim.fastq.gz
Output files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.PE2SE.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep1/ATAC13-trimmed-pair1.trim.PE2SE.align.log
Dependencies
 
# SYS command. line 135

 if [[ -f $(which activate) ]]; then source activate bds_atac; fi;  export PATH=/users/pangwei/bds_atac/.:/users/pangwei/bds_atac/modules:/users/pangwei/bds_atac/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds;

# SYS command. line 139

 bowtie2  -X2000 --mm --threads 2 -x /mnt/data/annotations/indexes/bowtie2_indexes/bowtie2/ENCODEHg19_male \
			-1 /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.fastq.gz -2 /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair2.trim.fastq.gz 2>/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep1/ATAC13-trimmed-pair1.trim.PE2SE.align.log | samtools view -bS - | samtools sort - /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.PE2SE

# SYS command. line 142

 samtools index /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep1/ATAC13-trimmed-pair1.trim.PE2SE.bam
 
    
--------------------Stderr--------------------
discarding /users/pangwei/anaconda3/bin from PATH
prepending /users/pangwei/anaconda3/envs/bds_atac/bin to PATH
[bam_sort_core] merging from 40 files...

 
Num 5
ID task.postalign_bam.dedup_bam_PE_rep2.line_215.id_10
Name dedup_bam_PE rep2
Thread thread_21
PID 125731
OK false
Exit Code 0
Retries
State RUNNING
Dep. WAIT
Cpus 1
Mem
Start 2016-05-16 19:06:40
End 2016-05-16 19:06:40
Elapsed 00:00:00
Timeout 00:00:-1
Wall Timeout 100 days
Input files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.bam
Output files /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dup.qc /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.flagstat.qc /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.pbc.qc
Dependencies
 
# SYS command. line 217

 if [[ -f $(which activate) ]]; then source activate bds_atac; fi;  export PATH=/users/pangwei/bds_atac/.:/users/pangwei/bds_atac/modules:/users/pangwei/bds_atac/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds;

# SYS command. line 228

 echo

# SYS command. line 229

 samtools view -F 1804 -f 2 -q 30 -u /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.bam | samtools sort -n - /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark

# SYS command. line 230

 echo

# SYS command. line 237

 samtools fixmate -r /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam.fixmate.bam

# SYS command. line 238

 samtools view -F 1804 -f 2 -u /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam.fixmate.bam | samtools sort - /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt

# SYS command. line 239

 rm -f /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam.fixmate.bam

# SYS command. line 243

 export _JAVA_OPTIONS="-Xms256M -Xmx4G -XX:ParallelGCThreads=1"

# SYS command. line 244

 export MAX_JAVA_MEM="12G"

# SYS command. line 247

 if [ -f ${PICARDROOT}/MarkDuplicates.jar ]; then \
			java -Xmx4G -jar ${PICARDROOT}/MarkDuplicates.jar \
				INPUT="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam" OUTPUT="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam" \
				METRICS_FILE="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dup.qc" VALIDATION_STRINGENCY=LENIENT \
				ASSUME_SORTED=true REMOVE_DUPLICATES=false; \
			else \
			java -Xmx4G -jar ${PICARDROOT}/picard.jar MarkDuplicates \
				INPUT="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam" OUTPUT="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam" \
				METRICS_FILE="/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dup.qc" VALIDATION_STRINGENCY=LENIENT \
				ASSUME_SORTED=true REMOVE_DUPLICATES=false; \
			fi

# SYS command. line 265

 mv /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam

# SYS command. line 267

 samtools view -F 1804 -f 2 -b /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam > /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam

# SYS command. line 269

 samtools index /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam

# SYS command. line 271

 if [ "/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam.bai" != "/srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bai" ]; then \
			cp /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam.bai /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bai; \
			fi

# SYS command. line 275

 samtools flagstat /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.bam > /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.flagstat.qc

# SYS command. line 286

 samtools sort -n /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam.tmp

# SYS command. line 287

 bedtools bamtobed -bedpe -i /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam.tmp.bam | \
			awk 'BEGIN{OFS="\t"}{print $1,$2,$4,$6,$9,$10}' | \
			grep -v 'chrM' | sort | uniq -c | \
			awk 'BEGIN{mt=0;m0=0;m1=0;m2=0} ($1==1){m1=m1+1} ($1==2){m2=m2+1} {m0=m0+1} {mt=mt+$1} END{printf "%d\t%d\t%d\t%d\t%f\t%f\t%f\n",mt,m0,m1,m2,m0/mt,m1/m0,m1/m2}' > /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/qc/rep2/ATAC14-trimmed-pair1.trim.PE2SE.nodup.pbc.qc

# SYS command. line 291

 rm /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam.tmp.bam

# SYS command. line 293

 rm -f /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.filt.bam

# SYS command. line 294

 rm -f /srv/scratch/pangwei/mesoderm/bulk-atac/output-hg19-scidata/ATAC13+14/align/rep2/ATAC14-trimmed-pair1.trim.PE2SE.dupmark.bam
 
   
--------------------Stdout--------------------


 
--------------------Stderr--------------------
discarding /users/pangwei/anaconda3/bin from PATH
prepending /users/pangwei/anaconda3/envs/bds_atac/bin to PATH

 

Global scope

Note: Global scope when program finished execution.
Type Name Value