ó QŽfXc@s¨ddlZddlZddlZddlZddlZddlZddlmZddlZddl Z yddl Z Wne k r ej dƒnXeej ƒejjdƒrÙddlmZmZn+ejjdƒrddlmZmZndZee jZdZd efd „ƒYZd „Zd „Zd efd„ƒYZd„Zed„Zdefd„ƒYZ d„Z!ed„Z"d„Z#dS(iÿÿÿÿN(t addsitedirsICan't process Lucigen NxMate reads! Python module regex is not installed!s2.(tParalleltdelayeds3.i¨ait ParseFastQcBsDeZdZddgd„Zd„Zd„Zd„Zd„ZRS(s@Returns a read-by-read fastQ parser analogous to file.readline()t@t+cCsL|jdƒr$tj|ƒ|_nt|dƒ|_d|_||_dS(sReturns a read-by-read fastQ parser analogous to file.readline(). Exmpl: parser.next() -OR- Its an iterator so you can do: for rec in parser: ... do something with rec ... rec is tuple: (seqHeader,seqStr,qualHeader,qualStr) s.gztrUiN(tendswithtgziptopent_filet_currentLineNumbert_hdSyms(tselftfilePatht headerSymbols((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyt__init__=s  cCs|S(N((R ((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyt__iter__NscCs |jƒS(N(t__next__(R ((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytnextQscCsƒg}x^tdƒD]P}|jjƒ}|jd7_|rV|j|jdƒƒq|jd ƒqWg|D]}t|ƒ^qnjt ƒ}|jd ƒ}|dkr³t ‚n|dksÒt d|jƒ‚|dj |j dƒs t d|j d|jfƒ‚|dj |j dƒsFt d|j d|jfƒ‚t|dƒt|d ƒksyt d |jƒ‚t|ƒS( s{Reads in next element, parses, and does minimal verification. Returns: tuple: (seqHeader,seqStr,qualHeader,qualStr)iis s¥** ERROR: It looks like I encountered a premature EOF or empty line. Please check FastQ file near line number %s (plus or minus ~4 lines) and try again**is¢** ERROR: The 1st line in fastq element does not start with '%s'. Please check FastQ file near line number %s (plus or minus ~4 lines) and try again**is¢** ERROR: The 3rd line in fastq element does not start with '%s'. Please check FastQ file near line number %s (plus or minus ~4 lines) and try again**is»** ERROR: The length of Sequence data and Quality data of the last record aren't equal. Please check FastQ file near line number %s (plus or minus ~4 lines) and try again**N(trangeR treadlineR tappendtstriptNonetbooltcounttTruet StopIterationtAssertionErrort startswithR tlenttuple(R telemListtitlinetxttruestnones((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyRTs0(   # cCs|jr|jjƒndS(N(R tclose(R ((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR'}s (t__name__t __module__t__doc__RRRRR'(((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR:s    )cCsWxPt||ƒD]?\}}x0|D](}x|D]}|j|dƒq0Wq#WqWdS(Ns (tziptwrite(t file_handlerst record_listst file_handlert record_listtrecordR#((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytwrite_to_files‚s  cCsmt|ƒt|ƒ}g}d}xD|t|ƒkrh|j|t|ƒt||ƒ!ƒ||7}q%W|S(Ng(RtfloatRtint(tltntavgtouttlast((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytsplit_into_chunks‰s$t CleanStatscBseZd„Zd„ZRS(c CsUd|_d|_d|_d|_ddddddddddddg |_dS(Ni(t readcountert matecountertTOTALmatecountert slagcountert csscounter(R ((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR”s     cCs|j|j7_|j|j7_|j|j7_|j|j7_gt|j|jƒD]\}}||^q^|_|S(N(R<R=R>R?R+R@(R totherR$ty((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyt__add__›s 5(R(R)RRC(((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR;“s cCsZtƒ}g}g}g}g}x |D]\}} |jd7_xút|ƒD]ì\} } tj| |dƒ} || } tj| | dƒ}| rÌ|rÌ|jd7_|j| cd7<| jƒ}|d}|d}|d}||}|d}||}|jƒ}|d}|d}| d}||}| d}||}t|ƒt krÈt|ƒt krÈ|j d7_ |j |d||d|gƒ|j | d|| d|gƒnPqP| dkrP|j d7_ |j |d|d|d|dgƒ|j | d| d| d| dgƒqPqPWq(W||||g|fS(Nii iiii ( R;R<t enumeratetregextsearchR>R@tspanRtminseqR=RR?(treadstcsslist1tcsslist2tstatstprocessed_out1tprocessed_out2tprocessed_slag1tprocessed_slag2trecR1trecR2tcssindextcss1tmtcss2R6t R1matchestmendtmySeqtmyR1tmyQual1tmyR1Qualt R2matchestnendtmySeq2tmyR2tmyQual2tmyR2Qual((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytchimera_clean_process_batch¤sL                 $!$ )4c! s²tjƒ}tjj|ƒ}tjj|ƒddkrStjj|ƒd}ntjj|ƒ}tjj|ƒddkrštjj|ƒd}ntjj|d|ƒ}t|dƒ} tjj|d|ƒ} t| dƒ} tjj|d|ƒ} t| dƒ} tjj|d|ƒ}t|dƒ}ddd d d d d dddddg ‰ddddddddddddg ‰t|ƒ}t|ƒ}tƒ}t j }x‹t rEt t j|tƒƒ}t t j|tƒƒ}t|ƒt|ƒkr2tjdt|ƒt|ƒf|ƒn|s<Pntt t||ƒƒ|ƒ}td |ƒ‡‡fd!†|Dƒƒ}g|D]}|d^q‰g|D]}|d^q£}}x@t||ƒD]/\}}t| | | |g|ƒ||7}qÍW|s»|jd"|jtjd#tjtjƒ|ƒƒfƒq»q»W|jƒ|jƒ| jƒ| jƒ| jƒ|jƒ|j|j|jkr®tjd$|ƒn|jdkrÐtjd%|ƒn|s¨d&|j|j}d&|j|j}|jd'ƒ|jd(|j|j||j|fƒ|j|j}|jd)|ƒtjd#tjtjƒ|ƒƒ} |jd*| ƒ|jd+t|j ƒƒn|| fS(,Nis.gzit mates_ICC4_twsnon-mates_ICC4_s(TGGACTCCACTGTG){e<=1}s(ACTTCGCCACTGTG){e<=1}s(TGAGTCCCACTGTG){e<=1}s(TGACTGCCACTGTG){e<=1}s(TCAGGTCCACTGTG){e<=1}s(ATGTCACCACTGTG){e<=1}s(GTATGACCACTGTG){e<=1}s(GTCTACCCACTGTG){e<=1}s(GTTGGACCACTGTG){e<=1}s(CGATTCCCACTGTG){e<=1}s(GGTTACCCACTGTG){e<=1}s(TCACCTCCACTGTG){e<=1}s(TCCAGACCAATGTG){e<=1}s(ACATCACCAATGTG){e<=1}s(TCACGACCAATGTG){e<=1}s(TAGCACCCAATGTG){e<=1}s(AACCTCCCAATGTG){e<=1}s(ACAACTCCAATGTG){e<=1}s(GTCTAACCAATGTG){e<=1}s(TACACGCCAATGTG){e<=1}s(GAGAACCCAATGTG){e<=1}s(GAGATTCCAATGTG){e<=1}s(GACCTACCAATGTG){e<=1}s(AGACTCCCAATGTG){e<=1}sglucigen_nxmate.py, chimera_clean: number of left reads (%d) is not equal to number of right reads (%d)!tn_jobsc3s'|]}ttƒ|ˆˆƒVqdS(N(RRc(t.0RI(RJRK(sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pys ssB==== chimera_clean progress: reads processed: %d, time elapsed: %ss%H:%M:%SsUlucigen_nxmate.py, chimera_clean: error in the script somewhere! Unequal read counts!sVlucigen_nxmate.py, chimera_clean: error in input data! Number of processed reads is 0!gY@s-==== chimera_clean info: processing finished!sn==== chimera_clean info: %d reads processed, %d true mate reads (%.2f %%) and %d non-mates/chimeras (%.2f %%).sB==== chimera_clean info: %d mates too short to keep after trimmings)==== chimera_clean info: time elapsed: %ss==== chimera_clean info: (!ttimetostpathtbasenametsplitexttjoinR RR;toptions_storagetthreadsRtlistt itertoolstislicetREADS_PER_BATCHRtsupportterrorR:R+RR2tinfoR<tstrftimetgmtimeR'R>R?R=tstrR@(!t infilename1t infilename2tdsttlogtsilentt starttimet basename1t basename2t outfilename1toutfile1t slagfilename1t slagfile1t outfilename2toutfile2t slagfilename2t slagfile2tparserR1tparserR2t all_statsRftreads1treads2tchunkstoutputsR$tresultsRLtresulttstatt percentmatest percentslagt shortmatest elapsedtime((RJRKsr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyt chimera_clean×s‚             ; 7         %t JunctionStatscBseZd„Zd„ZRS(cCsCd|_d|_d|_d|_d|_d|_d|_dS(Ni(R<t jctcountert splitcountertbothjctcountert R1jctcountert R2jctcountertR1R2jctcounter(R ((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR/s      cCs‚|j|j7_|j|j7_|j|j7_|j|j7_|j|j7_|j|j7_|j|j7_|S(N(R<RšR›RœRRžRŸ(R RA((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyRC8s(R(R)RRC(((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyR™.s cCs™tƒ}g}g}g}xh|D]`\}}|jd7_tj||dƒ}tj||dƒ} |rŠ| rŠ|jd7_|jƒ} | d} |d} | | } |d}|| }| jƒ}|d}|d}|| }|d}|| }t| ƒtkr‚t|ƒtkr‚|jd7_|j d7_ |j |d| |d|gƒ|j |d||d|gƒq‚q"| r]| jƒ}|d}|d}|| }|d}|| }t|ƒtkr‚|jd7_|j |d||d|gƒ|j |d|d|d|dgƒ|j d7_ |j d7_ q‚q"|r0|jƒ} | d} |d} | | } |d}|| }t| ƒtkr‚|jd7_|j |d| |d|gƒ|j |d|d|d|dgƒ|j d7_ |j d7_ q‚q"|j |d|d|d|dgƒ|j |d|d|d|dgƒq"W|||g|fS(Niiii(R™R<RERFRœRGRRHR›RŸRRšRžR(RItjctstrRLtprocessed_split1tprocessed_split2tprocessed_unsplitRQRRRUR6tmatcheststartRYtmyLefttmyQualt myLeftQualtnmatchestnstartR_tmyRight2Rat myRightQual2((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytnx_seq_junction_process_batchCsp              $!'      !)      !))-c s…tjƒ}tjj|ƒ}tjj|ƒddkrStjj|ƒd}ntjj|ƒ}tjj|ƒddkrštjj|ƒd}ntjj|d|ƒ}t|dƒ} tjj|d|ƒ} t| dƒ} tjj|d|jdd ƒƒ} t| dƒ} d ‰t|ƒ}t|ƒ}t ƒ}t j }x…t rÕt tj|tƒƒ}t tj|tƒƒ}t|ƒt|ƒkrÈtjd t|ƒt|ƒf|ƒn|sÒPntt t||ƒƒ|ƒ}td |ƒ‡fd †|Dƒƒ}g|D]}|d^qg|D]}|d^q6}}x=t||ƒD],\}}t| | | g|ƒ||7}q`W|sQ|jd|jtjdtjtjƒ|ƒƒfƒqQqQW|jƒ|jƒ| jƒ| jƒ| jƒ|jdkr*tjd|ƒn|jdkrLtjd|ƒn|sdd|j|j}d|j|j}d|j|j}d|j |j}|jdƒ|jd|jƒ|jd|j|fƒ|jd|j|fƒ|jd|j|fƒ|jd|j |fƒtjdtjtjƒ|ƒƒ}|jd|ƒn|jƒ|jƒ|| | fS(Nis.gzitR1_IJS7_RetR2_IJS7_t unsplit_IJS7_t_R1_t_R1R2_s"(GTTCATCGTCAGGCCTGACGATGAAC){e<=4}silucigen_nxmate.py, nx_seq_junction: number of left reads (%d) is not equal to number of right reads (%d)!Rfc3s$|]}ttƒ|ˆƒVqdS(N(RR­(RgRI(R (sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pys ­ssD==== nx_seq_junction progress: reads processed: %d, time elapsed: %ss%H:%M:%SsXlucigen_nxmate.py, nx_seq_junction: error in input data! Number of processed reads is 0!sTlucigen_nxmate.py, nx_seq_junction: error in input data! Number of split pairs is 0!ids/==== nx_seq_junction info: processing finished!s-==== nx_seq_junction info: %d reads processedsM==== nx_seq_junction info: %d total split pairs (%.2f %% of processed reads))sW==== nx_seq_junction info: %d junctions in both R1 and R2 (%.2f %% of split junctions))sX==== nx_seq_junction info: %d split junctions are in Read1 (%.2f %% of split junctions))sX==== nx_seq_junction info: %d split junctions are in Read2 (%.2f %% of split junctions))s+==== nx_seq_junction info: time elapsed: %s(!RhRiRjRkRlRmR treplaceRR™RnRoRRpRqRrRsRRtRuR:R+RR2RvR<RwRxR'R›RŸRRž( RzR{R|R}R~RR€Rtsplitfilenameleftt splitfile1tsplitfilenamerightt splitfile2tunsplitfilenamet unsplitfileRŠR‹RŒRfRRŽRRR$R‘RLR’R“t percentsplitt percentR1R2t percentR1t percentR2R—((R sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pytnx_seq_junction„s‚ %        ; 7          %  c Csƒ|jd|dtjj|ƒd|dƒt||||dtƒ\}}t||||dtƒ\}}}|||fS(Ns$== Processing Lucigen NxMate reads (s and s (results are in s directory)R~(RvRiRjRkR˜tFalseR¾( tleft_reads_fpathtright_reads_fpathR|R}tcleaned_filename1tcleaned_filename2tsplit_filename1tsplit_filename2tunsplit_filename((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyt process_readsÙs +!$($RiRhRtRRqtsystsiteRt spades_initRnREt ImportErrorRutext_python_modules_hometversionRtjoblib2RRtjoblib3tREADS_PER_THREADRoRsRHtobjectRR2R:R;RcRR˜R™R­R¾RÇ(((sr/oak/stanford/groups/akundaje/marinovg/programs/SPAdes-3.13.0-Linux/share/spades/spades_pipeline/lucigen_nxmate.pyts<           H   3 W A U