N>%Lc @spddlZddlZddlZddlmZddlmZdZdZddl Z ddl Z ddl Z ddl m Z e j dre j dZndZee _d Zd d eeed d d eeed d Zdeed eddZiegeddZd d eieed eed ZidedddZdddYZdS(iN(tstrftime(tarrayg@g?(tenvirontCISTEMATIC_TEMPs/tmpcCspyt|}Wn tk r2t|d}nXt|d}|jdtd||f|jdS(sZ create a log file to write a message from a messenger or append to an existing file. twtas %s: [%s] %s s%Y-%m-%d %H:%M:%SN(topentIOErrort writelinesRtclose(tlogFilet messengertmessagetlogfile((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytwriteLogs  iiiic -Cs$i} t|}|j}d}d}d| koGt|knr'g}x|D]}|ddkrd|krd}nd|krYd}qYqYn|jjd}|ddkrqYnt||j}|j|qYW|jd| } || }||kr'|}q'nd}t|}d}x`|D]X}|ddkrd|krqd}nd|krFd}qFqFn|jjd}|dkryt||j}Wn qFnX||krqFqn| rK||jd \}}|s|d }n|jd \}}t|}t|}n|dkrt j || d}|r||}n||d }t||d|}t||d |}n[|d}|r||}n||d }t||d|}t||d |}t ||} | ret|d ||}!t|d||}"n|| kr~g| |tsmerged %d timessreturning %d regions(Rt readlinestlentstriptsplittfloattappendtsorttinttstringtjointabstFalsetrangetTrueR (-tregionfilenametmaxDisttminHitstverboset keepLabelt fullChromt chromFieldt scoreFieldtpadtcompacttdoMergetkeepPeakt returnToptregionstinfiletlinest hasPvaluethasShifttscorestlinetfieldsthitstminScoret mergeCounttcounttchromtpostfronttbacktstarttstoptlabeltlengthtpeakPost peakHeighttmergedtindextrlabeltrstarttrstoptrlentrpeakPost rpeakHeighttmidpointt regionCount((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytgetMergedRegions#s  "                            # p       #    & #      iiKc Cs[tddg|} tddg|} d} d} g} |dkrd}d}x t|dD]}tddg|}x|D]}|d|}|ddkr||7}n ||8}|dks||krqn|r|d}nd }|ddkr||c|7 0 bp, e.g. X, return only the starting X bp from CDS. If lengthCDS < 0bp, return only the last X bp from CDS. isAgetLocusByChromDict: asked for no sequence - returning empty dictsRgetLocusByChromDict: asked for only upstream and downstream - returning empty dictsPgetLocusByChromDict: asked for partial CDS but not useCDS - returning empty dictsRgetLocusByChromDict: asked for TSS spanning and partial CDS - returning empty dictsigetLocusByChromDict: asked for discontinuous partial CDS from start and downstream - returning empty dictsfgetLocusByChromDict: asked for discontinuous partial CDS from stop and upstream - returning empty dictRXiRvRcSst|d|dS(Ni(R(RR((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyRRRwiRyi( tgenomeRzRRR R%R'R$tleftGeneDistancetrightGeneDistance(R|tupstreamt downstreamtuseCDSR}R~tupstreamSpanTSSt lengthCDSt keepSensetadjustToNeighbortlocusByChromDictRRRRARGRERFRHRRRRRRtgstarttgstoptglentdistancetnextGene((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytgetLocusByChromDictus        (                                     #! g?c "Csd} i} i} |r3d} d} d}d}nd} d}d} d}d}dGHt|dkrxY|D]}dg|| | %d s stop < %d is readID LIKE 's%' s mismatch != '' RXRs sense = '%s' s!select start, sense, sum(weight) s GROUP BY start, sense sselect ID, chrom, start, readIDs,stops,senses,weights,flags ,mismatchs LIMIT %d s from uniqs s UNION ALL s from multi s order by starts order by readID, start s order by chrom, startiiRRAis::t/RERFRRltflagtmismatchN(RRRtNoneRRRRRR!RRRRR((RR+tbothEndstnoSenseR-RAR&t withWeighttwithFlagt withMismatchtwithIDt withChromt withPairIDRRtfindallOptimizet readIDDicttreadLikeRERFtlimitt hasMismatchtflagLiketstrandRR RtgroupByt selectClauseRR R t currentChromt currentReadIDtpairIDRtdictKeyt theReadIDt multiplicitytnewrow((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt getReadsDicts                                    >          c!Csd}g}i}t}|dkr3||jksE|dksE|rNd}n|dkr||jkr|d|7}t}n|dkr|r|d7}n|r|d7}|d|7}|d7}n|d|7}t}n|r|r|d7}n|d 7}n|dkrC|dkr,|d7}nd}|d |7}n|d kr|dkrh|d7}nd}|d |7}n|d kr|dkr|d7}nd}|d |7}nd}|s|d7}n|r|d7}n|r|d7}n|r |d7}n|jr(|jj}n|jj}|dkr| rd|}|j|x\|D]1}|r|d}n|dd}g|| %d s stopR < %d s6select ID, chrom, startL, stopL, startR, stopR, readIDs, senses, weights, flags , mismatchs#select distinct chrom from splices RAis from splices s order by chrom, startLiRR%tstartLtstopLtstartRtstopRRRlR&R'ii( R%RR'RRRRRRR!RR(!RR+R*R-RAR&R+R,R-R.R/R0R2t splitReadR5R6RERFR7RR Rt conditionsR9R RR R;R<RR>R=R@((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytgetSplicesDictIs-                                    tbothc Cs d} d} d} d} |dkr1d|} n|rly"t|j|||| } Wqld} qlXn|ry"t|j|||| } Wqd} qXn|ry"t|j|||| } Wqd} qXn|r| | | } | S| | | fSdS(s0 return read counts for a given region. iRRXRs sense ='%s' N(RXR(RRRR(RRAtrmintrmaxRtmultitsplicestreportCombinedRRRRtrestrictR((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt getCountss2  " " " c Csd}d}|dksK|dkr3||jksK|dksK|dkrTd}n|dkr||jkr|d|d7}n|dkrt|dkr|d7}n|dt|7}n|dkrt|dkr|d7}n|d t|7}n|dkr@t|dkr3|d7}n||7}n|jr[|jj}n|jj}|r|jd |n|jd ||j} yt | d}Wn d}nX|S( s7 returns the number of row in the uniqs table. Ris where schrom="s" is and s start >= %ss start <= %ss3select count(distinct chrom+start+sense) from uniqssselect sum(weight) from uniqs( RRR{RRRRRtfetchoneR!( RRARJRKROtdistinctt whereclauseR@R tresult((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyRs<?           c Csd}d}|dksK|dkr3||jksK|dksK|dkrTd}n|dkr||jkr|d|d7}n|dkrt|dkr|d7}n|dt|7}n|dkrt|dkr|d7}n|d t|7}n|dkr@t|dkr3|d7}n||7}n|jr[|jj}n|jj}|jr|jj}n|jj}|jd ||j}yt |d}Wn d}nX|S( s9 returns the number of row in the splices table. Ris where schrom="s" is and s startL >= %ss startL <= %ssselect sum(weight) from splices( RRR{RRRRRRQR!( RRARJRKRORSR@R RT((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyR s>?            c Csd}d}|dksK|dkr3||jksK|dksK|dkrTd}n|dkr||jkr|d|d7}n|dkrt|dkr|d7}n|dt|7}n|dkrt|dkr|d7}n|d t|7}n|dkr@t|dkr3|d7}n||7}n|jr[|jj}n|jj}|jd ||j}yt |d}Wn d}nX|S( sD returns the number of distinct readIDs in the multi table. Ris where schrom="s" is and s start >= %ss start <= %ssselect sum(weight) from multi( RRR{RRRRRRQR!( RRARJRKRORSR@R RT((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyR5s8?           c Cs:d}d}|dkr%d|}n|r8|d7}n|rjt|dkr]|d7}n|d7}n|rt|dkr|d7}n|d7}n|d|7}|jr|jj}n|jj}|j||j} |rg| D]} | jd dd^qSg| D]} | d^q"Sd S( s get readID's. Ris LIMIT %d sselect readID from uniqs s union sselect readID from multi sselect readID from splices s group by readID R%N(RRRRRRRR( RRRLRMtpairedR4Rt limitPartR RTR((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt getReadIDsYs.          (c Csidd6dd6dd6dd6dd6}|j}|rG|g}n|j}|ji}x<|D]4}|rd|GHng||<|jdtd|d td td t} |ry|jd kry|jdtd|d td td t} | j} xu| D]j} | | d\} }}}}}|j d}x2|D]*}d|kr\qDnt |}|dkr|d}||d}t |d|d!}nN|dkr||d}|||d}|t |d|d!d}nt |t | d}d}t |t |kr=| |d}n||}||}||j | |||gqDWqWn|| krqjnx| |D]\}}}|j d}x|D]}d|krqnt |}|dkr|d}||d}t |d|d!}nN|dkrk||d}|||d}|t |d|d!d}n||d}||j ||||gqWqWqjW|S(sB returns the uniq and spliced mismatches in a dictionary. tTtAtCtGtNs%getting mismatches from chromosome %sR-RAR-R1R5RR2it,RXiR( RRR RAR'R%RRHRRRR!R(RtmischromR+t useSplicestrevcompR[t hitChromListtsnpDicttachromRt spliceDictt spliceIDListtktstartpostlefthalft rightstarttendsposRt mismatchestspMismatchListR't change_lent change_fromt change_baset change_post firsthalft secondhalft change_atREt mismatchList((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt getMismatchesxsp)       **         "  (       ")g?c Cs_|j} t| djdd} | d} t| } d|} |dkrw|j|d|d|| }n||| }tdd g|}|jd td |d td|d |d|dt}|dkrd}nx||D]\}}}x| D]y}|||}| s:|dkrQ|dkrQ||c| |7= ? and start < ? sGUPDATE multi SET flag = ? where chrom = ? and start >= ? and start < ? Rs\UPDATE splices SET flag = flag || ' L:' || ? where chrom = ? and startL >= ? and startL < ? s\UPDATE splices SET flag = flag || ' R:' || ? where chrom = ? and startR >= ? and startR < ? N(RRRR(Rt regionsListRRLRMRRO((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt flagReadss  cCsw|r|jjd|n|r:|jjd|n|jdkrf|rf|jjd|n|jjdS(sl set the flag fields in the entire dataset to clear. Useful for rerunning an analysis from scratch. sUPDATE uniqs SET flag = '%s'sUPDATE multi SET flag = '%s'RsUPDATE splices SET flag = '%s'N(RRRR(RR&RRLRM((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytsetFlags&scCsk|r|jjdn|r2|jjdn|jdkrZ|rZ|jjdn|jjdS(sn reset the flag fields in the entire dataset to clear. Useful for rerunning an analysis from scratch. sUPDATE uniqs SET flag = ''sUPDATE multi SET flag = ''RsUPDATE splices SET flag = ''N(RRRR(RRRLRM((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt resetFlags3scCs|jjd|dS(NsIUPDATE multi SET weight = ? where chrom = ? and start = ? and readID = ? (RR(RtreadList((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytreweighMultireads@stONcCs/y|jjd|Wnd|GHnXdS(NsPRAGMA SYNCHRONOUS = %ss-warning: couldn't set PRAGMA SYNCHRONOUS = %s(RR(RR((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytsetSynchronousPragmaCscCs5|jjd||r1|jjd|ndS(NsPRAGMA CACHE_SIZE = %dsPRAGMA DEFAULT_CACHE_SIZE = %d(RR(RRtdefault((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt setDBcacheIscCs|jr|jj}n|jj}|j||rM|j}|S|r||jrl|jjq||jjndS(N(RRRRRRR(RRR RR RT((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyRNs    cCs||jkr"|j|n|jd|jjddGH|jjddGH|jjddGH|jjdd GH|jd kr|jjd d GH|jjd dGH|jjddGHn|jj|jddS(s Builds the file indeces for the main tables. Cache is the number of 1.5 kb pages to keep in memory. 100000 pages translates into 150MB of RAM, which is our default. tOFFs-CREATE INDEX uPosIndex on uniqs(chrom, start)sbuilt uPosIndexs(CREATE INDEX uChromIndex on uniqs(chrom)sbuilt uChromIndexs-CREATE INDEX mPosIndex on multi(chrom, start)sbuilt mPosIndexs(CREATE INDEX mChromIndex on multi(chrom)sbuilt mChromIndexRs0CREATE INDEX sPosIndex on splices(chrom, startL)sbuilt sPosIndexs1CREATE INDEX sPosIndex2 on splices(chrom, startR)sbuilt sPosIndex2s*CREATE INDEX sChromIndex on splices(chrom)sbuilt sChromIndexRN(RRRRRRR(RR((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt buildIndex]s(  cCsy|jd|jjd|jjd|jjd|jjd|jdkr|jjdy|jjdWnnX|jjd n|jjWn d GHnX|jd d S( s5 drops the file indices for the main tables. RsDROP INDEX uPosIndexsDROP INDEX uChromIndexsDROP INDEX mPosIndexsDROP INDEX mChromIndexRsDROP INDEX sPosIndexsDROP INDEX sPosIndex2sDROP INDEX sChromIndexsproblem dropping indexRN(RRRRR(R((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyt dropIndexys"  c Csd|_tjd|_|j|j|jj}d}|dkrkd|GHd|}||_n d|_|jjd|jjd|jjd|jd}g}x)|D]!}|j|d |d fqW|jj d ||jd |}g}x_|D]W}|j|d |dt |dt |d|d|d|d|dfqW|jj d||jd|}g}x_|D]W}|j|d |dt |dt |d|d|d|d|dfqW|jj d||j dkr|jd|}g}xy|D]q}|j|d |dt |dt |dt |dt |d|d|d|d|df qCW|jj d|n|r|dkr5|jjd|jjd |j dkr|jjd!|jjd"qq|jjd#|jjd$|j dkr|jjd%|jjd&qnt |_ tj|j_|jjd'S((s makes a copy of the dataset into memory for faster access. Can be restricted to a "full" chromosome. Can also build the memory indices. Rs:memory:s memSync %ss where chrom = "%s" sPRAGMA temp_store = MEMORYsPRAGMA CACHE_SIZE = 1000000sdelete from metadatas select name, value from metadataRRs.insert into metadata(name, value) values (?,?)sKselect chrom, start, stop, sense, weight, flag, mismatch, readID from uniqsRRARERFRRlR&R'sninsert into uniqs(ID, readID, chrom, start, stop, sense, weight, flag, mismatch) values (NULL,?,?,?,?,?,?,?,?)sKselect chrom, start, stop, sense, weight, flag, mismatch, readID from multisninsert into multi(ID, readID, chrom, start, stop, sense, weight, flag, mismatch) values (NULL,?,?,?,?,?,?,?,?)Rs^select chrom, startL, stopL, startR, stopR, sense, weight, flag, mismatch, readID from splicesRBRCRDREsinsert into splices(ID, readID, chrom, startL, stopL, startR, stopR, weight, sense, flag, mismatch) values (NULL,?,?,?,?,?,?,?,?,?,?)s&CREATE INDEX uPosIndex on uniqs(start)s&CREATE INDEX mPosIndex on multi(start)s*CREATE INDEX sPosLIndex on splices(startL)s*CREATE INDEX sPosRIndex on splices(startR)s-CREATE INDEX uPosIndex on uniqs(chrom, start)s-CREATE INDEX mPosIndex on multi(chrom, start)s1CREATE INDEX sPosLIndex on splices(chrom, startL)s1CREATE INDEX sPosRIndex on splices(chrom, startR)N(RRRRRRRRRRR!RR'RRRR(RRARLRRSR tresults2R ((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pytmemSyncsb        U U o  N(1t__name__t __module__t__doc__R%R'RRRRRRRRRR RRRRRRRRR$RARHRPRRRRWR(RuRRRRRRRRRRRRRRRR(((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyRjsXH          " 'N?}! '($A5          ((trandomR"tsqlite3RttimeRRtcommoncodeVersionRRRRRtgettcisTempttempdirRR%R'RURuRRRR(((s9/oak/stanford/groups/akundaje/marinovg/code/commoncode.pyts$ $  -jY$^