o Uݢgko @sddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl Z ddlmZmZddlmZddlmZddlmZGddde ZGddde ZGd d d e ZGd d d ZGd ddeZGdddeZGdddeZ GdddZ!Gddde!Z"ej#$ej#%ej#%ej#%ej#&e'(ddZ)GdddZ*GdddZ+GdddZ,Gdd d e,Z-Gd!d"d"e,Z.Gd#d$d$e,Z/d.d(d)Z0Gd*d+d+eZ1Gd,d-d-e!Z2dS)/) annotationsN)Any NamedTuple) ensure_binary ensure_strc@s6eZdZUded<ded<ded<ded<ded<d S) Interval str | byteschromintstartendlengthz str | NonestrandN__name__ __module__ __qualname____annotations__rrc/oak/stanford/groups/akundaje/marinovg/programs/cellranger-9.0.1/lib/python/cellranger/reference.pyr  rc@s6eZdZUded<ded<ded<ded<d ed <d S) Generidstrname int | Noner float | None gc_contentzlist[Interval] | None intervalsNrrrrrr!rrc@s.eZdZUded<ded<ded<ded<d S) Transcriptrgenerr rrzlist[Interval]rNrrrrrr)s  rc@seZdZdZdddZdS) GtfParseraaThis is an old-style GtfParser class. It is used so that we can produce backwards compatible references that work with older releases of CR. For use within CR, use classes derived from `NewGtfParser`, below. In order for the `genes.pickle` file to unpickle in older CR builds, it must be an old-style class named `GtfParser`. FcCs|stddS)Nz_GtfParser is deprecated & is only present for backwards compatibility reasons. Use NewGtfParser) Exception)selfZskip_exceptionrrr__init__<s zGtfParser.__init__N)F)rrr__doc__r$rrrrr!0s r!cs eZdZdZfddZZS) GtfParseErrorz4Exception class for errors while parsing a GTF file.cstd|d|ddS)NzError while parsing GTF file  z& Please fix your GTF and start again.)superr$)r#filenamemsg __class__rrr$GszGtfParseError.__init__)rrrr%r$ __classcell__rrr+rr&Dsr&c@eZdZdZdS)BaseReferenceErrorz1Base class for all reference construction errors.Nrrrr%rrrrr/Mr/c@r.)GexReferenceErrorz>Exception class for errors while constructing a GEX reference.Nr0rrrrr2Qr1r2c@seZdZedZ  d*d+d d Zed d Zed,ddZ d-d.ddZ e d/d0d d!Z ed/d"d#Z ed1d&d'Z ed2d(d)ZdS)3 NewGtfParserz$(\S+?)\s+(".*?"|[^";\n\r]+)\s*;{0,1}NFr)os.PathLike | str | bytescontig_lengthsdict[str, int] | Noneno_transcript_failboolccsd}t|d[}tj|dd}d}t|D]C\}} t| dkr$q| ddr2| ddfVqt| d krIt|d t| |d d| ft | d } t | d } | | krlt|d|d d| d| d| |r| d|vrt t | } t|d|d d| dd| || d} | | krt|d|d d| d| d| dd| | d krt|d|d d| d| d| d }|tjvrt|d|d d| f| d}|j| d|d |dd}|dkr ||||| ||||| ||}nM|dkr?||||| ||||| ||}|r>||kr>t|d | d!|d d"|d#|d$ n|d%krV||}|durV|d&krV||=| d'|fVqWddS1siwYdS)(a[Return an iteator over the rows of a GTF. The iterator consists of (row, is_comment, properties) tuples of type (str, bool, dict). In case an entry's properties consist of multiple key-value pairs with the same key, the duplicate keys are made unique. To properly translate the properties dict back into GTF format, use format_properties_dict. Args: contig_lengths: dictionary of {contig: length}. If specified we check that start/end of gtf are consistent with contig lengths no_transcript_fail: should we accept a gtf that has no 'transcript' row, but has 'exon' rows that reference a transcript? Default is False because of backwards compatibility with cellranger. For ARC we exit since this is not valid gtf. transcript_idr ) delimiterNr#T z;Invalid number of columns (%d, expect 9) in GTF line %d: %szInvalid GTF annotation on line z: z Start position of feature = z > End position of feature = z,Invalid contig name encountered on GTF line z: z. The FASTA file has contigs: z End position of feature = z > contig z length = z < 1z!Invalid strand in GTF line %d: %s) uniquify_keys transcriptexonz"Supplied GTF is invalid. This row z on line z1 specifies an 'exon' annotation for a transcript z2, but there is no 'transcript' row in the GTF for z that immediately precedes it.r F)cr_ioZopen_maybe_gzipcsvreader enumeratelen startswithr&joinr sortedlistkeysencode cr_constantsSTRANDSget_properties_dictvalidate_transcript_idvalidate_gene_idget)r#r)r5r7r9frKZtranscript_row_idirowr r ZcontigsZmax_lenr annotation propertiestidrrrgtf_reader_iterYs                 $zNewGtfParser.gtf_reader_itercCsd|vrt|d|dd|f|ddkr&t|d|dd|d|dvr:t|d|dd|ftd |ddurRt|d |dd|fdS) Nr9z5Property 'transcript_id' not found in GTF line %d: %sr?r;rHz6Property 'transcript_id' is empty in GTF line {:d}: {};zEProperty 'transcript_id' has invalid character ';' in GTF line %d: %sz\szLProperty 'transcript_id' has invalid whitespace character in GTF line %d: %s)r&rOformatresearchr)r^r[r\rrrrWs6  z#NewGtfParser.validate_transcript_idr^dict[str, str | int]cCsxd|vrt|d|dd|f|ddkr&t|d|dd|d|dvr:t|d|dd|fdS) Ngene_idz/Property 'gene_id' not found in GTF line %d: %sr?r;rHz0Property 'gene_id' is empty in GTF line {:d}: {}raz?Property 'gene_id' has invalid character ';' in GTF line %d: %s)r&rOrbrerrrrXs$  zNewGtfParser.validate_gene_id in_gtf_fn fasta_parserFastaParser | Nonereturn(tuple[dict[str, Transcript], list[Gene]]c si}t}||D]t\}}}|rq |\ }} } } } } } } } | dkr#q t| d} t| } t| | }|d}|d}|d|}t||ddd}||vrVt|ddg||<||vr_t||<||j |kshJ||j t t || | || |||q ii|D]\}}tdd|j D|<|dur|||<qg}|D]\}}tfdd |D}|durtfd d |Dntd }g}|D] }|||j 7}q|jd d dg}t|dd D])\}}t|}tdd|D} tdd|D} t || | | | d}| |qt|j|j|||}| ||D]}t|||td ||j ||<q)q||fS)NrGr?r9rg gene_namecs|]}|jVqdSNr .0intervalrrr s z(NewGtfParser.load_gtf..cg|]}|qSrrrrr9)transcript_lengthsrr 'z)NewGtfParser.load_gtf..crurrrv)transcript_gc_contentsrrrx+rynancS|jSror rsrrr4z'NewGtfParser.load_gtf..keycSr|ror}r~rrrr8rcsrnro)r rqrrrrt;csrnro)r rqrrrrt<r) collections OrderedDictr`r absrYrrsetr rappendrradditemssumget_transcript_gc_contentnpmedianfloatsort itertoolsgroupbyrQminmaxrr)r#rhri transcriptsZgene_to_transcriptsr\ is_commentr^r _r]r r rr r9rgrmr rFgenesZtranscript_idsrZtranscript_intervalsrZchrom_intervals_iterZchrom_intervalsrsr)rzrwrload_gtfs          zNewGtfParser.load_gtfTproperties_strr line_numberr rEdict[str, int | str]c Cs"t|tr|St}|rd}tj|D]y}|d}|d}|r%|s&q|ddk} |ddk} | r=| r=|dd}n0| rHt|d||f| rSt|d||f| r\t |}nd|vsdd|vrmt|d||fd|vrzt|d ||f|r||vr|d7}t ||}|||<q|S) a.Parse the properties present in the 9th column of a gtf entry into a dictionary. If there are multiple properties with the same key, those keys will be made unique unless uniquify_keys is False, in which case only the final instance of a key will appear in the result. rr?rC"zError parsing GTF at line %d. Parsed attribute began, but did not end with a quote. Please ensure attributes that start with quotes end with them. Bad Attribute = %szError parsing GTF at line %d. Parsed attribute ended, but did not begin with a quote. Please ensure attributes that end with quotes end start with them. Bad Attribute = %szError parsing GTF at line %d. Parsed attribute had a quote in the middle of a value. Please ensure quotes are only used to encapsulate attribute values. Bad Attribute Value = %szError parsing GTF at line %d. Parsed attribute had a quote in the middle of a key. Please ensure quotes are only used to encapsulate attribute values. Bad Attribute Key = %s) isinstancedictrrr3_attribute_patternfinditergroupr&isdigitr _make_key_unique) rrr)rEr^Z repeat_nummrvalueZ start_quoteZ end_quoterrrrVMsb         z NewGtfParser.get_properties_dictcCsfg}|D]%\}}t|trt|}nd|d}|r!t|}||d|qd|dS)aTranslate a properties dict into a GTF-formatted string. Note: Keys beginning with "GtfParser_" are assumed to come from `_make_key_unique` and will be translated. This behavior can be disabled by setting `uniquify_keys=False`. r z; ra)rrr rr3_translate_unique_keyrrO)r^rErrrrrrformat_properties_dicts     z#NewGtfParser.format_properties_dictrnumcCsd|d|S)N GtfParser_rr)rrrrrrszNewGtfParser._make_key_uniquecCs |dr|d|dS|S)Nr r)rNrindexrrrrrs z"NewGtfParser._translate_unique_key)NF)r)r4r5r6r7r8)r^rfro)rhr4rirjrkrl)T) rrrr r)rrEr8rkr)rrrr rkr)rrrkr)rrrrccompilerr` staticmethodrWrXrrVrrrrrrrr3Us(  m   YF  r3c@seZdZdddZddZdS) GtfBuilderrcCs||_||_|p i|_dSro)rh out_gtf_fn attributes)r#rhrrrrrr$szGtfBuilder.__init__c Cstdt|jdM}tj|dtjddd}||jD]3\}}}|r)||qd}|dus1J| D]\}}||j vrG||j |vrGd}q5|sO||qWdn1sZwYtd dS) NzLWriting new genes GTF file (may take 10 minutes for a 1GB input GTF file)...wr;rHr')r<quoting quotecharlineterminatorFTz...done ) printopenrrJwriter QUOTE_NONEr`rhwriterowrr) r#rZrr\rr^removerrrrr build_gtfs*     zGtfBuilder.build_gtfN)r)rrrr$rrrrrrs  rsbinsgtf_to_gene_indexc@sVeZdZdddZeddd Zedd d ZdddZej fd ddZ d!ddZ dS)" FastaParser in_fasta_fnr4cCs|||_dSro) load_fastachroms)r#rrrrr$zFastaParser.__init__ fasta_headerbytesrkcCstd|}|ddS)Ns>(\S+)r?r)rcrdgroups)rmatchrrr_get_chrom_names zFastaParser._get_chrom_namedict[bytes, bytes]cCsi}d\}}t|d4}|D])}|}|dr2|dur*|dus#Jd|||<t|}g}q||qWdn1sBwY|durX|dusQJd|||<|S)NNNrb>)rstriprNrOrrr)rrZ current_chromZ current_seqrZlinerrrrs$      zFastaParser.load_fastar r r r cCs2t|tsJ||jvo|dko|t|j|kS)zSDetermine whether the half-open interval [start,end) is within the bounds of chrom.r)rrrrM)r#r r r rrris_valid_intervals$zFastaParser.is_valid_intervalrcCsPt|tsJ|j|||}|tjkr|S|tjkr!t|Std|)zlz)NewGeneIndex.__init__..)rhrrrrrrLr)r#rhrrirrrr$Ws    zNewGeneIndex.__init__reference_pathrc Csd}tt||g}z tj|tjdWntjy*}z tdt|j|d}~wwzt |Wt j |r=t |SSt j |rJt |ww)Load the GeneIndex directly from the reference path. Directly parses the GTF and FASTA files using helper Rust tool. Does not require the gene.pickle file. sgene_index.json)stderrzcouldn't load reference info: N)_GTF_TO_GENE_INDEXr subprocess check_outputSTDOUTCalledProcessError RuntimeErrorroutputrload_from_jsonospathexistsunlink)r gene_index_fncmdexcrrrload_from_referenceos    z NewGeneIndex.load_from_referencerstr | bytes | os.PathLikerkcsdfdd dd d dfd d dfdd }t| }t|}Wdn1s/wYt}|d|_|d|_||d|_fdd|dD|_ddt|jD|_ |S)robjdict[str, Any]rkrcsBfdd|dD}tt|dt|d|d|d|S)Ncg|]}|qSrrrrx conv_intervalrrrxryzBNewGeneIndex.load_from_json..conv_gene..rrrr r)rrrS)rrr rr conv_genes z.NewGeneIndex.load_from_json..conv_genedict[str, str | int | None]rcSsB|dr t|d}nd}tt|d|d|d|d|S)Nrr r r r )rrrS)rsrrrr s(z2NewGeneIndex.load_from_json..conv_intervalrcsN|d}fdd|dD}|djdkr|t||d|d|S) Nr crrrrr rrrxryzHNewGeneIndex.load_from_json..conv_transcript..rr-r r)rreverser)rr r)r r rrconv_transcripts z4NewGeneIndex.load_from_json..conv_transcriptdict[str, dict[str, Any]]dict[str, Transcript]cs.i}|D]\}}|}||t|<q|Sro)rr)rrestx_idtxZnew_tx)rrrconv_transcriptss z5NewGeneIndex.load_from_json..conv_transcriptsNrhrrcrrrr)r rrrxryz/NewGeneIndex.load_from_json..rcSrrrrrrrrrz/NewGeneIndex.load_from_json..)rrrkr)rr rkr)rrrkr)rrrkr) rjsonloadrrhrrrrLr)rrrZdatarr)r r rrrs     zNewGeneIndex.load_from_jsonrFrcC(t|tsJ||jvr|j|jSdSro)rrrr r#rFrrrget_transcript_length  z"NewGeneIndex.get_transcript_lengthcCrro)rrrrrrrrrrz&NewGeneIndex.get_transcript_gc_contentcCrro)rrrr rrrrget_gene_from_transcriptrz%NewGeneIndex.get_gene_from_transcriptrgrrcCs&t|tsJ||jvr|j|SdS)z?Return the integer index of gene_id if found or None otherwise.N)rrr)r#rgrrrgene_id_to_ints  zNewGeneIndex.gene_id_to_intcCr|rorr#rrr get_genesszNewGeneIndex.get_genes Gene | NonecCs<t|tsJ||}|dur|t|jkr|j|SdS)zBReturn the Gene object whose gene ID is gene_id or None otherwise.N)rrr rMr)r#rgindexrrrget_genes $zNewGeneIndex.get_gene list[int]cCdd|jDS)NcSg|]}|jqSrrprrr rrrrxz1NewGeneIndex.get_gene_lengths..r!r"rrrget_gene_lengthsrzNewGeneIndex.get_gene_lengthscCr()NcSr)r)rr*rrrrxr+z5NewGeneIndex.get_gene_gc_contents..r!r"rrrget_gene_gc_contentsrz!NewGeneIndex.get_gene_gc_contentscCr()NcSr)r)rrrrrrxr+z/NewGeneIndex.get_gene_names..r!r"rrrget_gene_namesrzNewGeneIndex.get_gene_namescCr()NcSr)rrrrrrrxr+z-NewGeneIndex.get_gene_ids..r!r"rrr get_gene_idsrzNewGeneIndex.get_gene_idscsNfdd|jD|_fdd|jD}||_ddt|jD|_dS)z.cs"i|] \}}|jjvr||qSr)r r)rrrrr1rrrs"z-NewGeneIndex.subset_index..cSrrrrrrrrrN)rrrrLr)r#r2Ztxsrr1r subset_indexszNewGeneIndex.subset_indexr)rhrrr)rr)rrrkr)rFr)rgrrkr)rgrrkr$)rkr')rrrr$rrrrrrr r#r&r,r-r.r/r3rrrrrVs&    3      r)rrrr)3 __future__rrrJrrrrcrtypingrrnumpyrsixrrcellranger.constants constantsrTZcellranger.cr_iorIZ tenkit.seqrrrrrr!r"r&r/r2r3rrrOdirnameabspath__file__rSrrrrrrrrrrrrrrsN     c&F