h> UdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlZddlZddlZddlZddlZddlmZmZddlmZded eefd Zded e ege fd eee ffd Zded e ege fd eee e d fffdZgdZeee d<ej!ej"ej"ej!ej#ej!ej$ej$ej$ej"g Z%ee e d<dede ede&d ej'fdZ(ej)gddZ*eje d<ej+fdede d eedffdZ,dej'ded eede&d e eed!feed"fff d#Z-d$eede&d e eed!feed!fffd%Z.d$eede&d e eed!feed!fffd&Z/ded efd'Z0d(eed)ede&d e eed!feed!fffd*Z1d+ed e eed!fe eed!feed"ffej'e2ffd,Z3 dqd-eed!fd.e eed!feed"ffd/ed0e ej'd df d1Z4d2eed3fd4e5d e e&e&ffd5Z6drd7eed3fd8e5d eed3ffd9Z7d:e ee fd e e e&e&fe e&efffd;Z8ded?e eee e&e&ffd@e eee5fdAe5dBedCe eedDe eeefdEe eee5fdFe5dGe2d e ej'eedHfeedIfeffdJZ: dsd>ed0ej'dLej'de&dMe&dNe2d e ej'ej;ffdOZe>dSej#izZ?dKe>fdTedNe2dUeee fd e ej'ej;ffdVZ@e>fdWe ej'ej;fd/edUe eee fd dfdXZAdWe ej'ej;fd0ej'dLej'dYej'dZed[e&d dfd\ZBdYej'd0ej'd/ed dfd]ZCdLej'd/ed dfd^ZDej#ej!ej!ej!ej#ej#ej$ej$d_ZEd`ed e ej'effdaZFdbeedHfd/ed dfdcZGdded eedHffdeZHdfeee fd/ed dfdgZIdhed eee ffdiZJdjej'd/ed dfdkZKdlej'd/ed dfdmZLdnej'doeeeeeffdZed dfdpZMdS)taData input/output module for the Fi-NeMo motif instance calling pipeline. This module handles loading and processing of various genomic data formats including: - Peak region files (ENCODE NarrowPeak format) - Genome sequences (FASTA format) - Contribution scores (bigWig, HDF5 formats) - Neural network model outputs - Motif data from TF-MoDISco - Hit calling results The module supports multiple input formats used for contribution scores and provides utilities for data conversion and quality control. N) ExitStack)ListDictTupleOptionalAnyUnionCallable)ndarray)FloatInt)tqdmpathreturncg}t|5}|D]E}|ddd}||F dddn #1swxYwY|S)zLoad a text file containing one item per line. Parameters ---------- path : str Path to the text file. Returns ------- List[str] List of strings, one per line (first column if tab-delimited).   rN)openrstripsplitappend)rentriesflineitems 5/srv/www/kundaje/kobbad/Fi-NeMo/src/finemo/data_io.pyload_txtr!sG d!q ! !D;;t$$**4003D NN4  !!!!!!!!!!!!!!!! NsA A((A,/A, value_typeci}t|5}|D];}|dd\}}||||<< dddn #1swxYwY|S)arLoad a two-column tab-delimited mapping file. Parameters ---------- path : str Path to the mapping file. Must be tab-delimited with exactly two columns. value_type : Callable[[str], Any] Type constructor to apply to values (e.g., int, float, str). Must accept a string and return the converted value. Returns ------- Dict[str, Any] Dictionary mapping keys to values of the specified type. Raises ------ ValueError If lines don't contain exactly two tab-separated values. FileNotFoundError If the specified file does not exist. rrN)rrr)rrmappingrrkeyvals r load_mappingr#7s.G d+q + +D{{4((..t44HC%:c??GCLL ++++++++++++++++ Ns?AA"%A".ci}t|5}|D]Z}|dd}|d}|dd}tfd|D||<[ dddn #1swxYwY|S)aLoad a mapping file where values are tuples from multiple columns. Parameters ---------- path : str Path to the mapping file. Must be tab-delimited with multiple columns. value_type : Callable[[str], Any] Type constructor to apply to each value element. Must accept a string and return the converted value. Returns ------- Dict[str, Tuple[Any, ...]] Dictionary mapping keys to tuples of values of the specified type. The first column is used as the key, remaining columns as tuple values. Raises ------ ValueError If lines don't contain at least two tab-separated values. FileNotFoundError If the specified file does not exist. rrrNc3.K|]}|VdSN).0irs r z%load_mapping_tuple..ws+ < <1A < < < < < <)rrrtuple)rrr rrrr!r"s ` rload_mapping_tupler.Ws4G d=q = =Dkk$''--d33G!*C!""+C < < < < < < <<>BB) chr peak_startpeak_end peak_name peak_score peak_strand peak_signal peak_pval peak_qval peak_summitNARROWPEAK_SCHEMANARROWPEAK_DTYPES peaks_pathchrom_order_path half_widthc tj|dtddtgdtjdtjdtjdz|z tjd  d }|t|}ng}t|fd | d dD}| |dt|D}|tjd|d}|S)aLoad peak region data from ENCODE NarrowPeak format file. Parameters ---------- peaks_path : str Path to the NarrowPeak format file. chrom_order_path : str, optional Path to file defining chromosome ordering. If None, uses order from peaks file. half_width : int Half-width of regions around peak summits. Returns ------- pl.DataFrame DataFrame containing peak information with columns: - chr: Chromosome name - peak_region_start: Start coordinate of centered region - peak_name: Peak identifier - peak_id: Sequential peak index - chr_id: Numeric chromosome identifier FrN).NAnullNaN) has_header new_columns separator quote_charschema_overrides null_valuesr/r0r8r2)r/peak_region_startr2peak_idnamecg|]}|v| Sr(r()r)r*chrom_order_sets r zload_peaks..s. O # # # # #r,T)maintain_orderci|]\}}|| Sr(r()r)indr"s r zload_peaks..sEEE(#sS#EEEr,chr_id)plscan_csvr9r:selectcolwith_row_indexcollectrset get_columnuniqueextend enumerate with_columnsreplace_strictalias)r;r<r=peaks chrom_orderchrom_order_peaks chrom_ind_maprNs @r load_peaksrgs2 ).222    u  f\22RVM5J5JJZWf[))   Y ' ' ! &#/00  +&&O!!%((//t/DD ()))EEi .D.DEEEM    u $$]3399(CC  E Lr,)ACGTS1dtype SEQ_ALPHABETsequencernz4 Lc|}tj|dd}|dddftdddfk|}|S)adConvert DNA sequence string to one-hot encoded matrix. Parameters ---------- sequence : str DNA sequence string containing A, C, G, T characters. dtype : np.dtype, default np.int8 Data type for the output array. Returns ------- Int[ndarray, "4 L"] One-hot encoded sequence where rows correspond to A, C, G, T and L is the sequence length. Notes ----- The output array has shape (4, len(sequence)) with rows corresponding to nucleotides A, C, G, T in that order. Non-standard nucleotides (N, etc.) result in all-zero columns. zUTF-8rlrmN)uppernp frombufferencoderoastype)rprn seq_chararrayone_hots rone_hot_encoderysj,~~HM(//'":":$GGGMT111W%aaag)>>FFuMMG Nr,rcfa_pathbw_pathszN 4 LzN Lc |j}|dz}tj|d|ftj}tj||ftj}t j|d}d|D} tjt||dzftj} tt| dd d | D]\} } | d } | d }|d|zz}|| ||}|j }|j }|j }||z }||z }||kr}t||| d d ||f<t| D]7\}}tj|| ||d| |d d f<8tj| d|| ||f< | D]}|n#| D]}|wxYw||fS)aLoad genomic sequences and contribution scores from FASTA and bigWig files. Parameters ---------- peaks : pl.DataFrame Peak regions DataFrame from load_peaks() containing columns: 'chr', 'peak_region_start'. fa_path : str Path to genome FASTA file (.fa or .fasta format). bw_paths : List[str] List of paths to bigWig files containing contribution scores. Must be non-empty. half_width : int Half-width of regions to extract around peak centers. Total region width will be 2 * half_width. Returns ------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of peaks, 4 represents A,C,G,T nucleotides, and L is the region length (2 * half_width). contribs : Float[ndarray, "N L"] Contribution scores averaged across input bigWig files. Shape is (N peaks, L region_length). Notes ----- BigWig files only provide projected contribution scores, not hypothetical scores. Regions extending beyond chromosome boundaries are zero-padded. Missing values in bigWig files are converted to zero. rmF)one_based_attributesc6g|]}tj|Sr()pyBigWigrr)r*s rrOz(load_regions_from_bw..$s" . . .8=   . . .r,T)namedNregions)disableunittotalr/rI)numpyraxis)heightrszerosint8float16pyfaidxFastalenrr_ iter_rowsseqstartendry nan_to_numvaluesmeanclose)rcrzr{r= num_peaks region_width sequencescontribsgenomebwscontrib_bufferrRrowchromrr sequence_datarp start_adjend_adjabjbws rload_regions_from_bwrsCD I>L)Q 5RWEEEIxL1DDDH]7 ? ? ?F . .X . . .CXs8}}j1n=RZPPPN eooDo11 2 2     E EHC JE+,E!j.(C17uSy1IM)-H*0I(,GE!A%A1uu)7)A)A #qqq!A#+&&s^^EAr+-= %G4 HH,,N1aaa4((&(W^!%D%D%Dac"3 E8  B HHJJJJ #  B HHJJJJ  h s 'DGG!h5_pathsct5fd|D}|ddjddz|z d|zz|ddddddftj}tjfd|Ddtj}dddn #1swxYwY||fS) aLoad genomic sequences and contribution scores from ChromBPNet HDF5 files. Parameters ---------- h5_paths : List[str] List of paths to ChromBPNet HDF5 files containing sequences and SHAP scores. Must be non-empty and contain compatible data shapes. half_width : int Half-width of regions to extract around the center. Total region width will be 2 * half_width. Returns ------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of regions, 4 represents A,C,G,T nucleotides, and L is the region length (2 * half_width). contribs : Float[ndarray, "N 4 L"] SHAP contribution scores averaged across input files. Shape is (N regions, 4 nucleotides, L region_length). Notes ----- ChromBPNet files store sequences in 'raw/seq' and SHAP scores in 'shap/seq'. All input files must have the same dimensions and number of regions. Missing values in contribution scores are converted to zero. c^g|])}tj|*Sr( enter_contexth5pyFiler)r*stacks rrOz3load_regions_from_chrombpnet_h5..h/CCCQu""49Q<<00CCCr,rzraw/seqr}Nc bg|]+}tj|dddddf,S)zshap/seqN)rsrr)rrrs rrOz3load_regions_from_chrombpnet_h5..os> H H HqR]1Z=AAAuSy9 : : H H Hr,rrn)rshapervrsrrrrr=h5srrrrrs @@@rload_regions_from_chrombpnet_h5rJs$:   CCCC(CCCAy!'+q0:=a*n$F9%aaaE#Io6==bgFF 7 H H H H HC H H H*                    h sBB<<CCct5fd|D}|ddjddz|z d|zz|ddddddfddtj}t jfd|Ddtj }dddn #1swxYwY||fS) azLoad genomic sequences and contribution scores from BPNet HDF5 files. Parameters ---------- h5_paths : List[str] List of paths to BPNet HDF5 files containing sequences and contribution scores. Must be non-empty and contain compatible data shapes. half_width : int Half-width of regions to extract around the center. Total region width will be 2 * half_width. Returns ------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of regions, 4 represents A,C,G,T nucleotides, and L is the region length (2 * half_width). contribs : Float[ndarray, "N 4 L"] Hypothetical contribution scores averaged across input files. Shape is (N regions, 4 nucleotides, L region_length). Notes ----- BPNet files store sequences in 'input_seqs' and hypothetical scores in 'hyp_scores'. The data requires axis swapping to convert from (n, length, 4) to (n, 4, length) format. All input files must have the same dimensions and number of regions. Missing values in contribution scores are converted to zero. c^g|])}tj|*Sr(rrs rrOz.load_regions_from_bpnet_h5..rr,r input_seqsr}Nr%c g|]?}tj|dddddfdd@S) hyp_scoresNr%r})rsrswapaxesrs rrOz.load_regions_from_bpnet_h5..s^    a oaaasAAAo>GG1MMNN   r,r)rrrrvrsrrrrs @@@rload_regions_from_bpnet_h5rwsI<  CCCC(CCCA|$*2.!3j@a*n$F<(E#Iqqq9BB1aHHOOPRPWXX 7        *                   h sB2CCCcxtj|}t|tjr|}n|d}|S)aLoad array data from .npy or .npz file. Parameters ---------- path : str Path to .npy or .npz file. File must exist and contain valid NumPy data. Returns ------- ndarray Loaded array data. For .npz files, returns the first array ('arr_0'). For .npy files, returns the array directly. Raises ------ FileNotFoundError If the specified file does not exist. KeyError If .npz file does not contain 'arr_0' key. arr_0)rsload isinstancer )rrarrs rload_npy_or_npzrs:*  A!RZ  j Jr, shaps_pathsohe_pathct|}|jddz|z d|zz|ddddftj}fd|D}tj|dtj}||fS)aLoad genomic sequences and contribution scores from TF-MoDISco format files. Parameters ---------- shaps_paths : List[str] List of paths to .npy/.npz files containing SHAP/attribution scores. Must be non-empty and all files must have compatible shapes. ohe_path : str Path to .npy/.npz file containing one-hot encoded sequences. Must have shape (n_regions, 4, sequence_length). half_width : int Half-width of regions to extract around the center. Total region width will be 2 * half_width. Returns ------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of regions, 4 represents A,C,G,T nucleotides, and L is the region length (2 * half_width). contribs : Float[ndarray, "N 4 L"] SHAP contribution scores averaged across input files. Shape is (N regions, 4 nucleotides, L region_length). Notes ----- All SHAP files must have the same shape as the sequence file. Missing values in contribution scores are converted to zero. The center of the input sequences is used as the reference point for extraction. rr}Nc pg|]2}tjt|ddddf3Sr')rsrr)r)prrs rrOz1load_regions_from_modisco_fmt..sB U U UAR]?1--aaaE#Io> ? ? U U Ur,rr)rrrvrsrrr) rrr= sequences_rawrshapsrrrs @@rload_regions_from_modisco_fmtrs@$H--M   #q (: 5E !j. CaaaE#Io.55bg>>I U U U U U U U UEwu1BJ777H h r,npz_pathcjtj|}d|vrtjdd}|djd}tjdg|zdtj|tjtj |tj tj|tjtjdg|zdd }n(d }|d|d |d |d |dd }tj |}|d|d||fS)aFLoad preprocessed genomic regions from NPZ file. Parameters ---------- npz_path : str Path to NPZ file containing sequences, contributions, and optional coordinates. Must contain 'sequences' and 'contributions' arrays at minimum. Returns ------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of regions, 4 represents A,C,G,T nucleotides, and L is the region length. contributions : Union[Float[ndarray, "N 4 L"], Float[ndarray, "N L"]] Contribution scores in either hypothetical format (N, 4, L) or projected format (N, L). Shape depends on the input data format. peaks_df : pl.DataFrame DataFrame containing peak region information with columns: 'chr', 'chr_id', 'peak_region_start', 'peak_id', 'peak_name'. has_peaks : bool Whether the file contains genomic coordinate information. If False, placeholder coordinate data is used. Notes ----- If genomic coordinates are not present in the NPZ file, creates placeholder coordinate data and issues a warning. The placeholder data uses 'NA' for chromosome names and sequential indices for peak IDs. Raises ------ KeyError If required arrays 'sequences' or 'contributions' are missing from the file. r/zaNo genome coordinates present in the input .npz file. Returning sequences and contributions only.Frrr@Urm)r/rTrIrJr2TrTrrJr2 contributions) rsrkeyswarningswarnrarrayarangeuint32rint32rU DataFrame)rdata has_peaks num_regions peak_datapeaks_dfs rload_regions_npzrs4T 78  D DIIKK o    ;'-a0 8TF[0<<<i 29===!#+RX!F!F!FyBI>>>4&;"6cBBB    ;8n!%gIk*   |I&&H  d?3Xy HHr,rrout_pathrc |-tjdtj|||dS|j}||jdks||jdkr"t d|jd|jd||d d }|d }|d }|d }|d  d } tj|||||||| dS)aWrite genomic regions and contribution scores to compressed NPZ file. Parameters ---------- sequences : Int[ndarray, "N 4 L"] One-hot encoded DNA sequences where N is the number of regions, 4 represents A,C,G,T nucleotides, and L is the region length. contributions : Union[Float[ndarray, "N 4 L"], Float[ndarray, "N L"]] Contribution scores in either hypothetical format (N, 4, L) or projected format (N, L). out_path : str Output path for the NPZ file. Parent directory must exist. peaks_df : Optional[pl.DataFrame] DataFrame containing peak region information with columns: 'chr', 'chr_id', 'peak_region_start', 'peak_id', 'peak_name'. If None, only sequences and contributions are saved. Raises ------ ValueError If the number of regions in sequences/contributions doesn't match peaks_df. FileNotFoundError If the parent directory of out_path does not exist. Notes ----- The output file is compressed using NumPy's savez_compressed format. If peaks_df is provided, genomic coordinate information is included in the output file for downstream analysis. NzINo genome coordinates provided. Writing sequences and contributions only.)rrrzInput sequences of shape z% and/or input contributions of shape z. are not compatible with peak region count of r/rrTrIrJr2)rrr/rTrrJr2) rrrssavez_compressedrr ValueErrorr\to_numpyrv) rrrrrchr_arr chr_id_arr start_arr peak_id_arr peak_name_arrs rwrite_regions_npzr<sH W    H WWWWWWo 9?1- - - =.q1 1 1NIONN0=0CNN@KNN  %%e,,5577>>sCC((22;;== ''(;<<EEGG )))44==??  ++K88AACCJJ3OO   '# r,cwmz4 Wtrim_thresholdctjtj|d}tj||z}tj||k}tt tj|d}t t tj|dzt|}||fS)arDetermine trimmed start and end positions for a motif based on contribution magnitude. This function identifies the core region of a motif by finding positions where the total absolute contribution exceeds a threshold relative to the maximum. Parameters ---------- cwm : Float[ndarray, "4 W"] Contribution weight matrix for the motif where 4 represents A,C,G,T nucleotides and W is the motif width. trim_threshold : float Fraction of maximum score to use as trimming threshold (0.0 to 1.0). Higher values result in more aggressive trimming. Returns ------- start : int Start position of the trimmed motif (inclusive). end : int End position of the trimmed motif (exclusive). Notes ----- The trimming is based on the sum of absolute contributions across all nucleotides at each position. Positions with contributions below trim_threshold * max_score are removed from the motif edges. Adapted from https://github.com/jmschrei/tfmodisco-lite/blob/570535ee5ccf43d670e898d92d63af43d68c38c5/modiscolite/report.py#L213-L236 rrr%)rssumabsmaxnonzerointminr)rrscore trim_thresh pass_indsrrs r trim_motifrs< F26#;;Q ' ' 'E&--.0K 5K/00I BF9%%&& * *E c"&##$$q(#e** 5 5C #:r,dxtempc|tj|ddz }tj||z}|tj|ddz S)aApply softmax transformation with temperature scaling. Parameters ---------- x : Float[ndarray, "4 W"] Input array to transform where 4 represents A,C,G,T nucleotides and W is the motif width. temp : float, default 100 Temperature parameter for softmax scaling. Higher values create sharper probability distributions. Returns ------- Float[ndarray, "4 W"] Softmax-transformed array with same shape as input. Each column sums to 1.0, representing nucleotide probabilities at each position. Notes ----- The softmax is applied along the nucleotide axis (axis=0), normalizing each position to have probabilities that sum to 1. The temperature parameter controls the sharpness of the distribution. r%Trkeepdimsr)rsrexpr)rrnorm_xrs rsoftmaxrsN0T222 2F &  C !d333 33r,rc&|d} dt|ddfS#ttf$rK dt|ddfcYS#ttf$r d|fcYcYSwxYwwxYw)aGenerate sort key for TF-MoDISco motif names. This function creates a sort key that orders motifs by pattern number, with non-standard patterns sorted to the end. Parameters ---------- data : Tuple[str, Any] Tuple containing motif name as first element and additional data. The motif name should follow the format 'pattern_N' or 'pattern#N' where N is an integer. Returns ------- Union[Tuple[int, int], Tuple[int, str]] Sort key tuple for ordering motifs. Standard pattern names return (0, pattern_number) while non-standard names return (1, name). Notes ----- This function is used internally by load_modisco_motifs to ensure consistent motif ordering across runs. r_r#r%)rrr IndexError)r pattern_names r_motif_name_sort_keyr s.7L%3|))#..r23344  #%%% %s<--c2226778 8 8 8J' % % %|$ $ $ $ $ $ %%s-)4B)A2/B2B B B  B pos_patterns neg_patternsmodisco_h5_path trim_coordstrim_thresholdstrim_threshold_default motif_typemotifs_includemotif_name_map motif_lambdasmotif_lambda_default include_rczM 4 WzM Wc  gggggggd} g} g} |t|} nd} |i}|i}|i}|i}t|tt|krtdt j|d5}t D]z}||vr||}tt| tD]!\}\}}|d|}| || vr| ||}|}| ||}|dddj }tj|dz}||z }|ddd ddd f}||vr ||\}}n)| ||}t#||\}}|jd }||z ||z } }tj|jd tj }!d |!||<tj|jd tj }"d |"|| <|d kr|}#|}$|}%n|d krT|dddj }&tj|&dz}%|&|%z }#|#ddd ddd f}$n|dkrB|dddj }&d }%|&tj|&ddz }#|#ddd ddd f}$nP|dkr7|dddj }&d }%t+|&}#|#ddd ddd f}$ntd|d| d|| d|| dd| d|| d|| d|%| d|| r| d|| d|| dd| d|| d| | d|%| d|| |#|$g| |!|"g| |#| |!#| dddn #1swxYwYt1j| d !}'tj| tjd"}(tj| tjd"})|'t1jddkd }*|'|(|)|*fS)#aG Load motif data from TF-MoDISco HDF5 file with customizable processing options. This function extracts contribution weight matrices and associated metadata from TF-MoDISco results, with support for custom naming, trimming, and regularization parameters. Parameters ---------- modisco_h5_path : str Path to TF-MoDISco HDF5 results file containing pattern groups. trim_coords : Optional[Dict[str, Tuple[int, int]]] Manual trim coordinates for specific motifs {motif_name: (start, end)}. Takes precedence over automatic trimming based on thresholds. trim_thresholds : Optional[Dict[str, float]] Custom trim thresholds for specific motifs {motif_name: threshold}. Values should be between 0.0 and 1.0. trim_threshold_default : float Default trim threshold for motifs not in trim_thresholds. Fraction of maximum contribution used for trimming. motif_type : str Type of motif to extract. Must be one of: - 'cwm': Contribution weight matrix (normalized) - 'hcwm': Hypothetical contribution weight matrix - 'pfm': Position frequency matrix - 'pfm_softmax': Softmax-transformed position frequency matrix motifs_include : Optional[List[str]] List of motif names to include. If None, includes all motifs found. Names should follow format 'pos_patterns.pattern_N' or 'neg_patterns.pattern_N'. motif_name_map : Optional[Dict[str, str]] Mapping from original to custom motif names {orig_name: new_name}. New names must be unique across all motifs. motif_lambdas : Optional[Dict[str, float]] Custom lambda regularization values for specific motifs {motif_name: lambda}. Higher values increase sparsity penalty for the corresponding motif. motif_lambda_default : float Default lambda value for motifs not specified in motif_lambdas. include_rc : bool Whether to include reverse complement motifs in addition to forward motifs. If True, doubles the number of motifs returned. Returns ------- motifs_df : pl.DataFrame DataFrame containing motif metadata with columns: motif_id, motif_name, motif_name_orig, strand, motif_start, motif_end, motif_scale, lambda. cwms : Float[ndarray, "M 4 W"] Contribution weight matrices for all motifs where M is the number of motifs, 4 represents A,C,G,T nucleotides, and W is the motif width. trim_masks : Int[ndarray, "M W"] Binary masks indicating core motif regions (1) vs trimmed regions (0). Shape is (M motifs, W motif_width). names : ndarray Array of unique motif names (forward strand only). Raises ------ ValueError If motif_type is not one of the supported types, or if motif names in motif_name_map are not unique. FileNotFoundError If the specified HDF5 file does not exist. KeyError If required datasets are missing from the HDF5 file. Notes ----- Motif trimming removes low-contribution positions from the edges based on the position-wise sum of absolute contributions across nucleotides. The trimming helps focus on the core binding site. Adapted from https://github.com/jmschrei/tfmodisco-lite/blob/570535ee5ccf43d670e898d92d63af43d68c38c5/modiscolite/report.py#L252-L272 ) motif_namemotif_name_origstrand motif_start motif_end motif_scalelambdaNz$Specified motif names are not uniquerr!r?contrib_scoresr}rr%rmrhcwmhypothetical_contribspfmrprTr pfm_softmaxzInvalid motif_type: z5. Must be one of 'cwm', 'hcwm', 'pfm', 'pfm_softmax'.rrr+rrrr-motif_idrK)rnr)!r[rrrrrMODISCO_PATTERN_GROUPSrr_sorteditemsr getrkrssqrtrrrrrrrr^rUrrYrrfilterrXr\r)+rrrrrrrrrrmotif_data_lsts motif_lst trim_mask_lstmotifs_include_setmodisco_resultsrL metaclusterrr pattern pattern_tag motif_lambdapattern_tag_origcwm_rawcwm_normcwm_fwdcwm_rev start_fwdend_fwdrcwm_len start_revend_rev trim_mask_fwd trim_mask_rev motif_fwd motif_rev motif_norm motif_raw motifs_dfcwms trim_masksnamess+ rload_modisco_motifsrMsjOIM! 00!   > " "##s3~/D/D/F/F+G+G'H'HHH?@@@ ?C ( (`8O*_ 8_ 8D?//1111)$/K.7{((**0DEEE//Z 8Z 8**L'"&66 66 '2#+===,00>RSS #. ,00kJJ !"23AAA687GQJ#3#3#5#566!H,!$$B$"*-+--)4[)A&Iww%4%8%8#%;&&N*4G^)L)L&Iw!-*%,w%6)8K7 "q)9 I I I 34 i/0 "q)9 I I I 34 i/0&& 'I 'I!)JJ6)) '(? @ C EI!#)Q,););)=)=!>!>J )J 6I )$$B$"* 5II5(( ' 3AAA 6 8I!"J )BF91t,T,T,T TI )$$B$"* 5II=00 ' 3AAA 6 8I!"J ' 2 2I )$$B$"* 5II%pzppp -44[AAA 1299:JKKK)00555 .55i@@@ ,33G<<< .55jAAA)00>>>8#L188EEE#$56==>NOOO#H-44S999#M299)DDD#K077@@@#M299*EEE#H-44\BBB$$i%;<<<!((-)GHHHH$$Y///!((7777uZ 8 _ 8`8`8`8`8`8`8`8`8`8`8`8`8`8`8`8D _--<<*<MMI 8IRZa 8 8 8D-rwQ???J))S011<<\JJSSUU  dJ --s(RT99T=T=FrImodisco_half_widthlazycg}g}g}g} g} g} tj|d5} tD]} | | vr| | }t}t t ||D]}\}\}}| d||dddtj }|dddtj }|dddt}d|D}|d ddtj }t|d d }||||||| || || fd t!|D dddn #1swxYwYtj|tj|tj|| tj| | d }||z }t%j||dd|ddt%jdt%jdt%jdt%jdzt%jdz|zt%jdt%jdzt%jdz|zt%jdt%jdz|zt%jdt%jdz|zt%jdt%jdt%jdt%jdt%jd gd}|r|n|}|S)aELoad seqlet data from TF-MoDISco HDF5 file and convert to genomic coordinates. This function extracts seqlet instances from TF-MoDISco results and converts their relative positions to absolute genomic coordinates using peak region information. Parameters ---------- modisco_h5_path : str Path to TF-MoDISco HDF5 results file containing seqlet data. peaks_df : pl.DataFrame DataFrame containing peak region information with columns: 'peak_id', 'chr', 'chr_id', 'peak_region_start'. motifs_df : pl.DataFrame DataFrame containing motif metadata with columns: 'motif_name_orig', 'strand', 'motif_name', 'motif_start', 'motif_end'. half_width : int Half-width of the current analysis regions. modisco_half_width : int Half-width of the regions used in the original TF-MoDISco analysis. Used to calculate coordinate offsets. lazy : bool, default False If True, returns a LazyFrame for efficient chaining of operations. If False, collects the result into a DataFrame. Returns ------- Union[pl.DataFrame, pl.LazyFrame] Seqlets with genomic coordinates containing columns: - chr: Chromosome name - chr_id: Numeric chromosome identifier - start: Start coordinate of trimmed motif instance - end: End coordinate of trimmed motif instance - start_untrimmed: Start coordinate of full motif instance - end_untrimmed: End coordinate of full motif instance - is_revcomp: Whether the motif is reverse complemented - strand: Motif strand ('+' or '-') - motif_name: Motif name (may be remapped) - peak_id: Peak identifier - peak_region_start: Peak region start coordinate Notes ----- Seqlets are deduplicated based on chromosome ID, start position (untrimmed), motif name, and reverse complement status to avoid redundant instances. The coordinate transformation accounts for differences in region sizes between the original TF-MoDISco analysis and the current analysis. r r!r?z seqlets/startNz seqlets/endzseqlets/is_revcompcg|]}|sdnd S)r'r(r(rs rrOz(load_modisco_seqlets..s!FFFQa033SFFFr,zseqlets/example_idxzseqlets/n_seqletsrcg|]}Sr(r()r)rr7s rrOz(load_modisco_seqlets..$s$K$K$KQ[$K$K$Kr,) seqlet_start seqlet_end is_revcomprrJr)rrinneronhowrJr/rTrIrSrrrTrUrr) r/rTrrstart_untrimmed end_untrimmedrUrrrJrI)rTrZrrU)subset)rrr*rr r_r+r,rvrsrboolrrrr^range concatenaterU LazyFramejoinrOrWrXr]rZ)rrrIr=rNrO start_lstend_lstis_revcomp_lst strand_lst peak_id_lst pattern_tagsr4rLr5r!rr r6startsends is_revcompsstrandspeak_ids n_seqletsdf_dataoffset seqlets_dfr7s @rload_modisco_seqletsrqstIGNJKL ?C ( (MO* M MD?//1111)$/K&C.7{((**444// M M**L'"&66 66  1!!!4;;BHEE}-aaa077AA%&:;AAA>EEdKK FF+FFF"#89!!!<CCBINN(; +..' G, ,F W inn#@g N N hmmoo) 9 9 u 6(##&,--f^$$%f]##$*++f^$$%f[!!"F#677f^$$%&!455|8L8LLvUvl++6(##vl++F9%% f%899'   * PPP Q Q38 $=););)=)=J sG"HHHrpc|ddg}t|tjr|}||ddS)aWrite TF-MoDISco seqlets to TSV file. Parameters ---------- seqlets_df : Union[pl.DataFrame, pl.LazyFrame] Seqlets DataFrame with genomic coordinates. Must contain columns that are safe to drop: 'chr_id', 'is_revcomp'. out_path : str Output TSV file path. Notes ----- Removes internal columns 'chr_id' and 'is_revcomp' before writing to create a clean output format suitable for downstream analysis. rTrUrrEN)droprrUr`rZ write_csv)rprs rwrite_modisco_seqletsrvRs^$(L!9::J*bl++*'')) T22222r,)r/rrrZr[rhit_coefficienthit_coefficient_globalhit_similarityhit_correlationhit_importancehit_importance_sqrr2rJ is_primary hits_pathschemactj|dd|tjdd}|r|n|S)a3Load motif hit data from TSV file. Parameters ---------- hits_path : str Path to TSV file containing motif hit results. lazy : bool, default False If True, returns a LazyFrame for efficient chaining operations. If False, collects the result into a DataFrame. schema : Dict[str, Any], default HITS_DTYPES Schema defining column names and data types for the hit data. Returns ------- Union[pl.DataFrame, pl.LazyFrame] Hit data with an additional 'count' column set to 1 for aggregation. rN)rErFrr%count)rUrVr`litrbrZ)r~rOrhits_dfs r load_hitsr~sb(kTd6l26!99??7++,,  177 1 11r,rc|'||}t|tjr|}||ddS)aWrite processed hit data to TSV file with optional column filtering. Parameters ---------- hits_df : Union[pl.DataFrame, pl.LazyFrame] Hit data to write to file. out_path : str Output path for the TSV file. schema : Optional[Dict[str, Any]], default HITS_DTYPES Schema defining which columns to include in output. If None, all columns are written. Nrrs)rWrrrUr`rZru)rrrs rwrite_hits_processedrse"..//'2<(($//## h$/////r,qc_dfout_dir motif_widthcf tj|dtj|d}tj|d}tj|d}||dd|dd|d dt jd t jd t jd t jd zt jdzt jd t jd zt jdzt jd t jd zt jd t jd z|zt jdt jdt jdt jddzzt jdt jdt jdt jdzt jdt jddzzt jdt jdt jdt jdd dgt } | t jd   gdd} | t jd t jdt jdt jdt jdt jd } | |d!"| |d!"| |d#d!$d%S)&aWrite comprehensive hit results to multiple output files. This function combines hit data with peak, motif, and quality control information to generate complete output files including genomic coordinates and scores. Parameters ---------- hits_df : Union[pl.DataFrame, pl.LazyFrame] Hit data containing motif instance information. peaks_df : pl.DataFrame Peak region information for coordinate conversion. motifs_df : pl.DataFrame Motif metadata for annotation and trimming information. qc_df : pl.DataFrame Quality control data for normalization factors. out_dir : str Output directory for results files. Will be created if it doesn't exist. motif_width : int Width of motif instances for coordinate calculations. Notes ----- Creates three output files: - hits.tsv: Complete hit data with all instances - hits_unique.tsv: Deduplicated hits by genomic position and motif (excludes rows with NA chromosome coordinates) - hits.bed: BED format file for genome browser visualization Rows where the chromosome field is NA are filtered out during deduplication to ensure that data_unique only contains well-defined genomic coordinates. Texist_okzhits.tsvzhits_unique.tsvzhits.bedrJrVrWr)rTr/rI hit_startrrrrw global_scaler}ryr{r|rr2r)rTr/rrrZr[rrwrxryrzr{r|rr2rJr8r)r/rrr)r\rPrr)r/rrrrrrrsF)include_headerrEN)osmakedirsrrarOrWrUrXsort HITS_DTYPESrr/ is_not_nullr]rrZru) rrrIrrr out_path_tsvout_path_tsv_unique out_path_beddata_all data_uniquedata_beds r write_hitsrsLK$''''7<<44L',,w0ABB7<<44L   hmmoo) 9 9 ejjllyg 6 6 inn:7 ; ; 6(##u &,--f[!!"f]##$*++bf[.A.AABF;DWDWWF#677"&:M:MM&!455f[!!"vl++F#455#%6*;#<#<vn%%*$,6"233F#3446"233bf^6L6LL f%899vn%%*,6(##f[))F9%%))/   2 x! " "    "" # #? D//"&--";";"="=>>EE777FK!! F5MMfWoo F5MM6,''fQiivh "H    >>>##$74#HHH   et TTTTTr,c ||ddddgd}||ddS) a@Write quality control data with peak information to TSV file. Parameters ---------- qc_df : pl.DataFrame Quality control metrics for each peak region. peaks_df : pl.DataFrame Peak region information for coordinate annotation. out_path : str Output path for the TSV file. rJrVrWrTrIrrsN)rOrarrtrZru)rrrdfs rwrite_qcrst  hmmoo) 9 9 x,- . . h  LLTL*****r,c4||ddS)zWrite motif metadata to TSV file. Parameters ---------- motifs_df : pl.DataFrame Motif metadata DataFrame. out_path : str Output path for the TSV file. rrsNru)rIrs rwrite_motifs_dfr(s#D11111r,)r)rrrrrrr motifs_pathctj|dt}|tjddkd}||fS)a8Load motif metadata from TSV file. Parameters ---------- motifs_path : str Path to motif metadata TSV file. Returns ------- motifs_df : pl.DataFrame Motif metadata with predefined schema. motif_names : ndarray Array of unique forward-strand motif names. r)rErrr'r)rUread_csv MOTIF_DTYPESr/rXr\r)rrI motif_namess rload_motifs_dfrAsf K4 MMMI))S011<<\JJSSUU k !!r,rJc0tj||dS)aWrite motif contribution weight matrices to .npy file. Parameters ---------- cwms : Float[ndarray, "M 4 W"] Contribution weight matrices for M motifs, 4 nucleotides, W width. out_path : str Output path for the .npy file. N)rssave)rJrs rwrite_motif_cwmsrXsGHdr, cwms_pathc*tj|S)zLoad motif contribution weight matrices from .npy file. Parameters ---------- cwms_path : str Path to .npy file containing CWMs. Returns ------- Float[ndarray, "M 4 W"] Loaded contribution weight matrices. )rsr)rs rload_motif_cwmsres 79  r,paramsct|d5}tj||dddddS#1swxYwYdS)zWrite parameter dictionary to JSON file. Parameters ---------- params : Dict[str, Any] Parameter dictionary to serialize. out_path : str Output path for the JSON file. wr~)indentN)rjsondump)rrrs r write_paramsrus h  ' &!A&&&&''''''''''''''''''s 6:: params_pathc|t|5}tj|}dddn #1swxYwY|S)zLoad parameter dictionary from JSON file. Parameters ---------- params_path : str Path to JSON file containing parameters. Returns ------- Dict[str, Any] Loaded parameter dictionary. N)rrr)rrrs r load_paramsrsx k  a1 Ms 155occ_dfc4||ddS)zWrite occurrence data to TSV file. Parameters ---------- occ_df : pl.DataFrame Occurrence data DataFrame. out_path : str Output path for the TSV file. rrsNr)rrs r write_occ_dfrs# X.....r,seqlet_confusion_dfc4||ddS)zWrite seqlet confusion matrix data to TSV file. Parameters ---------- seqlet_confusion_df : pl.DataFrame Seqlet confusion matrix DataFrame. out_path : str Output path for the TSV file. rrsNr)rrs rwrite_seqlet_confusion_dfrs#!!(d!;;;;;r, report_dfmotifsc tj|d}tj|d|D]\}}tj||}tj|d|D];\}}t jtj||d|<|tj|dddS) aQWrite comprehensive motif report data including CWMs and metadata. Parameters ---------- report_df : pl.DataFrame Report metadata DataFrame. motifs : Dict[str, Dict[str, ndarray]] Nested dictionary of motif names to motif types to arrays. out_dir : str Output directory for report files. rTrz.txtzmotif_report.tsvrrsN)rrrarr,rssavetxtru) rrr motifs_dirmv motif_dirrmotifs rwrite_report_datars gx00JK T**** LL1GLLQ//  I----!" L L J Jrw||I*/B/B/BCCU K K K K L W.@AATRRRRRr,r')r)F)N__doc__rrr contextlibrtypingrrrrrr r rrsr r hdf5pluginpolarsrUrr jaxtypingr r rstrrr#r.r9__annotations__StringInt32UInt32Float32r:rrrgrrorryrrrrrr]rrfloatrrr r*rMr`rqrvrHITS_COLLAPSED_DTYPESrrrrrrrrrrrrrrr(r,rrs      DDDDDDDDDDDDDDDDDD  349,s#(<c3h@" "#SE3J/" #uS#X """"L   49   IHHIIIJJJH 49   =='/}=BE=\====B$28$8$8$8EEE bjEEE02wS3w~;N