hfP"dZddlZddlmZmZmZmZmZddlZ ddlm Z ddl Z ddl mZmZde jdeedee jee d fffd Zd ee d fd e jdedee dffdZd ee d fdee d fdee je jfde jdee je jdfde jdee dfdeededededeeeeeeffe jeeeeee dfffeeeeeeefffffdZdee je jfdee je jfde jdeededee jee d fff dZdS)aoEvaluation module for assessing Fi-NeMo motif discovery and hit calling performance. This module provides functions for: - Computing motif occurrence statistics and co-occurrence patterns - Evaluating motif discovery quality against TF-MoDISco results - Analyzing hit calling performance and recall metrics - Generating confusion matrices for seqlet-hit comparisons N)ListTupleDictAnyUnion)ndarray)FloatInthits_df motif_namesreturnzM Mc4|tjddddddd}t|t|jz }|d|Dtj |  dg}|j }t|}tj||ftj }t!|D]3\}}|||d d |f<4|dktj} | j| z} || fS) aCompute motif occurrence statistics and co-occurrence matrix. This function analyzes motif occurrence patterns across peaks by creating a pivot table of hit counts and computing pairwise co-occurrence statistics. Parameters ---------- hits_df : pl.LazyFrame Lazy DataFrame containing hit data with required columns: - peak_id : Peak identifier - motif_name : Name of the motif Additional columns are ignored. motif_names : List[str] List of motif names to include in analysis. Missing motifs will be added as columns with zero counts. Returns ------- occ_df : pl.DataFrame DataFrame with motif occurrence counts per peak. Contains: - peak_id column - One column per motif with hit counts - 'total' column summing all motif counts per peak coocc : Int[ndarray, "M M"] Co-occurrence matrix where M = len(motif_names). Entry (i,j) indicates number of peaks containing both motif i and motif j. Diagonal entries show total peaks containing each motif. Notes ----- The co-occurrence matrix is computed using binary occurrence indicators, so multiple hits of the same motif in a peak are treated as a single occurrence. count motif_namepeak_idsum)onindexvaluesaggregate_functionrc\g|])}tjd|*S)r)pllitalias).0ms 8/srv/www/kundaje/kobbad/Fi-NeMo/src/finemo/evaluation.py z(get_motif_occurences..Bs,FFFARVAYY__Q//FFF)totaldtypeN)collect with_columnsrrrpivot fill_nullsetcolumnssum_horizontalsortheightlennpzerosint16 enumerate get_columnto_numpyastypeint32T) r r occ_df missing_cols num_peaks num_motifsocc_matirocc_bincooccs rget_motif_occurencesr?stJ  bfQiioog.. / / 9WQV    1  {##c&.&9&99LFFFFFGG B-{; < < yk    I[!!Jh :.bh???G+&&881))!,,55771 {""28,,G I E 5=r regionszN 4 L positions_df motif_widthzH 4 Wc |tjdtjdtjdz tjd}|d}|d}|dt }|dk||z|jd kz}||}||}||}|d d d d f}|d d d d ftj d d |ft z} | |d d d d fxxtj |d d d d fz cc<| |d d d d fxxtj |d d d d d fz cc<tj |jddd ft } | |d d d d fxxtj dd d d d fz cc<| |d d d d fxxtj dd d d d d fz cc<||| | f} tj 5tjddtjdd| d} d d d n #1swxYwY| S)aExtract contribution weight matrices from regions based on hit positions. This function extracts motif-sized windows from contribution score regions at positions specified by hit coordinates. It handles both forward and reverse complement orientations and filters out invalid positions. Parameters ---------- regions : Float[ndarray, "N 4 L"] Input contribution score regions multiplied by one-hot sequences, OR Input one-hot encoded sequences. Shape: (n_peaks, 4, region_width) where 4 represents DNA bases (A,C,G,T). positions_df : pl.DataFrame DataFrame containing hit positions with required columns: - peak_id : int, Peak index (0-based) - start_untrimmed : int, Start position in genomic coordinates - peak_region_start : int, Peak region start coordinate - is_revcomp : bool, Whether hit is on reverse complement strand motif_width : int Width of motifs to extract. Must be positive. Returns ------- motifs : Float[ndarray, "H 4 W"] Extracted motif matrices for valid hits. Shape: (n_valid_hits, 4, motif_width) Invalid hits (outside region boundaries) are filtered out. Notes ----- - Start positions are converted from genomic to region-relative coordinates - Reverse complement hits have their sequence order reversed - Hits extending beyond region boundaries are excluded - The mean is computed across all valid hits, with warnings suppressed for empty slices or invalid operations Raises ------ ValueError If motif_width is non-positive or positions_df lacks required columns. rstart_untrimmedpeak_region_start is_revcomp)peak_idx start_idxrFrGrHrNrr"ignorez#invalid value encountered in divide)actionmessagezMean of empty slice)axis)selectrcolr2r3r4boolshaper.r/intarangewarningscatch_warningsfilterwarningsmean) r@rArBidx_dfrGrHrF valid_maskrow_idxpos_idxnuc_idxseqsmotifss r get_motifsraTsFX  ""&*++bf5H.I.II6,''!F   ,,5577H!!+..7799I""<0099;;BB4HHJq.Y%< a@P%PQJ #H*%IJ'Jqqq$}%G4 &1a2ES)Q)Q)QQG ZKAAA ")K"8"8tQQQ"GG J111 ;!7!7dDDbD8H!IIhq)1a0<<!?? 7GW, -D  " "##%J     x9NOOOO"" ############### Ms8AKK K  sequencespeaks_df seqlets_df motifs_df cwms_modiscozM 4 Wmodisco_half_widthcompute_recallz4 Wc Ht|tjr|}|tjdtj|dd tjdtjdtjdtjddktjd tjd tjd }| gd  } |j d} | dz } | tjdtjd z | |z ktjdtjd z | |zkz gd  }| d d}| d d}|d}d}nGt|tjr| }|}n|}|}||d d}ni}| r|||gdd }||gdd }||gdd }|d d}|d d}|d d}ni}i}i}i}i}i}| }|D]A}||f|}||f|}|} |}!|}"|}#|||f|} | rG|E||f|}!||f|}"||f|}#|j|jd||<|| j||d<| rR|P||xx|!j|"j|#j| jdkr!t'j|!j| jz nddzcc<|tjd |ktjddkzd}$|tjd |ktjddkzd}%t-||| ||$d||%dd||<||dddddddf||d<t-||| ||d <||d ddddddf||d!<| r3|1||xxt-||"| t-||#| d"zcc<|$d#|$d$f}&|%d#|%d$f}'|&|&|'|'|&|'d%||<| r|||xx|&|&d"zcc<||d}(||d&})t'j|(dz}*t'j|)dz}+|(|)z|*|+zz },|,||d'<Cd(|D}-tj|-}.||.||fS))a! Compare Fi-NeMo hits with TF-MoDISco seqlets and compute evaluation metrics. This function performs comprehensive comparison between Fi-NeMo hit calls and TF-MoDISco seqlets, computing recall metrics, CWM similarities, and extracting contribution weight matrices for visualization. Parameters ---------- regions : Float[ndarray, "N 4 L"] Contribution score regions multiplied by one-hot sequences. Shape: (n_peaks, 4, region_length) sequences : Int[ndarray, "N 4 L"] One-hot encoded sequences corresponding to regions. Shape: (n_peaks, 4, region_length) hits_df : Union[pl.DataFrame, pl.LazyFrame] Fi-NeMo hit calls with required columns: - peak_id, start_untrimmed, end_untrimmed, strand, motif_name peaks_df : pl.DataFrame Peak metadata with columns: - peak_id, chr_id, peak_region_start seqlets_df : Optional[pl.DataFrame] TF-MoDISco seqlets with columns: - chr_id, start_untrimmed, is_revcomp, motif_name If None, only basic hit statistics are computed. motifs_df : pl.DataFrame Motif metadata with columns: - motif_name, strand, motif_id, motif_start, motif_end cwms_modisco : Float[ndarray, "M 4 W"] TF-MoDISco contribution weight matrices. Shape: (n_modisco_motifs, 4, motif_width) motif_names : List[str] Names of motifs to analyze. modisco_half_width : int Half-width for restricting hits to central region for fair comparison. motif_width : int Width of motifs for CWM extraction. compute_recall : bool Whether to compute recall metrics requiring seqlets_df. Returns ------- report_data : Dict[str, Dict[str, Any]] Per-motif evaluation metrics including: - num_hits_total, num_hits_restricted, num_seqlets - num_overlaps, seqlet_recall, cwm_similarity report_df : pl.DataFrame Tabular format of report_data for easy analysis. cwms : Dict[str, Dict[str, Float[ndarray, "4 W"]]] Extracted CWMs for each motif and condition: - hits_fc, hits_rc: Forward/reverse complement hits - modisco_fc, modisco_rc: TF-MoDISco forward/reverse - seqlets_only, hits_restricted_only: Non-overlapping instances cwm_trim_bounds : Dict[str, Dict[str, Tuple[int, int]]] Trimming boundaries for each CWM type and motif. Notes ----- - Hits are filtered to central region defined by modisco_half_width - CWM similarity is computed as normalized dot product between hit and TF-MoDISco CWMs - Recall metrics require both hits_df and seqlets_df to be non-empty - Missing motifs are handled gracefully with empty DataFrames Raises ------ ValueError If required columns are missing from input DataFrames. rinnerrhowchr_idrD end_untrimmedstrand-rrE)rmrDrnrFrrErrmrDrrFsubsetrIT)as_dictN)rmrDrFranti)num_hits_totalnum_hits_restricted num_seqletsrg) num_overlapsnum_seqlets_onlynum_hits_restricted_only seqlet_recall+) by_predicatenamedmotif_id)hits_fc modisco_fc modisco_rcrrJhits_rc hits_ppm_fc hits_ppm_rc) seqlets_onlyhits_restricted_only motif_start motif_end)rrrrrrrcwm_similarityc"g|] \}}d|i|z S)r)rkvs rrz(tfmodisco_comparison..s&EEEA a 1$EEEr ) isinstancer DataFramelazyr%rQcastUInt32joinrPuniquerSfilterr$ partition_by LazyFramecleargetr,r.float64rowrasqrtritems from_dicts)/r@rbr rcrdrerfr rgrBrh hits_unique region_lencenter hits_filtered hits_by_motifhits_filtered_by_motifseqlets_collected seqlets_lazyseqlets_by_motif overlaps_dfseqlets_only_dfhits_only_filtered_dfoverlaps_by_motifseqlets_only_by_motifhits_only_filtered_by_motif report_datar`cwm_trim_boundsdummy_dfrhitsseqletsoverlapsrhits_only_filtered motif_data_fc motif_data_rc bounds_fc bounds_rchits_cwm modisco_cwmhnormsnormcwm_simrecords report_dfs/ rtfmodisco_comparisonrs n'2<((!,,.. RVI..33BI>>?? hmmoo) 9 9 6(##F#455&11vh''3.vl++ f%899F9%%    ..HHH!Kq!J !^FNN V% & &0C)D)D D++ -VO $ $rv.A'B'B B++ -     fMMMfNN ''))66|T6RRM*2244AAdB  J - -)&..00! &!(( $,99,PT9UU),2#(( HHH)   '))  '++ HHH,   '))  !. 2 2 HHH!3! !  ')) (44\44PP / < <\SW < X X&;&H&H $'I' ' ## "&(#K FO}}&&((H V3V3  !x00.22A4BB  %  !&**A4::G  Qj4(,,aT8<,E>A%%"$HO!z$seqlet_confusion..s;;;DAq1a;;;r )motif_name_seqletsr frac_overlaprr)r\col_idxrr\r)rrrrr%rQrrrrrPgroup_byr-r$r.r/float32r1replace_strictr3)r rdrcr rBbin_size hits_binnedrseqlets_binnedr seqlet_countsoverlap_countsr: confusion_mat name_to_idx confusion_dfconfusion_idx_dfr\rrs rseqlet_confusionrswrH'2<((!,,.. F9%%**2955vh''3.    hmmoo) 9 9 PPP Q Q 6(##f.//8;F?++x7vl++   ??$$L!((vh&*++x7''836,'' )N!%%:::PW&K  --11}1EEMMOO l,=>?? . ! !  [!!JHj*5RZHHHM;;Ik$:$:;;;K!&&,G' f6,//011VN++bf].C.CC $**+,,;;KHH())88EEVN+++ y)2244Gy)2244G#N3<<>>L&2M'7"#  &&r )__doc__rVtypingrrrrrnumpyr.rpolarsr jaxtypingr r rstrrr?rTrarRrrrr rrs00000000000000 > \>(,S > 2<We^, ,->>>>BL 7G# $L46LLORL 7G LLLL^J; 7G# $J;7G#$J;2<- .J;l J; blBL$67 J; | J;()J;cJ;J;J;J; d38n Ld3gun--. ./d3c3h'( ()+J;J;J;J;Z{' 2<- .{'blBL01{'l{'c {'  {'  2<w~. ./ {'{'{'{'{'{'r