h׹1 dZddlmZddlZddlZddlZddlZddlmZm Z ddl Z dDde dee d e d e e d e d e d dfdZ dDdee dee de e d e d e d df dZ dDdee dee de e d e d e d df dZ dDdee dee de e de d e d e d dfdZ dEde dee de dee d ee d!ee d"ee d#e d$ee d%ee d&ed'ed(ed)ed*ed+ed,e d-e d.ed/ee d0e d1ed2ed df0d3Z dFde d6e dee dee d ee d!ee d#e d7e d8ed9ed:ed dfd;ZdGd=e d e d>ed dfd?ZdHd@e e d e dAed dfdBZdIdCZdS)JaMain CLI module for the Fi-NeMo motif instance calling pipeline. This module provides the command-line interface for all Fi-NeMo operations: - Data preprocessing from various genomic formats - Motif hit calling using the Fi-NeMo algorithm - Report generation and result visualization - Post-processing operations (hit collapsing, intersection) The CLI supports multiple input formats including bigWig, HDF5 (ChromBPNet/BPNet), and TF-MoDISco format. )data_ioN)OptionalList peaks_pathchrom_order_pathfa_pathbw_pathsout_path region_widthreturnc|dz}tj|||}tj||||\}} tj|| ||dS)axExtract genomic regions and contribution scores from bigWig and FASTA files. Parameters ---------- peaks_path : str Path to ENCODE NarrowPeak format file. chrom_order_path : str, optional Path to chromosome ordering file. fa_path : str Path to genome FASTA file. bw_paths : List[str] List of bigWig file paths containing contribution scores. out_path : str Output path for NPZ file. region_width : int, default 1000 Width of regions to extract around peak summits. Notes ----- BigWig files only provide projected contribution scores. peaks_dfN)r load_peaksload_regions_from_bwwrite_regions_npz) rr r r r r half_widthr sequencescontribss 2/srv/www/kundaje/kobbad/Fi-NeMo/src/finemo/main.pyextract_regions_bwrsg:"J!*.> KKH!6'8ZIx  i8hOOOOOOh5_pathsc|dz}|tj|||}nd}tj||\}}tj||||dS)a Extract genomic regions and contribution scores from ChromBPNet HDF5 files. Parameters ---------- peaks_path : str, optional Path to ENCODE NarrowPeak format file. If None, lacks absolute coordinates. chrom_order_path : str, optional Path to chromosome ordering file. h5_paths : List[str] List of ChromBPNet HDF5 file paths. out_path : str Output path for NPZ file. region_width : int, default 1000 Width of regions to extract around peak summits. rNr)rrload_regions_from_chrombpnet_h5r rr rr r rrrrs rextract_regions_chrombpnet_h5r Asg,"J%j2BJOO!A(JWWIx i8hOOOOOOrc|dz}|tj|||}nd}tj||\}}tj||||dS)aExtract genomic regions and contribution scores from BPNet HDF5 files. Parameters ---------- peaks_path : str, optional Path to ENCODE NarrowPeak format file. If None, output lacks absolute coordinates. chrom_order_path : str, optional Path to chromosome ordering file. h5_paths : List[str] List of BPNet HDF5 file paths. out_path : str Output path for NPZ file. region_width : int, default 1000 Width of regions to extract around peak summits. rNr)rrload_regions_from_bpnet_h5rrs rextract_regions_bpnet_h5r#csg,"J%j2BJOO!r?r@rArBrCrDrEparamsrXrGrrr has_peaksr rrS motif_typeuse_hypothetical_contribsmotifs_includemotif_name_map motif_lambdas trim_coordstrim_thresholds motifs_dfcwms trim_masks_rU motif_widthlambdas device_objhits_dfqc_df out_path_qcout_path_motif_dfout_path_motif_cwmsout_path_paramss6 r call_hitsrszXXFLLL   o   07/G /U/U,Ix9?1%La1QLQQQRRR"J.#K  T   %j2BJOO   _    t|| $)!!  $(!!  $)!!  $(!! KT K K K   & )*=>># -.>DD%,-?GG  '01EsKK  +!./GOO%,%@"  & &"ItZAJ*Q-K""8,,55t5DDG)/);f%%%J++ ! !NGU&K$''''',,w77K w)UG[QQQ UHk222 W.>?? I'8999',,w0@AA T#6777 $"!, Fgll7,=>>O 11111rThits_dirmodisco_region_widthcwm_trim_thresholdcompute_recall use_seqletsc 8 ddlm} m} tj|\} }}}t |jdkr|| z}nBt |jdkr|dddddf| z}ntd|j|jddz}|dz}|*tj dtj |d|}| drtj d |}tj |d }|tj |}nd}|tj|t}nd}|#tj|dd|d ||dd d \}}}}ntdt"j|d}tj |d }t"j|d}tj|\}}t"j|d}tj|}t"j|d}tj|}|d}| stj dd} n!|d} d} ntj|||||d } |jd}!t1|t2jr|}"n|}"t9|}#| |"|#\}$}%| || ||| |||#||!| \}&}'}(})| | || ||#|!\}*}+nd\}*}+t#j |d t"j|d},tj!|$|,tj"|'|(|| #|"|#|| $|$|#|| %|%|#|t"j|d}-| &|(|)|-| t1| t2jr| 'n| }.t"j|d}/tj(|.|/|*d|+bt"j|d}0tj)|*|0| *|&|| +|+|#|t"j|d}1| ,|'|#|1| | dudS)aGenerate comprehensive HTML report with statistics and visualizations. This function creates detailed analysis reports comparing Fi-NeMo hit calling results with TF-MoDISco seqlets, including performance metrics, distribution plots, and motif visualization. The report provides insights into hit calling quality and motif discovery accuracy. Parameters ---------- regions_path : str Path to NPZ file containing the same regions used for hit calling. hits_dir : str Path to directory containing Fi-NeMo hit calling outputs. modisco_h5_path : str, optional Path to TF-MoDISco H5 file. If None, seqlet comparisons are skipped. peaks_path : str, optional DEPRECATED. Peak coordinates should be included in regions file. motifs_include_path : str, optional DEPRECATED. This information is inferred from hit calling outputs. motif_names_path : str, optional DEPRECATED. This information is inferred from hit calling outputs. out_dir : str Output directory for report files. modisco_region_width : int, default 400 Width of regions used by TF-MoDISco (needed for coordinate conversion). cwm_trim_threshold : float, default 0.3 DEPRECATED. This information is inferred from hit calling outputs. compute_recall : bool, default True Whether to compute recall metrics against TF-MoDISco seqlets. use_seqlets : bool, default True Whether to include seqlet-based comparisons in the report. Notes ----- The generated report includes: - Hit vs seqlet count comparisons - Motif CWM visualizations - Hit statistic distributions - Co-occurrence heatmaps - Confusion matrices for overlapping motifs r) evaluation visualizationrNzUnexpected contribs shape: zProviding a peaks file to `report` is deprecated, and this option will be removed in a future version. Peaks should instead be provided in the preprocessing step to be included in `regions.npz`.z.tsvz|Passing a hits.tsv file to `finemo report` is deprecated. Please provide the directory containing the hits.tsv file instead.TlazyrHg?zCmodisco_h5_path is required when providing a hits.tsv file directlyzhits.tsvrQrRrVr9zUsage of the `--no-seqlets` flag is deprecated and will be removed in a future version. Please omit the `--modisco-h5` argument instead.F)NNrOzmotif_occurrences.tsvmotifsz seqlets.tsvzseqlet_confusion.tsvz report.html)-rYrrrr\lenr]r^rZr[rendswith load_hitsr_r`rarerirkrlload_motifs_dfload_motif_cwms load_paramsload_modisco_seqlets isinstancepl LazyFramerlistget_motif_occurencestfmodisco_comparisonseqlet_confusionrj write_occ_dfwrite_report_dataplot_hit_stat_distributionsplot_hit_peak_distributions!plot_peak_motif_indicator_heatmap plot_motifscollectwrite_modisco_seqletswrite_seqlet_confusion_dfplot_hit_vs_seqlet_countsplot_seqlet_confusion_heatmap write_report)2r1rr2rr3r4r6rrrrrrrrrr~regionsrmodisco_half_width hits_pathrrvrwr{ cwms_modisco motif_names hits_df_pathmotifs_df_pathcwms_modisco_path params_pathrr seqlets_dfr hits_df_lazymotif_names_listocc_dfcoooc report_data report_dfr trim_bounds confusion_df confusion_matocc_pathplot_dirseqlets_collected seqlets_pathseqlet_confusion_path report_paths2 rreportrsl,+++++++'.'? 'M'M$Ix1 8>aY& X^   ! !111dAAA:&2Gx~GGHHHq!Q&J-2  Q   %j$ CC  3B  K    #ID999  *$-.ABBNN!N  '$12BCHHNN!N  &6=6Q" 7 7 3I|Q U  w||Hj99 #Lt<<<h0@AA!(!7!G!G ;GLL3CDD./@AA gll8->?? $[11#$@A    W      1         $Q'K'2<((4%, %,\\^^ "&{"3"333LBRSSMFE2<2Q2Q 3 3/KFK&0&A&A Z+;[' ' # mm'1# mK$''''w||G%<==H *** i999--lz"intersect_hits..s(UUUY!)%888UUUrNr)rYrintersect_hitsrr)rr rrhits_dfsrs rrrs\2!     UU*UUUH++Hg>>G  (4@@@@@@rc\tj}|dd}|dtjd}|ddt dd |d d t d d|ddt dd |ddt ddd|ddt dd |ddttj tj dj d|dtjd }|ddt d d!|d d t d d|d"d#t ddd$|ddt dd |ddttj tj dj d|d%tjd&}|ddt d d!|d d t d d|d"d#t ddd$|ddt dd |ddttj tj dj d|d'tjd(}|ddt d d!|d d t d d|d"d#t ddd$|ddt dd |ddttj tj dj d|d)tjd*}|ddt d d!|d d t d d|d+d,t dd- |d.d/t ddd0|ddt dd |ddttj tj dj d|d1tjd2}|d3d4t tj t j d5j hd6d78|d9d:t dd; |d |ddt d d?|d d t d d?|d@dAt d dB|dCdDt d dE|ddFt ddG |dHdIt"tj t j dJj dK|dLdMt d dN|dOdPt d dQ|dRdSt"tj t j dTj dU|dVdWt d dX|d.dYt"d dZ|d[d\t d d]|dd^d_d`a|dbdcd_dda|d+det"tj t j dfj dg|dhdit"tj t j djj dk|dldmt"tj t j dnj do|d"dpt"tj t j dqj dr|dsdtttj t j duj dv|ddwttj t j dxj dy|dzd{t d d||d}d~d_da|dtjd}|d9d:t dd |ddt dd |ddt d d?|drLrJrIr0zThe type of attributions to use for CWM's and input contribution scores, respectively. 'h' for hypothetical and 'p' for projected.)rrchoicesrz-rz --regionsz~A .npz file of input sequences, contributions, and coordinates. Can be generated using `finemo extract-regions-*` subcommands.z-mz --modisco-h5z2A tfmodisco-lite output H5 file of motif patterns.zfDEPRECATED: Please provide this file to a preprocessing `finemo extract-regions-*` subcommand instead.z-Iz--motifs-includezA tab-delimited file with tfmodisco motif names (e.g pos_patterns.pattern_0) in the first column to include in hit calling. If omitted, all motifs in the modisco H5 file are used.z-Nz --motif-nameszA tab-delimited file with tfmodisco motif names (e.g pos_patterns.pattern_0) in the first column and custom names in the second column. Omitted motifs default to tfmodisco names.z --out-dirz!The path to the output directory.z-tz--cwm-trim-thresholdr9zVThe default threshold to determine motif start and end positions within the full CWMs.z-Tz--cwm-trim-thresholdszA tab-delimited file with tfmodisco motif names (e.g pos_patterns.pattern_0) in the first column and custom trim thresholds in the second column. Omitted motifs default to the `--cwm-trim-threshold` value.z-Rz--cwm-trim-coordszA tab-delimited file with tfmodisco motif names (e.g pos_patterns.pattern_0) in the first column and custom trim start and end coordinates in the second and third columns, respectively. Omitted motifs default to `--cwm-trim-thresholds` values.z-lz--global-lambdar:zFThe default L1 regularization weight determining the sparsity of hits.z-Lz--motif-lambdaszA tab-delimited file with tfmodisco motif names (e.g pos_patterns.pattern_0) in the first column and motif-specific lambdas in the second column. Omitted motifs default to the `--global-lambda` value.z--alphaz7DEPRECATED: Please use the `--lambda` argument instead.z-Az--motif-alphasz>DEPRECATED: Please use the `--motif-lambdas` argument instead.z--no-post-filter store_truezDo not perform post-hit-calling filtering. By default, hits are filtered based on a minimum cosine similarity of `lambda` with the input contributions.)actionrz-qz--sqrt-transformz\Apply a signed square root transform to the input contributions and CWMs before hit calling.z--step-size-maxr;z The maximum optimizer step size.z-iz--step-size-minr<z The minimum optimizer step size.z-jz --step-adjustrAzThe optimizer step size adjustment factor. If the optimizer diverges, the step size is multiplicatively adjusted by this factorz--convergence-tolr>zoThe tolerance for determining convergence. The optimizer exits when the duality gap is less than the tolerance.z-Sz --max-stepsr?z)The maximum number of optimization steps.z --batch-sizer@z%The batch size used for optimization.z-dz--devicezaDEPRECATED: Please use the `CUDA_VISIBLE_DEVICES` environment variable to specify the GPU device.z-Jz --compilezZJIT-compile the optimizer for faster performance. This may not be supported on older GPUs.rzOGenerate statistics and visualizations from hits and tfmodisco-lite motif data.z}A .npz file containing input sequences, contributions, and coordinates. Must be the same as that used for `finemo call-hits`.z-Hz--hitsziThe output directory generated by the `finemo call-hits` command on the regions specified in `--regions`.zThe tfmodisco-lite output H5 file of motif patterns. Must be the same as that used for hit calling unless `--no-recall` is set. If omitted, seqlet-derived metrics will not be computed.zTDEPRECATED: This information is now inferred from the outputs of `finemo call-hits`.z(The path to the report output directory.z-Wz--modisco-region-widthrzGThe width of the region around each peak summit used by tfmodisco-lite.rz-nz --no-recallz$Do not compute motif recall metrics.z --no-seqletszr?r@rArBrCrDcompilehitsr no_recall no_seqletsrrr) parser subparsersextract_regions_bw_parser$extract_chrombpnet_regions_h5_parserextract_regions_h5_parserextract_bpnet_regions_h5_parser"extract_regions_modisco_fmt_parsercall_hits_parser report_parsercollapse_hits_parserintersect_hits_parserargss rclirsJ $ & &F&&5&AAJ * 5 5 > O!6!! **   ? +**  q +**   r +**  F +**   0 +**  !"455 N$  N+,6+@+@' > Z,A,,( )55  G 6)55  q 6)55  B 6)55   0 6)55  !"?@@ N$  N6!+ 5 5 >T!6!! **  G +**  q +**  B +**   0 +**  !"?@@ N$  N+'1&;&;" > U'<''# $00  G 1$00  q 1$00  B 1$00   0 1$00  !":;; N$  N1*4)>)>% > S*?**& '33  G 4'33  q 4'33   H 4'33  Y 4'33   0 4'33  !"=>> N$  N4",, > O- !!  !),,7?G(((R "!!  N "!!   A "!!   u "!!   u "!!  C "!!  B "!!   0 "!!  !),, 02  e"!!  ] "!!  C "!!  !),,78HIQ U "!!  X "!!   F "!!   M "!! g " !!  k " !!  !),,7HP / "!!  !),,7HP / "!!  !),,7 FNO "!!  !),,78IJR ~ "!!  !),,7 DL 8 "!!  !),,7 EM 4 "!!   p "!!  i ")) > ^*M   M    x    u   H    c    c    7   !&))45KLT V   !&))45IJR c   3    K &00 > T1 %%   I &%%   W &%%  !-00;NKSc &'11 > 32 &&   a '&&   | '&& B '     D x''' J   J L M         4 4 4% J($(DM4CT      ) ) ) s    & J($(DM4CT      / / / J($(DM4CT      2 2 2# J     N M         [ : ! MK   "&D    ( MR   "&!2D  L J O         L   $  #          N O   K I   L/     4 X   L I O J     L  %  #    _ $ $di0ABBBBB % % %ty$->>>>> & %r)r)NNr)r*r+r,Fr-r.r/r*Nr0FF)rr)TT)r)F)rN)__doc__rYrrirrZrtypingrrpolarsrrardrr r#r(rbboolrrrrrrrrs   !!!!!!!! &P&P&Psm&P&P3i &P  &P  &P &P&P&P&P\ PP PsmP3iP P  P  PPPPN PP PsmP3iP P  P  PPPPP $P$P $Psm$Pc$P $P  $P  $P $P$P$P$P`+/.2(+ #  #/\2\2\2 \2\2sm \2 "# \2 sm \2! \2\2#3-\2'sm\2!&\2\2\2\2\2 !\2"#\2$%\2&'\2( SM)\2* +\2,-\2./\20 1\2\2\2\2N!$ #RRRRc]R R "# R sm RRRRRR RRRRjSCutDAAtCyACA$ASWAAAABt ?t ?t ?t ?t ?t ?r