hx'rdZddlZddlmZmZmZmZmZddlm Z m Z ddl Z ddl m Z ddlZddlmcmZddlmZddlZddlmZmZmZddlmZd eed fd eed fd eedfdeedfdeeedfefdeedfdeedfdeeed feedfeedfffdZd eedfdeedfd eed fdeeedfefdeed fdeed fdeedfdeedfdedeedfdeeed feed feedfeedfffdadedejfdZd ejdejfd!ZGd"d#e Z Gd$d%e Z!Gd&d'e Z"Gd(d)e Z# dEd ee dfdeee d3fee d4ffdee d3fd5ee d6fd7e$dee d8fd9e%d:e%d;e$ded?e%d@e$dAeej&dBe$dCe%deej'ej'ff$dDZ(dS)FaGHit caller module implementing the Fi-NeMo motif instance calling algorithm. This module provides the core functionality for identifying transcription factor binding motif instances in neural network contribution scores using a competitive optimization approach based on proximal gradient descent. The main algorithm fits a sparse linear model where contribution scores are reconstructed as a weighted combination of motif contribution weight matrices (CWMs) at specific genomic positions. The sparsity constraint ensures that only the most significant motif instances are called. N)TupleUnionOptionalDictList)ABCabstractmethod)ndarray)Tensor)FloatIntBool)tqdm coefficientszB M Pimportance_scalezB 1 PcwmszM 4 WcontribsB 4 L sequenceslambdasz1 M 1 step_sizeszB 1 1return BcZ||z}tj||}||z} || z } tj| ||z} | dzd} | |z d} t jd| z ddzdzdz }| |z}| |zd}t j|dd|zd}||z |z}||| |z zz}tj|}||| fS) uPerform a proximal gradient descent optimization step for non-negative lasso. This function implements a single optimization step of the Fi-NeMo algorithm, which uses proximal gradient descent to solve a sparse reconstruction problem. The goal is to represent contribution scores as a sparse linear combination of motif contribution weight matrices (CWMs). Dimension notation: - B = batch size (number of regions processed simultaneously) - M = number of motifs - L = sequence length - W = motif width (length of each motif) - P = L - W + 1 (number of valid motif positions) Parameters ---------- coefficients : Float[Tensor, "B M P"] Current coefficient matrix representing motif instance strengths. importance_scale : Float[Tensor, "B 1 P"] Scaling factors for importance-weighted reconstruction. cwms : Float[Tensor, "M 4 W"] Motif contribution weight matrices for all motifs. 4 represents the DNA bases (A, C, G, T). contribs : Float[Tensor, "B 4 L"] Target contribution scores to reconstruct. sequences : Float[Tensor, "B 4 L"] | int One-hot encoded DNA sequences. Can be a scalar (1) for hypothetical mode. lambdas : Float[Tensor, "1 M 1"] L1 regularization weights for each motif. step_sizes : Float[Tensor, "B 1 1"] Optimization step sizes for each batch element. Returns ------- c_next : Float[Tensor, "B M P"] Updated coefficient matrix after the optimization step (shape: batch_size × motifs × positions). dual_gap : Float[Tensor, " B"] Duality gap for convergence assessment (shape: batch_size). nll : Float[Tensor, " B"] Negative log likelihood (proportional to MSE, shape: batch_size). Notes ----- The algorithm uses proximal gradient descent to solve: minimize_c: ||contribs - conv_transpose(c * importance_scale, cwms) * sequences||²₂ + λ||c||₁ subject to: c ≥ 0 References ---------- - Proximal gradient descent: https://yuxinchen2020.github.io/ele520_math_data/lectures/lasso_algorithm_extension.pdf, slide 22 - Duality gap computation: https://stanford.edu/~boyd/papers/pdf/l1_ls.pdf, Section III rdimr?maxT)rkeepdim) Fconv_transpose1dconv1dsumamaxtorchclampabsrelu)rrrrrrrcoef_adj pred_unmaskedpred residualsngradnll dual_norm dual_scale nll_scaled dual_diffl1_termdual_gapc_nexts 7/srv/www/kundaje/kobbad/Fi-NeMo/src/finemo/hitcaller.pyprox_grad_stepr;sa@..H&x66M ! 4I HY % %(8 8E a<    ( (C&&6&22I+a)m555:Q>!CJz!JX%**v*66Iy&&**q$*??'INN OGY&05577HJ%'/: :F VF^^F 8S  coef_intercoefiLc |} t|||||| |\}} } | |z } | d|zz } ||dzz } d| z|z| | zz }||| | fS)ugPerform a non-negative lasso optimizer step with Nesterov momentum. This function combines proximal gradient descent with momentum acceleration to improve convergence speed while maintaining the non-negative constraint on coefficients. Dimension notation: - B = batch size (number of regions processed simultaneously) - M = number of motifs - L = sequence length - W = motif width (length of each motif) - P = L - W + 1 (number of valid motif positions) Parameters ---------- cwms : Float[Tensor, "M 4 W"] Motif contribution weight matrices. contribs : Float[Tensor, "B 4 L"] Target contribution scores. importance_scale : Float[Tensor, "B 1 P"] Importance scaling factors. sequences : Union[Int[Tensor, "B 4 L"], int] One-hot encoded sequences or scalar for hypothetical mode. coef_inter : Float[Tensor, "B M P"] Intermediate coefficient matrix (with momentum). coef : Float[Tensor, "B M P"] Current coefficient matrix. i : Float[Tensor, "B 1 1"] Iteration counter for each batch element. step_sizes : Float[Tensor, "B 1 1"] Step sizes for optimization. L : int Sequence length for normalization. lambdas : Float[Tensor, "1 M 1"] Regularization parameters. Returns ------- coef_inter : Float[Tensor, "B M P"] Updated intermediate coefficients with momentum (shape: batch_size × motifs × positions). coef : Float[Tensor, "B M P"] Updated coefficient matrix (shape: batch_size × motifs × positions). gap : Float[Tensor, " B"] Normalized duality gap (shape: batch_size). nll : Float[Tensor, " B"] Normalized negative log likelihood (shape: batch_size). Notes ----- Uses Nesterov momentum with momentum coefficient i/(i+3) for improved convergence properties. The duality gap and NLL are normalized by sequence length for scale-invariant convergence assessment. References ---------- https://yuxinchen2020.github.io/ele520_math_data/lectures/lasso_algorithm_extension.pdf, slides 22, 27 r@r)r;)rrrrr=r>r?rr@r coef_prevgapr2mom_terms r:optimizer_steprF{sTI$$dHi*ND#s 'C Q-CAG}Hh,$&I)==J tS# %%r<tensorc z|dddddddfjddtji|dS)aTConvert tensor to channel-last memory layout for optimized convolution operations. Parameters ---------- tensor : torch.Tensor Input tensor to convert. **kwargs Additional keyword arguments passed to tensor.to(). Returns ------- torch.Tensor Tensor with channel-last memory layout. N memory_format)tor) channels_lastsqueeze)rGkwargss r:_to_channel_last_layoutrPsO !qqq!!!QQQ} MMu/BMfMMUUVWXXr<xcxtj|tjtj|zS)aApply signed square root transformation to input tensor. This transformation preserves the sign while applying square root to the absolute value, which can help with numerical stability and gradient flow. Parameters ---------- x : torch.Tensor Input tensor. Returns ------- torch.Tensor Transformed tensor with same shape as input. )r)signsqrtr+)rQs r: _signed_sqrtrUs) :a==5:eill33 33r<c 2eZdZdZdeeedfeedffdeedfdede j dd f d Z d ed ede eed fe edfffdZ ed ed ede eedfeeedfefeedfffdZd S)BatchLoaderBaseaBase class for loading batches of contribution scores and sequences. This class provides common functionality for different input formats including batch indexing and padding for consistent batch sizes. Dimension notation: - N = number of sequences/regions in dataset - L = sequence length - B = batch size (number of regions processed simultaneously) Parameters ---------- contribs : Union[Float[Tensor, "N 4 L"], Float[Tensor, "N L"]] Contribution scores array. sequences : Int[Tensor, "N 4 L"] One-hot encoded sequences array. L : int Sequence length. device : torch.device Target device for tensor operations. rN 4 LN Lrr@devicerNc>||_||_||_||_dS)N)rrr@rZ)selfrrr@rZs r:__init__zBatchLoaderBase.__init__s$! " r<startendz Z.c||z }t||jjd}|||z z }ddddd|f}tjt j||t jd|fd|j }||fS)aGet indices and padding lengths for batch loading. Parameters ---------- start : int Start index for batch. end : int End index for batch. Returns ------- inds : Int[Tensor, " Z"] Padded indices tensor with -1 for padding positions (shape: padded_batch_size). pad_lens : tuple Padding specification for F.pad (left, right, top, bottom, front, back). r)dtype)value)rZ) minrshaper$padr)arangeintrLrZ)r\r^r_Noverhangpad_lensindss r:_get_inds_and_pad_lensz&BatchLoaderBase._get_inds_and_pad_lenss& %K#t}*1-..e $q!Q8,u L59 5 5 58}B   "DK"  X~r<rrcdS)uLoad a batch of data. Dimension notation: - B = batch size (number of regions in this batch) - L = sequence length Parameters ---------- start : int Start index (used by subclasses). end : int End index (used by subclasses). Returns ------- contribs_batch : Float[Tensor, "B 4 L"] Batch of contribution scores (shape: batch_size × 4_bases × L). sequences_batch : Union[Int[Tensor, "B 4 L"], int] Batch of one-hot encoded sequences (shape: batch_size × 4_bases × L) or scalar 1 for hypothetical mode. inds_batch : Int[Tensor, " B"] Batch indices mapping to original sequence indices (shape: batch_size). Notes ----- This is an abstract method that must be implemented by subclasses. Parameters are intentionally unused in the base implementation. NrK)r\r^r_s r: load_batchzBatchLoaderBase.load_batch=s B r<)__name__ __module__ __qualname____doc__rr r r rhr)rZr]rrmr rorKr<r:rWrWs<, fgo.fem0DDE vw'          " s64< %S/1 2<    "   fgoc&'/&:C&? @#fdlBSS     ^    r<rWc heZdZdZdededeeedfeedfeedfffdZ dS) BatchLoaderCompactFmtzBatch loader for compact format contribution scores. Handles contribution scores in shape (N, L) representing projected scores that need to be broadcasted to (N, 4, L) format. r^r_rrrcl|||\}}tj|j||dddf|}t ||jt j}tj|j||ddddf|}t ||jt j }||z}|||fSNrZra rmr$rfrrPrZr)float32rint8)r\r^r_rlrkcontribs_compactcontribs_batchsequences_batchs r:roz BatchLoaderCompactFmt.load_batchhs44UC@@h5uSy$/A!BHMM0 T[    %uSy!!!QQQ ?JJ1 DKuz   (/944r<N rprqrrrsrhrr r r rorKr<r:ruruasq 55"5 uVW_%s67?';S=NN O555555r<ruc heZdZdZdededeeedfeedfeedfffdZ dS) BatchLoaderProjzBatch loader for projected contribution scores. Handles contribution scores in shape (N, 4, L) where scores are element-wise multiplied by one-hot sequences to get projected contributions. r^r_rrrcp|||\}}tj|j||ddddf|}t ||jt j}tj|j||ddddf|}t ||jt j }||z}|||fSrwry)r\r^r_rlrk contribs_hypr~r}s r:rozBatchLoaderProj.load_batchs44UC@@huT]59aaa?;XFF . EM   %uSy!!!QQQ ?JJ1 DKuz   &744r<NrrKr<r:rr{sq 55"5 uVW_%s67?';S=NN O555555r<rc XeZdZdZdededeeedfeeedfffdZ dS) BatchLoaderHypzBatch loader for hypothetical contribution scores. Handles hypothetical contribution scores in shape (N, 4, L) where scores represent counterfactual effects of base substitutions. r^r_rrrc|||\}}tj|j||ddddf|}t ||jt j}|d|fS)Nrxr)rmr$rfrrPrZr)rz)r\r^r_rlrkr}s r:rozBatchLoaderHyp.load_batchsu44UC@@ht}U3Y111_=xHH0 4;em   q$&&r<NrrKr<r:rrsh ' '" ' uVW_%sC ,== > ' ' ' ' ' 'r<rrB{Gz?FMb@?'ffffff?Tr rXrY cwm_trim_maskzM Wuse_hypotheticalz M step_size_max step_size_minsqrt_transformconvergence_tol max_steps batch_size step_adjust post_filterrZcompile_optimizerepsc |j\}}}|j\}}}| }|atjrtjd}n.tjd}t jdt|rtjtda tj |}tj |}tj |}tj |dddddf ddd}tj |ddddf |tj }t||tj }t||tj }||z}|rJt|}|d zd }||ddddfz }t%|jd kr(|rt'||||}nft)||||}nSt%|jd kr$|rt+dt-||||}nt+d|jg}g}g} g}!g}"ggggggd}#tj||||z dzf}$t|$|tj }$tj|$}%tj|ddftj|}&tj|ddf|tj |}'tj|fdtj|}(|})tj|d|f}*t|*|tj }*|rd}+n3tj|d|f}+t|+|tj }+tj||||z dzf},t|,|tj },tj|ftj|}-tj|ftj|}.t=dd|d5}/d}0d}1|0|kr|)dkr`|1}2|2|)z}3t?|3|jd}1| t3|2t3|3}4|4\}5}6}7|rt|5}5|5d zd |z }8tj!|5|8ddddfz }5tEj#|5d z||zdz}9|9$d}9|5|*|(ddddf<|s |6|+|(ddddf<|9|,|(ddddf<|7|-|(<|8|.|(<|$|(ddddfxxdzcc<|%|(ddddfxxdzcc<|&|(xxdzcc<||'|(<t||*|,|+|$|%|&|'|| \}$}%}:};|&dz }&|-dk}|>'dkr.|-|>}?|?D]#}@t jd|@d| dt$|'|k&|z|Az|>> hits_df, qc_df = fit_contribs( ... cwms=motif_cwms, ... contribs=contrib_scores, ... sequences=onehot_seqs, ... cwm_trim_mask=trim_masks, ... use_hypothetical=False, ... lambdas=np.array([0.7, 0.8]), ... step_size_max=3.0, ... step_size_min=0.08, ... sqrt_transform=False, ... convergence_tol=0.0005, ... max_steps=10000, ... batch_size=1000, ... step_adjust=0.7, ... post_filter=True, ... device=None, ... compile_optimizer=False ... ) Ncudacpuz!No GPU available. Running on CPU.T) fullgraphrrxrrrrJz=Input regions do not contain hypothetical contribution scoresz0Input contributions array is of incorrect shape )r2r8 num_steps step_size global_scalepeak_id)rarZregionsx)disableunittotalncolsrg r!zRegion z$ has not converged within max_steps=z iterations.zOptimizer failed for region .)forcer2r8rrrraxis)rmotif_id hit_starthit_coefficienthit_similarityhit_importancehit_importance_sqcBi|]\}}|tj|dS)rr)np concatenate).0kvs r: z fit_contribs..6s-WWWtq!a!:!:!:WWWr<):rer)r is_availablerZwarningswarnRuntimeWarningcompilerF from_numpyrepeatrLrzrPrUr'rTlenrr ValueErrorruzeros zeros_likerhfullboolr{floatrrdro nan_to_numr$r&r*isfiniterNitemr+ to_sparsecloneindices embeddingunbindvaluesappendnumpyTastyperuint32updateritemspl DataFrame)`rrrrrrrrrrrrrrrZrrM_Wrir@B cwms_tensorcontribs_tensorsequences_tensorcwm_trim_mask_tensorlambdas_tensorcwm_norm batch_loader hit_idxs_lstcoefficients_lstsimilarity_lstimportance_lstimportance_sq_lstqc_lstsr=r>r?r convergednum_load contribs_bufseqs_bufimportance_scale_bufinds_bufglobal_scale_bufpbar num_completenext_ind load_startload_end batch_datar} seqs_batch inds_batchglobal_scale_batchimportance_scale_batchrDr2activedivergedtimeouts timeout_indsindfails fail_indsinds_outglobal_scale_outcoef_outimportance_scale_out_dense importance_sq xcor_scalecontribs_convergedimportance_sum_out_densexcov_out_densexcor_out_dense hit_idxs_out ind_tupleimportance_outimportance_sq_outxcor_outscores_out_rawgap_outnll_outstep_outstep_sizes_outhit_idxsscores_coefficientscores_similarityscores_importancescores_importance_sqhitsqchits_dfqc_dfs` r: fit_contribsr s xjGAq!oGAq!A ~ : " " $ $ O\&))FF\%((F M=~ N N NG~FFF!& 0 6 6K$)$4X$>$>O%*%5i%@%@ +M::111dAAA:FMMaQRTUVV#(#3G#<#q)ABB)44S__c(mmTT 9C6 J!B%1.%A%AN'5q'8&=&=&&=&I&IA&M%S%S%U%U"!&!1"%74 %FF"" H^Q.0DEEK*&*@)E)E")E)M)M&0> Y111_-';0:HY111_-8N$Y111_5&0#.@ +9aaa?+++q0+++Y111_%%%*%%%) ! (5 9%*8$ * * &Jc3 FA]Fs+++f4H xAAA~ & & &! + & & & 111aaa A % hKKK1 KKK xAAA~ & & &+ 5 & & &I ..0069H||~~""$$q(('1 'CMb#bb9bbb&  -/88::VCEyy{{!!A%%$UO $YYCM"G"G"G"GXXXX0H  111aaa0-A)QQQPQPQPQ/-R* :r BS H *//11 %1)QQQ/%B"+,8I0113G,,("#*^+KLH$--// %{8+;+;+=+=>> %&[ AAA&D(9&&'))QT" "L)9)9););<< !9)!D$1)$<!))4!)!2!2i.i.Y1_-!+Iq!O!<##L$6$6T$6$B$B$DEEE ''(<(<4(<(H(HIII%%hnn4n&@&@AAA%%n&:&:&:&F&FGGG!(():)@)@t)@)L)LMMM%%gmm$m&?&?@@@ #**7==t=+D+DEEE $++HNNN,F,FGGG'../?/E/ED/E/Q/QRRR $++N,@,@t,@,L,LMMM "))(..t.*D*D*K*KBI*V*VWWW(  H%%%WQN&N&N&N&N&N&N&N&N&N&N&N&N&N&N&b~l333H(8qAAA~A>>>~A>>>>*;!DDDAAAqD>((33QQQTN))")44aaad^-++1  DXWw}}WWWBl4  G L  E E>s.[*l$$l(+l() rBrFrrrrTNFr ))rsrtypingrrrrrabcrr rrr r)torch.nn.functionalnn functionalr$r polarsr jaxtypingr r rrrhr;rFrPrUrWrurrrrrZrr rKr<r:r(sN  55555555555555######## &&&&&&&&&&\!(\!FGO,\!  \!FGO$ \! S)3./ \! 67? # \!fgo&\! 5 !5#6fdl8K KL\!\!\!\!~W&  W&FGO$W&FGO,W&S)3./ W& fgo& W&  W& VW_W&fgo&W& W&67? #W&  &'/ &'/ &$, &$,W&W&W&W&tF(4EL4U\4444&b b b b b cb b b J55555O555455555o5552'''''_'''6 #%)##SS  !SE'7*+U7E>-BBCS7G#$S%( S  S 7D= ! SSSSSSSSS U\ "S !S" #S$ 2< %&%SSSSSSr<