o Uݢgy@sdZddlmZddlmZddlmZddlmZddl Z ddl Z ddl ZddlmZmZddlmZddlmZdd lmZmZdd lmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%gd Z&d Z'd Z(dZ)dZ*dZ+dZ,dZ-dZ.dZ/dZ0ee,e'egZ1dZ2ddZ3 dedfddZ4d d!Z5dgdhd%d&Z6   didjd/d0Z7dkd7d8Z8dld?d@Z9 " " A " A A A " "     "dmdndEdFZ:dodpdGdHZ;dqdKdLZ< drdsdOdPZ=dhdQdRZ>dhdSdTZ?dhdUdVZ@dde2dfdtd[d\ZAdudadbZBdde2dfdvdcddZCdS)wzXUtils for summarizing and computing statistics on RNA reads and UMIs from molecule_info.) annotations)Iterable)deepcopy)AnyN) ensure_binary ensure_str)molecule_counter_extensions)GENOME_FEATURE_TAGFeatureReference) BARCODE_IDX_COL_NAMECOUNT_COL_NAMEFEATURE_IDX_COL_NAMEGEM_GROUP_COL_NAMELIBRARY_IDX_COL_NAMEMOLECULE_INFO_COLUMNS UMI_COL_NAMEUMI_TYPE_COL_NAME BarcodeInfoMoleculeCounter) feature_typeidnameindexis_cellz{}_cellsZ num_readsZnum_umisZdup_fracbarcode num_barcodes num_featuresZ feature_idZ feature_namei-1cCstjtjtjtjg}|j|vr0tt|}t|tj r"|t|j}t d|t j |ddS|d| |j}|D]}t|}|j}t||k}|rV||Sq<|S)a A roll our own version of `pd.to_numeric(...,downcast="unsigned")`. Profiling showed that the pandas version was incredibly memory and CPU intensive due to some inefficient code leading to an expensive call to `np.allclose(new_result, result, rtol=0)` z=Downcasting function expected a uint64 or smaller, type was: unsigneddowncastN)npuint8uint16uint32uint64dtypestrtype isinstancendarrayprintpd to_numericriinfomaxallastype)xlevels type_nameZto_checkZ smaller_typeiimax_valbelowr7f/oak/stanford/groups/akundaje/marinovg/programs/cellranger-9.0.1/lib/python/cellranger/pandas_utils.py_downcast_large_uint_vector<s     r9rr1,float | int | tuple | np.ndarray | pd.Seriesr str | NonecCsL|dkrt|dkr|dSt|S|dvrtj||dStd|d)aWrapper around pandas.to_numeric to avoid bugs and slow performance. Args: x: value to be converted downcast: one of (None, 'unsigned', 'float') Returns: (depends on input type): input value as numeric type, possibly downcast rrr!)Nfloatrz1to_numeric wrapper not implemented for downcast='')lenr0r9r+r,NotImplementedError)r1rr7r7r8to_numeric_safeUs  r@cCst|tr |S|SN)r(bytesdecodevr7r7r8 _conv_bytesqs rFFdf pd.DataFramecCs|s|jdd}|jjtkr|jtdddn|jjtkr%|jtddd|jtddd|jD]T}||jtkrC|| t||<q0||jtkrT|| t||<q0t ||jt j r||j jjtkrq||j t||<q0||j jjtkr||j t||<q0|S)z*Convert any bytes in the DataFrame to str.F)deeprTaxisinplacecolumns)copyrr%rBrenamerobjectrFrMapplyr(r+CategoricalDtypecat categoriesrename_categories)rGrLcolr7r7r8sanitize_dataframexs&    rWmcrfilter_library_idxint | list[int] | Nonefilter_feature_idxfilter_barcode_idxreturn#np.ndarray[int, np.dtype[np.bool_]]cCsv|durtj|td}n t|t|}|dur&|t|t|M}|dur9|t |t gdd|DM}|S)Nr%cSsg|]}|fqSr7r7.0r1r7r7r8 sz _get_idx_mol..) r onesnrowsboolisin get_columnrget_column_lazyr cr_mceget_indices_for_valuesr )rXrYr[r\idx_molr7r7r8 _get_idx_molsrlrk exclude_cellsreexclude_noncellswith_cell_call*np.ndarray[int, np.dtype[np.bool_]] | Nonec Cs|r|rtd|s|s|r{|j}tjt|td}t|ddtj fD].}||ddtj f|kdf}t |t g|fg|} | t |t g||M} || O}~ q(|ri||dk|M<||S|ry||dk|M<||S|SdS)z#Get vector of cell calls if needed.z&Can't exclude both cells and non-cellsr_Nr) ValueErrorget_barcode_info pass_filterr zerossumresetrPASS_FILTER_LIBRARY_IDXrirjrr ) rXrkrmrnrortr library_idxZwhich_barcodesZ is_bc_and_libr7r7r8 _get_is_cells:    rzwith_umiwith_gem_groupwith_library_idx with_umi_type downsample float | Nonec Cs@t||| | } t|| |||} i}| durUd| kr dks#JJ|t| dk}tj|| }| dur?|r?| |dk} | | dk|dkM<||dktj}|r[| |t <~ t t }|se|t =|sj|t =|so|t=|st|t=|D]#}|tkr| dur|}~n||| }t|}|tkrt}|||<qv~ t|S)Nrrq)rlrzrgr r randombinomialr0r#MOL_INFO_CELL_COLrrrrrrr@FEATURE_DF_COUNT_COLr+ DataFrame)rXrnrmr{r|r}ror~rYr[r\rrkrZ dict_constZnew_read_countsZ mol_info_colsr1valr7r7r8_mol_info_df_from_h5sD    rTwith_barcode_seqwith_feature_info genome_colc Cst|tr d}|}nd}t|d}t|||||||| | | | | }|r6||t|t<|td|t<|rNt||}|j ddidd|j |ddd }|sT| |S) avLoad molecule_info.h5 file as a pandas DataFrame. Returns a data frame containing the per-molecule columns from the molecule_info file (barcode_idx, count, feature_idx, gem_group, library_idx, umi) along with a column 'is_cell'. Numeric columns are automatically downcast to the smallest appropriate type. Args: mol_info (str or object): path to a cellranger molecule_info.h5 file or instance of MoleculeCounter object exclude_noncells (bool): exclude molecules from non-cell-associated barcodes exclude_cells (bool): exclude molecules from cell-associated barcodes with_umi (bool): include a column 'umi' containing the 2-bit encoded UMI with_barcode_seq (bool): include a column 'barcode' containing the full barcode sequences with_gem_group (bool): include a column gem_group with_library_idx (bool): include `library_idx` column with_cell_call (bool): include a column 'is_cell' indicating whether the barcode associated with each molecule was called as a cell with_feature_info (bool): merge the entire result with the feature reference (not memory-efficient, but useful for small analysis tasks) with_umi_type (bool): include a column `umi_type` indicating whether the UMI is transcriptomic or not [default: False] filter_library_idx: library indices (i.e., into the library_info dataset) to include, or None to include all of them filter_barcode_idx: barcode indices (i.e., into the barcodes dataset) to include, or None to include all of them. Note that exclude_noncells and exclude_cells will still be applied if True filter_feature_idx: feature indices (i.e., into the features dataset) to include, or None to include all of them downsample: between 0 and 1 exclusive, or None. The rate at which to downsample reads in the molecule info. genome_col: If with_feature_info is set, will add a genome column to the dataframe. Returns: pd.DataFrame: molecule_info data (see above) TFrcategoryr feature_idxrMrLlefthowon) r(ropenr get_barcodesr FEATURE_DF_BARCODE_SEQ_COLr0feature_ref_from_h5rOmergeclose)Zmol_infornrmr{rr|r}rorr~rYr[r\rrZ obj_passedrXrG feature_refr7r7r8mol_info_from_h5s: 7  rc Cs<t|tr |}n"tt|d}d|vr|dd}n|d}t|}| |j }t t }dd| D}|rZt| D]\}} ||| jtdqD|ttj||d} ~~|durq| j| d|} | jd d d | jd d d | jD]} | d vr| | d| | <q| jttdd d| S)a?Load feature reference from h5 file as a pandas DataFrame. Args: fname_or_mol_info (str or object): path to a cellranger molecule_info.h5 file or instance of MoleculeCounter object, or a path to a matrix h5 file genome_col (bool): Do we want a column with the "Genome" in it? This is to preserve older behaviour used by puppy.panda_utils filter_feature_types (list or None): list of feature types to select from feature reference, all are kept if None Returns: pd.DataFrame: the feature reference rmatrixfeaturescsg|] fddtDqS)csg|]}t|qSr7)getattr)rarVrDr7r8rb|z2feature_ref_from_h5...)FEATURE_REF_COLS)rar7rDr8rb|sz'feature_ref_from_h5..N)rMrrT)rL)droprL)rrr)rrr)r(rget_feature_refh5Filerkeysr Z from_hdf5rZid_maprrvalues enumerateappendtagsgetr r+rlocrf sort_values reset_indexrMr0rOFEATURE_ID_COLFEATURE_NAME_COL) Zfname_or_mol_inforZfilter_feature_typesrfgrouprMvalsirrGrVr7r7r8rbs>      rfeature_ref_df mol_info_dfcCslt|}t|t}dd|jD|_tj||gdd}|j}|j|ddtd}||jdd d ||<|S) arCompute a per-feature summary from a molecule_info DataFrame. For each feature, these metrics are computed: num_umis - # UMIs associated with the feature num_reads - # reads associated with the feature num_barcodes - # distinct barcodes containing at least one read/molecule mapped to the feature dup_frac - proportion of reads corresponding to already observed molecules, i.e., sequencing saturation. This is (1 - num_umis/num_reads), and 0 in case there are no reads. Args: feature_ref_df (pandas.DataFrame) mol_info_df (pandas.DataFrame) Returns: pandas.DataFrame: per-feature summary statistics from mol_info_df with the following columns * feature_type category * feature_id object * feature_name category * index int64 * num_umis int64 * num_reads int64 * num_barcodes int64 * dup_frac float64 * num_umis_cells int64 * num_reads_cells int64 * num_barcodes_cells int64 * dup_frac_cells float64 cSsg|]}t|qSr7)IS_CELL_FORMAT_STRINGformatr`r7r7r8rbrz(summarize_by_feature..rqrKrr)rleft_onright_onrinferr) _feature_summaryqueryrrMr+concatrr fillna)rrrawfilteredcombinedZ combined_colsresultr7r7r8summarize_by_features rfeature_idx_subsetsdict[str, list[int]] | Nonec Cst|}|rfd|d<dd|jD}||}|D]5\}}t|j|}t||}|j|ddd|j|dddd }|jd d dd ||d<t ||g}q|j d |_ dd|jD} |dg| }|S)aCompute a per-barcode summary from a molecule_info DataFrame. For each feature, these metrics are computed:: num_umis - # UMIs associated with the feature num_reads - # reads associated with the feature num_features - # distinct features containing at least one read/molecule assigned to this barcode dup_frac - proportion of reads corresponding to already observed molecules, i.e., sequencing saturation. This is (1 - num_umis/num_reads), and 0 in case there are no reads. All features are currently treated the same (regardless of feature_type or genome). The subset_feature_idx argument can be used to produce separate metrics on a defined subset of features. Any metadata columns will be propagated (barcode_idx, barcode sequence, and is_cell, as defined by BARCODE_METADATA) Args: mol_info_df (pandas.DataFrame): TODO feature_idx_subsets (Dict[str, List[int]]): produce separate metrics for certain named subsets of features, specified by their indices within the feature reference (feature_idx column in MoleculeCounter). There will be an additional output column called 'feature_subset'; the rows marked 'all_features' contain the metrics over all features. Returns: pandas.DataFrame: per-feature summary statistics from mol_info_df Z all_featuresfeature_subsetcSg|]}|tvr|qSr7BARCODE_METADATAr`r7r7r8rbz(summarize_by_barcode..rqTrJr)r left_index right_indexrr)rrLrcSsg|]}|dkr|qS)rr7r`r7r7r8rbr) _barcode_summaryrMitemsr rfrrrrr+rrr0) rrrZ metadata_colsmetadataZ subset_namesubsetZ idx_subsetZsummary_subsetZ main_colsr7r7r8summarize_by_barcodes&! rcCs$d||jdd||jddS)zCalculate the fraction of duplication/sequencing saturation. Args: df: pandas dataframe umi_col_name: column with umi counts read_col_name: column with read counts Returns: A vector of duplicate fractions rqlower)clip)rGZ umi_col_nameZ read_col_namer7r7r8calculate_duplicate_fractions$ rc CsX|t}ttt|ttt|tt t|t  i}t |tt|t <|S)z%Aggregation for summarize_by_feature.)groupbyr r+rFEATURE_DF_UMI_COLr@rcountrvFEATURE_DF_BARCODE_COUNT_COLr nuniquerFEATURE_DF_DUP_COL)rGgroupedrr7r7r8rs  rc s|tdd|jD}tfdd|D}ttttttt t tt  i}d|tj dd|tj dd|t<tj||gddS)z%Aggregation for summarize_by_barcode.cSrr7rr`r7r7r8rb,rz$_barcode_summary..csi|] }||qSr7)firstr`rr7r8 -sz$_barcode_summary..rqrr)rr rMr+rrr@rrrvFEATURE_DF_FEATURE_COUNT_COLr rrrr)rGZmetadata_presentrrr7rr8r&s rlist[int] | Nonebarcode_genome tgt_chunk_lenintcs}|j|dur|j|}ddtjf|k|dur+ddD}}t j |t j dt j |t j dt j |t j dt j |t j ddfd d }j |d d D] \} } || | |qet tttttdttdtttt jdi} dD]} d| t| jdd| t| jdd| t| <qtd d} | jdtid d| j| dtd} | S)auGet read and UMI counts per feature by aggregating counts over mol_info. in chunks. Args: mc (MoleculeCounter): instance of MoleculeCounter object filter_library_idx (list of ints or None): specifies whether to restrict counts to only certain libraries barcode_genome (str | None): Use only barcodes from this genome. If None, use all barcodes. tgt_chunk_len (int): number of rows by which to chunk mol_info downsample (float): downsample fraction Returns: pandas.DataFrame containing per-feature summary statistics (UMIs, reads, dup_rate, and feature metadata) NcSsg|]}t|dqS) library_id)r)ralibr7r7r8rb[rz+collapse_feature_counts..r_ chunk_startr chunk_lenrY list[int]c sr||}tj|dd}|r|tt|||M}tj|jdd}|D] }ddtjf|kdf}|tt |||O}q&||}t |||}t |||} ~durdkrtj | } || dk}|| dk}| | dk} tj|t|jdtj|| ||}| |} tj|t|jdtj|| dS)Nrer_r?)r rcrfrhrrushaperrxr r r rraddat) rrrY chunk_stoprkrryZcell_barcode_indicesZfeature_indices read_countsZbarcode_passing_filterrrXread_counts_by_indexZread_counts_in_cells_by_indexumi_counts_by_indexZumi_counts_in_cells_by_indexr7r8collapse_chunkcs<   z/collapse_feature_counts..collapse_chunkTZpreserve_boundaries_cellsr)rrqr)rrrrr)rrrrrYr)rsrtgenomesrrZPASS_FILTER_GENOME_IDXget_library_inforZget_num_featuresr rur$ get_chunksr+rrr@rr arangerrrrrOr)rXrYrrrZ barcode_infoZ genome_idxrrrrrsuffixrr7rr8collapse_feature_counts>sF  '  r library_infoIterable[dict[str, Any]]bcs_passing_filter np.ndarrayc Csp|D]3}t|d}|dur|d|vrq|t}||dddf|kdf}t||d|}d||<q|S)NrrqrT)rrr r) rrrrYrrZlib_idxggZ cell_indicesr7r7r8_barcode_is_cells  rc s0ttt}tt|}tj t|tj d}tj t|tj d}dfdd } j |d d D] \} } | | | ||q>t tt|tt|i} ~~|| t<~ttj| jd d tdj| t<| | td k| tB} d| tjdd| tjdd| t<| S)a6Get read and UMI counts per feature by aggregating counts over mol_info in chunks. Args: mc (MoleculeCounter): instance of MoleculeCounter object filter_library_idx (list of ints or None): specifies whether to restrict counts to only certain libraries filter_feature_idx: TODO: document tgt_chunk_len (int): number of rows by which to chunk mol_info downsample (float): downsample fraction Returns: pandas.DataFrame containing per-barcode summary statistics (UMIs, reads, dup_rate, and barcode metadata) r_rrrr$np.ndarray[int, np.dtype[np.uint64]]rc s4||}tj|td}r|tt||M}r-|tt||M}t|||}t|||}t |||}durodkrotj |}||dk}||dk}||dk}~t t |d} tj|t|| t|jdtj|t|| |dS)Nr_rrrq)r rcrerfrhrr r rr rrmultiplysubtractrrr) rrrrrrkbarcode_indicesZ gg_indicesrZ bc_idx_offsetrr[rYrXrr7r8rs<     z/collapse_barcode_counts..collapse_chunkTrrF)r fill_valuer%rqrN)rrrrrrrr)sortedlistrwZget_gem_groupsr>rcr_utilsZformat_barcode_seqsr rur$rr+rrr@rrrfullrrerrsrtrrr) rXrYr[rrZ gem_wellsZordered_barcodesrrrrrrr7rr8collapse_barcode_countss8 , r )r)r1r:rr;)F)rGrH)NNN) rXrrYrZr[rZr\rZr]r^) rXrrkr^rmrernrerorer]rp)rXrrnrermrer{rer|rer}rerorer~rerYrZr[rZr\rZrr)FFTFTTTFFNNNNF)rnrermrer{rerrer|rer}rerorerrer~rerYrZr[rZr\rZrrrrer]rH)FN)rre)rrHrrHrA)rrHrr) rXrrYrrr;rrrr)rrrrrYr)rXrrYrrrrr)D__doc__ __future__rcollections.abcrrNrtypingrZh5pyrnumpyr pandasr+sixrrZcellranger.utilsutilsr  cellrangerrriZcellranger.feature_refr r Zcellranger.molecule_counterr r r rrrrrrrrrrrrrrrrrrrZ CHUNK_SIZEr9r@rFrWrlrzrrrrrrrrrrr r7r7r7r8s      0    $> Z :0 :   e