o Uݢgh"@sddlmZddlZddlZddlmZddlZddlm Z m Z Gddde Z Gddde Z d d Zd d Z d!d"ddZd#ddZd$ddZd%dd ZdS)&) annotationsN)Counter) ensure_binary ensure_strcs$eZdZfddZddZZS)CSVParseExceptioncst|||_dSN)super__init__msg)selfr  __class__c/oak/stanford/groups/akundaje/marinovg/programs/cellranger-9.0.1/lib/python/cellranger/csv_utils.pyr s  zCSVParseException.__init__cCs|jSr)r )r rrr__str__szCSVParseException.__str__)__name__ __module__ __qualname__r r __classcell__rrr rr s rc@seZdZdZdS)CSVEmptyExceptionzThe CSV file has no data rows.N)rrr__doc__rrrrrsrc csjt|ddd}d}d}z:|D]5}|}|sq|dsEd}|r1ddd |dD}d}|sBtd |d |d ||VqWnOty}zCt|j d d}|j |} |j ||j d j ddd dd dd} d| d} td |d |d|j d|j d| d| |d}~wwWdn1swY|std |d |ddS)zIterates through non-comment lines in the file, returning them as unicode strings. Also skips the BOM if present. Raises: CSVParseException: The file has no non-comment lines or some lines can't be decoded as ascii. z utf-8-sigN)encodingnewlineFT#,css|]}|VqdSr)strip).0xrrr 2sz"_iter_file_rows..The csv file z& contains non-ascii characters. Row: rzutf-8replace)errors   ^z+ has one or more invalid utf-8 characters: z2. The first bad character is at absolute position z9. This snippet shows the offending character in context: z has no data.)openr startswithjoinsplitisasciirUnicodeDecodeErrormaxstartobjectdecoder"reasonr) filename_bytesfilenamedescriptive_namefZhas_datafirstrowerrZ context_startZbadchar_offsetZsnippetZcaretrrr_iter_file_rowssb   ,r:c Cs|j}t|}|t|std|d|d||rSt|}||sSt||}d|d|d|}|rG|dd|7}|d|d7}t|t |t |krsd|dtdd t | D}t|dS) NzThe {} file header does not contain one or more required comma-separated fields: "{}". The following fields were found: "{}". Please check that your file is in CSV format and has the required field names.z, zThe {} file header contains invalid columns: "{}". Only the following comma-delimited fields may be used in the file header: "{}".z* The following columns are required: "{}".z* Please check that you have formatted the z< file correctly and included the appropriate column headers.z"{} csv has a duplicated column: {}css |] \}}|dkr|VqdS)Nr)rnamecountrrrrvsz$_validate_columns..) Z fieldnames frozenset issupersetsetrformatr*issubsetsortedlenritems) readerr5 required_cols valid_colsZ col_namesZcol_setZ valid_setZ extra_columnsr rrr_validate_columnsRsF   rIr4 os.PathLikecCsxtt|}tj|std|d|t|tjs*d|d|}t|t t |||}t |||||S)a1Returns non-comment lines from the csv files. Verifies ASCII encoding and no duplicate columns. Args: filename: The csv file to open. descriptive_name (str): The description of the file, to use in error messages. required_cols (sequence of str): columns which must be present. valid_cols (sequence of str): columns which may be present. If provided, then it is an error for other columns to be present. Returns: csv.DictReader: The non-comment rows. Raises: CSVParseException zCould not find the r rz5 csv is not readable, please check file permissions: ) rosfspathpathisfileraccessR_OKcsvZ DictReaderr:rI)r4r5rGrHr3r rFrrrload_csv_filter_comments{s rRout_csvbcs_per_genomedict[str, list[str]]cCspt|d)}tj|dd}|D]\}}|D] }|t|t|gqqWddS1s1wYdS)zWrite the barcodes to a CSV. Args: bcs_per_genome (dict of str to list): Map each genome to its cell-associated barcodes wr&ZlineterminatorN)r(rQwriterrEwriterowr)rSrTr6rXZgenomeZbcsZbcrrrwrite_filtered_barcodess "rZ str | bytes barcode_list np.ndarray in_tissue list[bool]normalization_arrayc Cs|j|jksJdd|jdd|jd|jdt|ks4Jdd|jddt|dt|d 6}tj|d d }|gd t|||D]\}}}|t|tt |t j |d dgqNWddS1srwYdS)aKWrite the normalization constant for each barcode to a CSV. Args: out_csv (Union[str, bytes]): Path of the CSV barcode_list (np.ndarray): list of barcodes in_tissue (list[bool]): boolean list of in tissue normalization_array (np.ndarray): normalization for each barcode - in the same order z;Lengths of barcodes and normalization factorsdo not match. zbarcode list length: z. zNormalization array shape: .rz6Lengths of barcodes and in tissue vectordo not match. zIn tissue length: rVr&rW)barcoder^Znormalization_factor)Z precisionN) shaperDr(rQrXrYziprstrintnpZformat_float_positional) rSr\r^r`r6rXrbZ bc_in_tissueZnormalized_valuerrrwrite_isotype_normalization_csvs8      "rir;c Cs|sd}dSt|dN}t|D]@\}}t|0}g}t|D] }|t|q|dkr7|D]} || q/|D]} || q9Wdn1sKwYqWddS1s\wYdS)zCombine a list of CSV files. Files specified in input_csvs are combined into a single output csv in output_csv. It is assumed that all CSVs have the same header structure. The number of header lines is specified in header_lines. NrVr)r( enumeraterangeappendnextwrite) Z input_csvsZ output_csvZ header_linesoutiZicsvinfileheader_Zhlinelinerrr combine_csvs&     "rur)r4rJ)rSrJrTrU)rSr[r\r]r^r_r`r])r;) __future__rrQrK collectionsrZnumpyrhZsixrr Exceptionrrr:rIrRrZrirurrrrs   8* # )