B îq\á˜ã@sÐdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl Z ddl m Z ddl mZddl mZddl mZdd l mZdd l mZdd l mZdd l mZdd l mZddl mZddl mZddl mZddl mZddlmZmZmZddlmZddl m!Z!m"Z"gZ#yddl$Z$dZ%Wne&k rJdZ%YnXda'd4dd„Z(dd„Z)d5dd„Z*dd„Z+d d!„Z,d6d"d#„Z-d$d%„Z.d&d'„Z/d(d)„Z0d*d+„Z1d,Z2d7d-d.„Z3d8dd/œd0d1„Z4d2d3„Z5dS)9zèAn extensible ASCII table reader and writer. ui.py: Provides the main user functions for reading and writing tables. :Copyright: Smithsonian Astrophysical Observatory (2010) :Author: Tom Aldcroft (aldcroft@head.cfa.harvard.edu) éN)ÚStringIOé)Úcore)Úbasic)Úcds)Údaophot)Úecsv)Ú sextractor)Úipac)Úlatex)Úhtml)Úrst)Ú fastbasic)Úcparser)Ú fixedwidth)ÚTableÚvstackÚ MaskedColumn)Úget_readable_fileobj)ÚAstropyWarningÚAstropyDeprecationWarningTFé †cstˆtƒs€y`ˆdˆdd…d}x*tˆƒD]\}}|t|ƒ7}||kr.Pq.Wtj ˆd|d…¡‰Wntk r~YnXtˆtƒrˆd|…‰t  dˆtj tj B¡r²dSt  dˆdd…tj ¡rÚtj  ˆ¡rÚdSt  dˆtj tj B¡rôdSt‡fd d „d DƒƒrdSd S) zo Determine if ``table`` probably contains HTML content. See PR #3693 and issue #3691 for context. rNrz+( http[s]? | ftp | file ) :// .+ \.htm[l]?$Tz \.htm[l]?$éûÿÿÿz\s* ]* >N)ÚreÚsearchÚformatÚ IGNORECASEÚVERBOSE)Ú.0Úelement)Útable©ú2lib/python3.7/site-packages/astropy/io/ascii/ui.pyú \sz!_probably_html..)r ZtrZtdF)Ú isinstanceÚstrÚ enumerateÚlenÚosÚlinesepÚjoinÚ ExceptionrÚmatchrrrÚpathÚexistsÚall)r ZmaxcharsÚsizeÚiÚliner!)r r"Ú_probably_html5s2     $  r3cCs|adS)z° Set the default value of the ``guess`` parameter for read() Parameters ---------- guess : bool New default ``guess`` value (e.g., True or False) N)Ú_GUESS)Úguessr!r!r"Ú set_guesscs r6cKsF|dkr*t|ƒ}|ddkr$tj}ntj}tj|f||dœ|—Ž}|S)a  Initialize a table reader allowing for common customizations. Most of the default behavior for various parameters is determined by the Reader class. Parameters ---------- Reader : `~astropy.io.ascii.BaseReader` Reader class (DEPRECATED). Default is :class:`Basic`. Inputter : `~astropy.io.ascii.BaseInputter` Inputter class Outputter : `~astropy.io.ascii.BaseOutputter` Outputter class delimiter : str Column delimiter string comment : str Regular expression defining a comment line in table quotechar : str One-character string to quote fields containing special characters header_start : int Line index for the header line not counting comment or blank lines. A line with only whitespace is considered blank. data_start : int Line index for the start of data not counting comment or blank lines. A line with only whitespace is considered blank. data_end : int Line index for the end of data not counting comment or blank lines. This value can be negative to count from the end. converters : dict Dictionary of converters. data_Splitter : `~astropy.io.ascii.BaseSplitter` Splitter class to split data columns. header_Splitter : `~astropy.io.ascii.BaseSplitter` Splitter class to split header columns. names : list List of names corresponding to each data column. include_names : list, optional List of names to include in output. exclude_names : list List of names to exclude from output (applied after ``include_names``). fill_values : dict Specification of fill values for bad or missing table values. fill_include_names : list List of names to include in fill_values. fill_exclude_names : list List of names to exclude from fill_values (applied after ``fill_include_names``). Returns ------- reader : `~astropy.io.ascii.BaseReader` subclass ASCII format reader instance NÚenableÚforce)ÚInputterÚ Outputter)Ú_get_fast_reader_dictrÚ FastBasicrÚBasicrZ _get_reader)ÚReaderr9r:ÚkwargsÚ fast_readerÚreaderr!r!r"Ú get_readerqs6 rBcCsV|dk r|dk rtd |¡ƒ‚|dk rR|tjkrYn0Xt d|¡sX|tj}|d snt |ƒ|d <t||||ƒ} | dkrŠd}|sÆ|dkrªtf|Ž} | j}|dr˜d |¡tjkr˜t |¡} tjd |¡| d <tf| Ž} y(|   |¡} t t | ¡| jddœ¡Wn~tjtjtfk r”} zT|ddkrVt d | j| ¡¡‚tf|Ž} |   |¡} t t |¡| jddœ¡Wdd} ~ XYnXn.tf|Ž} |   |¡} t t |¡| jddœ¡| S)aC Read the input ``table`` and return the table. Most of the default behavior for various parameters is determined by the Reader class. See also: - http://docs.astropy.org/en/stable/io/ascii/ - http://docs.astropy.org/en/stable/io/ascii/read.html Parameters ---------- table : str, file-like, list, pathlib.Path object Input table as a file name, file-like object, list of strings, single newline-separated string or pathlib.Path object . guess : bool Try to guess the table format. Defaults to None. format : str, `~astropy.io.ascii.BaseReader` Input table format Inputter : `~astropy.io.ascii.BaseInputter` Inputter class Outputter : `~astropy.io.ascii.BaseOutputter` Outputter class delimiter : str Column delimiter string comment : str Regular expression defining a comment line in table quotechar : str One-character string to quote fields containing special characters header_start : int Line index for the header line not counting comment or blank lines. A line with only whitespace is considered blank. data_start : int Line index for the start of data not counting comment or blank lines. A line with only whitespace is considered blank. data_end : int Line index for the end of data not counting comment or blank lines. This value can be negative to count from the end. converters : dict Dictionary of converters data_Splitter : `~astropy.io.ascii.BaseSplitter` Splitter class to split data columns header_Splitter : `~astropy.io.ascii.BaseSplitter` Splitter class to split header columns names : list List of names corresponding to each data column include_names : list List of names to include in output. exclude_names : list List of names to exclude from output (applied after ``include_names``) fill_values : dict specification of fill values for bad or missing table values fill_include_names : list List of names to include in fill_values. fill_exclude_names : list List of names to exclude from fill_values (applied after ``fill_include_names``) fast_reader : bool or dict Whether to use the C engine, can also be a dict with options which defaults to `False`; parameters for options dict: use_fast_converter: bool enable faster but slightly imprecise floating point conversion method parallel: bool or int multiprocessing conversion using ``cpu_count()`` or ``'number'`` processes exponent_style: str One-character string defining the exponent or ``'Fortran'`` to auto-detect Fortran-style scientific notation like ``'3.14159D+00'`` (``'E'``, ``'D'``, ``'Q'``), all case-insensitive; default ``'E'``, all other imply ``use_fast_converter`` chunk_size : int If supplied with a value > 0 then read the table in chunks of approximately ``chunk_size`` bytes. Default is reading table in one pass. chunk_generator : bool If True and ``chunk_size > 0`` then return an iterator that returns a table for each chunk. The default is to return a single stacked table for all the chunks. Reader : `~astropy.io.ascii.BaseReader` Reader class (DEPRECATED) encoding: str Allow to specify encoding to read the file (default= ``None``). Returns ------- dat : `~astropy.table.Table` OR Output table Nr@r7Ú chunk_sizeZ fill_values)ÚÚ0r:Frr>Ú guess_htmlZreadmeÚencoding)rOz[\r\n]zfast_{0}z&Success with fast reader (no guessing))r?r>Ústatusr8zfast reader {} exception: {}z>Success with slow reader after failing with fast (no guessing)z1Success with specified Reader class (no guessing))Ú _read_tracerFrGr;rHÚ_read_in_chunksrEÚ _format_namer4r3rÚreadrCr+rrr(r)Ú_guessrBrrÚ FAST_CLASSESÚappendÚ __class__ÚParameterErrorrÚ CParserErrorÚUnicodeEncodeErrorÚInconsistentTableError)r r5r?r@rZ new_kwargsr>rOÚfileobjÚdatrAÚ fast_kwargsZfast_reader_rdrÚerrr!r!r"rTÌsˆX                           rTc sÎg}t|ƒ}|drV|dk rVd |¡tjkrVt |¡}tjd |¡|d<|g|}nd}g}| d¡}x |D]}|ddkr¼|dtj ¡kr¼t  t |¡|dj dd d ¡d œ¡qp|dd kr|dtj ¡krt  t |¡|dj d d d ¡d œ¡qpd } xN|  ¡D]B\} } | |kr6t | ¡|| <n| || kr||krd} PqW| sbqp||krp|  |¡qpWt |ƒdkrŠdStj ttttjtjtjf} x"|D]Ö}t ¡} y^d|krÌd |d<tf|Ž}d |_| |¡}t  t |¡|j dd t ¡| d¡d œ¡|S| k r~}zHt  t |¡d |j jt|ƒ¡d t ¡| d¡dœ¡|  |¡Wdd}~XYnXq¬Wy2tf|Ž}| |¡}t  t |¡|j ddœ¡|S| k rÈ}zðt  t |¡d |j jt|ƒ¡dœ¡|  |¡dg}x„|D]|‰tdd„tˆƒDƒƒ}tˆ dtj¡ƒ}dt  d|¡ !d¡g}‡fdd„|Dƒ}| "dd„|Dƒ¡|  d #|¡¡qWd d!d"d#d$d%d&d'd!g }| "|¡t  d( #|¡¡‚Wdd}~XYnXdS))a1 Try to read the table using various sets of keyword args. Start with the standard guess list and filter to make it unique and consistent with user-supplied read keyword args. Finally, if none of those work then try the original user-supplied keyword args. Parameters ---------- table : str, file-like, list Input table as a file name, file-like object, list of strings, or single newline-separated string. read_kwargs : dict Keyword arguments from user to be supplied to reader format : str Table format fast_reader : dict Options for the C engine fast reader. See read() function for details. Returns ------- dat : `~astropy.table.Table` or None Output table or None if only one guess format was available r7Nzfast_{0}r>r@Fz/Disabled: reader only available in fast versionz {0:.3f} msg)r?r>rPÚdtr8z-Disabled: no fast version of reader availableTrZ strict_nameszSuccess (guessing)ièz{0}: {1})r?rPrazrP)r?rPzC ERROR: Unable to guess table format with the guesses listed below:cSsg|]}|dkr|‘qS))r>r:r!)rÚxr!r!r"ú 1sz_guess..zReader:z \.(\w+)'>c3s|]}|ˆ|fVqdS)Nr!)rÚkey)r?r!r"r#5sz_guess..cSsg|]\}}d ||¡‘qS)z{}: {!r})r)rrdÚvalr!r!r"rc6sú rLzH************************************************************************zH** ERROR: Unable to guess table format with the guesses listed above. **zH** **zH** To figure out why the table did not read, use guess=False and **zH** fast_reader=False, along with any appropriate arguments to read(). **zH** In particular specify the format and any known attributes like the **zH** delimiter. **Ú )$Ú_get_guess_kwargs_listrrrVrFrGrHÚvaluesrQrWrXÚitemsr'r\rCÚ TypeErrorÚAttributeErrorZOptionalTableImportErrorrYrrZÚtimerBZguessingrTÚ__name__r%rDÚreprrr=rrÚgroupÚextendr*)r Ú read_kwargsrr@Z failed_kwargsZfull_list_guessr_Zfiltered_guess_kwargsZ guess_kwargsZguess_kwargs_okrdreZguess_exception_classesZt0rAr^r`ÚlinesZ sorted_keysZ reader_reprZ keys_valsZ kwargs_sortedÚmsgr!)r?r"rU¢s¶               "         rUcCsäg}| dd¡r"| ttjd¡| ttjd¡xTtjt j t j t jt jt jt jt jtjtjtjtjtjtjfD]}| t|d¡qrWxTt jt jt j t jt j t j!fD]4}x.dD]&}x dD]}| t|||d¡q¼Wq²Wq¨W|S)aQ Get the full list of reader keyword argument dicts that are the basis for the format guessing process. The returned full list will then be: - Filtered to be consistent with user-supplied kwargs - Cleaned to have only unique entries - Used one by one to try reading the input table Note that the order of the guess list has been tuned over years of usage. Maintainers need to be very careful about any adjustments as the reasoning may not be immediately evident in all cases. This list can (and usually does) include duplicates. This is a result of the order tuning, but these duplicates get removed later. Parameters ---------- read_kwargs : dict User-supplied read keyword args Returns ------- guess_kwargs_list : list List of read format keyword arg dicts rNN)r>)ú|ú,rfz\s)ú"ú')r>Ú delimiterÚ quotechar)"ÚpoprWrIr ZHTMLrZEcsvrZFixedWidthTwoLiner ZRSTrr<rr=ZFastRdbZRdbZFastTabZTabrZCdsrZDaophotr Z SExtractorr ZIpacr ZLatexZAASTexÚFastCommentedHeaderÚCommentedHeaderZ FastNoHeaderZNoHeader)rrZguess_kwargs_listrAr>ryrzr!r!r"rhFs&      rhc sP|d}| d¡}| dd¡}d|d<t||f|Ž}|r<|St|ƒ}|j}dd„| ¡Dƒ‰d}xÀ|D]¸} || jO}x¨| j ¡D]š\} } ˆ| | j} } | jj |krÈ| jj |krÈt j |   ¡| jj d } n.| jj |krö| jj |kröt j |   ¡| jj d } |rt j jnt j}|| | gƒˆ| <q€WqfW‡fd d „|jDƒ‰|jˆ|j|jdd }|S) z† For fast_reader read the ``table`` in chunks and vstack to create a single table, OR return a generator of chunk tables. r@rKÚchunk_generatorFZparallelcSsi|]}|j ¡|j“qSr!)ÚdatarFÚname)rÚcolr!r!r"ú –sz#_read_in_chunks..)ÚSÚU)Údtypecsg|] }ˆ|‘qSr!r!)rr€)Úout_colsr!r"rc¯sz#_read_in_chunks..)ÚnamesÚmetarF)r{Ú_read_in_chunks_generatorÚnextÚmaskedÚitercolsÚcolumnsrjrr…ZkindÚnpZarrayÚtolistZmaÚ concatenateÚcolnamesrXrˆ)r r?r@rKr~Z tbl_chunksZtbl0r‹Z str_kindsÚtblr€rZcol1Zcol2rÚoutr!)r†r"rR„s2    rRc ksTtjddd„ƒ}t|tƒr8d|ks*d|kr8t|ƒ}|}nt|dƒrRt|dƒrR|}nt}d|d d <d }d }d}||| d ¡d Ê}xÂ| |¡} t | ƒ|k} | r¬t   d| ¡s¬Px6t t | ƒdddƒD]} | sÖ| | dkrÂPqÂWt dƒ‚||| d| d…} | | dd…}t| fddi|—Ž} |r6| j d¡}d}| V| r„Pq„WWdQRXdS)zk For fast_reader read the ``table`` in chunks and return a generator of tables for each chunk. Ncss |VdS)zoStub for get_readable_fileobj, which does not seem to work in Py3 for input File-like object, see #6460Nr!)r]rOr!r!r"Úpassthrough_fileobj¼sz6_read_in_chunks_generator..passthrough_fileobjrgú rTÚseekTr@Zreturn_header_charsrLrO)rOz\Sréÿÿÿÿz1no newline found in chunk (chunk_size too small?)r5FZ"__ascii_fast_reader_header_chars__)N)Ú contextlibÚcontextmanagerr$r%rÚhasattrrrHrTr'rrÚrangerCrˆr{)r rKr?r”Zfileobj_contextÚheaderZprev_chunk_charsZ first_chunkZfhÚchunkZ final_chunkÚidxZcomplete_chunkr’r!r!r"r‰¶s>     r‰)ryÚcommentrzÚformatsr‡Ú include_namesÚ exclude_namesÚstrip_whitespacecKs^|dkrtj}d|krd|d<tj||f|Ž}t|tjtjfƒrZt| dd¡t ƒsZt dƒ‚|S)aE Initialize a table writer allowing for common customizations. Most of the default behavior for various parameters is determined by the Writer class. Parameters ---------- Writer : ``Writer`` Writer class (DEPRECATED). Defaults to :class:`Basic`. delimiter : str Column delimiter string comment : str String defining a comment line in table quotechar : str One-character string to quote fields containing special characters formats : dict Dictionary of format specifiers or formatting functions strip_whitespace : bool Strip surrounding whitespace from column values. names : list List of names corresponding to each data column include_names : list List of names to include in output. exclude_names : list List of names to exclude from output (applied after ``include_names``) fast_writer : bool Whether to use the fast Cython writer. Returns ------- writer : `~astropy.io.ascii.BaseReader` subclass ASCII format writer instance Nr£TrŸrLz´for the commented_header writer you must supply a string value for the `comment` keyword. In order to disable writing table comments use `del t.meta['comments']` prior to writing.) rr=rZ _get_writerr$r}rr|rHr%rC)ÚWriterÚ fast_writerr?Úwriterr!r!r"Ú get_writers!r§)Ú overwritecKsÐt|tƒrDtj |¡rD|dkr2t d |¡t¡n|sDt d |¡ƒ‚|dkrRt j }|  d¡}t|t ƒr°|j||d}x4t| ¡| ¡ƒD]\} } t| tƒrˆ| jj| j_qˆW|}n t ||d}|dd… ¡} t | |  d¡|  d¡|  d¡¡t|  d g¡ƒt| jƒ} | r t d  | ¡t¡|jr,d }t||d ƒ}tf||d œ|—Ž} | jtjkrj|  ||¡dS|  |¡}tj  !|¡}t"|dƒs¶t#|dƒ}| |¡| tj ¡| $¡n| |¡| tj ¡dS)aWrite the input ``table`` to ``filename``. Most of the default behavior for various parameters is determined by the Writer class. See also: - http://docs.astropy.org/en/stable/io/ascii/ - http://docs.astropy.org/en/stable/io/ascii/write.html Parameters ---------- table : `~astropy.io.ascii.BaseReader`, array_like, str, file_like, list Input table as a Reader object, Numpy struct array, file name, file-like object, list of strings, or single newline-separated string. output : str, file_like Output [filename, file-like object]. Defaults to``sys.stdout``. format : str Output table format. Defaults to 'basic'. delimiter : str Column delimiter string comment : str String defining a comment line in table quotechar : str One-character string to quote fields containing special characters formats : dict Dictionary of format specifiers or formatting functions strip_whitespace : bool Strip surrounding whitespace from column values. names : list List of names corresponding to each data column include_names : list List of names to include in output. exclude_names : list List of names to exclude from output (applied after ``include_names``) fast_writer : bool Whether to use the fast Cython writer. overwrite : bool If ``overwrite=None`` (default) and the file exists, then a warning will be issued. In a future release this will instead generate an exception. If ``overwrite=False`` and the file exists, then an exception is raised. This parameter is ignored when the ``output`` arg is not a string (e.g., a file object). Writer : ``Writer`` Writer class (DEPRECATED). Nzx{} already exists. Automatically overwriting ASCII files is deprecated. Use the argument 'overwrite=True' in the future.z{} already existsr‡)r‡rr¡r¢r zKThe keys {} specified in the formats argument does not match a column name.Fr¤)r¤r¥ÚwriteÚw)%r$r%r(r-ÚlexistsÚwarningsÚwarnrrÚOSErrorÚsysÚstdoutrHrrXÚziprŒrÚinfoZserialize_methodrFrZ_apply_include_exclude_namesÚsetr‘rZhas_mixin_columnsrEr§rSrVr©r)r*ršÚopenÚclose)r Úoutputrr¤r¥r¨r?r‡Znew_tblZnew_colrZtable0Zdiff_format_with_namesr¦rsZoutstrr!r!r"r©7sT0                  r©cCs t t¡S)a¨ Return a traceback of the attempted read formats for the last call to `~astropy.io.ascii.read` where guessing was enabled. This is primarily for debugging. The return value is a list of dicts, where each dict includes the keyword args ``kwargs`` used in the read call and the returned ``status``. Returns ------- trace : list of dicts Ordered list of format guesses and status )rFrGrQr!r!r!r"Úget_read_trace«sr·)r)NNN)N)NT)NNNT)6Ú__doc__rr(r¯rFrmr¬r˜ÚiorZnumpyrŽrLrrrrrr r r r r rrrZ astropy.tablerrrZastropy.utils.datarZastropy.utils.exceptionsrrrQZyamlZHAS_YAMLÚ ImportErrorr4r3r6rBrEr;rTrUrhrRr‰Zextra_writer_parsr§r©r·r!r!r!r"Ú sb                 . B  W%>2G 6s