\c @`sddlmZmZmZddlmZddlmZmZm Z yddl Z Wne k rqdZ nXddl ZddlZddlmZmZmZmZmZddlmZmZddlmZmZdd lmZm Z m!Z!m"Z"dd l#m#Z#dd l$m%Z%m&Z&d d l$m'Z'ddlm(Z(dde)e*ddZ+dZ,e)e*dddZ-dZ.e dk re<e de/e j0j1Z2e j3Z4e.e2e4Z5WdQXnd#Z5e5e)ddde*e*de*d Z6dZ7dZ8e8ej9ddZ9e8ej:ddZ:e8ej;ddZ;d Z<dde)dde*d!Z=e rd d"l>m?Z?e=j@e?j=_@ndS($i(tprint_functiontdivisiontabsolute_import(tBytesIO(twarntcatch_warningst simplefilterN(tis_integer_dtypetis_float_dtypetis_object_dtypetis_datetime64_any_dtypetCategoricalDtypei(t read_bytest open_files(tseekable_filestfiles(tPY2tPY3tMappingtunicode(tdelayed(t asciitablet parse_bytesi(tclear_known_categoriesi(t from_delayedc C`s't} |r5|j|j r5| j|n| j|| jd|| |} |rtt| |n|r|rt| jt|krtd| j|n|r|| _n|r#|\} }} | j |} | j it j j tjt| | | | 6} n| S(sS Convert a block of bytes to a Pandas DataFrame Parameters ---------- reader : callable ``pd.read_csv`` or ``pd.read_table``. b : bytestring The content to be parsed with ``reader`` header : bytestring An optional header to prepend to ``b`` kwargs : dict A dictionary of keyword arguments to be passed to ``reader`` dtypes : dict DTypes to assign to columns path : tuple A tuple containing path column name, path to file, and all paths. See Also -------- dask.dataframe.csv.read_pandas_from_bytes isColumns do not match(Rt startswithtrstriptwritetseekt coerce_dtypestlisttcolumnst ValueErrortindextassigntpdt Categoricalt from_codestnptfulltlen(treadertbtheadertkwargstdtypesRt write_headertenforcetpathtbiotdftcolnametpathstcode((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pytpandas_read_texts$   '  1c C`sg}g}g}x|jD]}||kr|j|||kr|j|}||}t|rt|r|j|||fqt|rt|r|j|qy||j||||jstkeycS`st|dS(Ni(tstr(tx((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pytktsAThe following columns also raised exceptions on conversion: %s R>sf Alternatively, provide `assume_missing=True` to interpret all unspecified integer columns as floats.cS`st|dS(Ni(R;(R<((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyR=vR>tColumntFoundtExpecteds dtype={%s}s , cs`s(|]\}}}d||fVqdS(s%r: '%s'N((R7tktvt_((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pys yss{table} {exceptions}Usually this is due to dask's dtype inference failing, and *may* be fixed by specifying dtypes manually by adding: {dtype_kw} to the call to `read_csv`/`read_table`.{extra}ttablet exceptionstdtype_kwtextras also t cs`s|]}d|VqdS(s- %sN((R7R8((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pys ssThe following columns{also}failed to properly parse as dates: {cols} This is usually due to an invalid value in that column. To diagnose and fix it's recommended to drop these columns from the `parse_dates` keyword, and manually convert them to dates later using `dd.to_datetime`.talsotcolss %s t-i=s=Mismatched dtypes found in `pd.read_csv`/`pd.read_table`. %sN(RR-RRtappendR R tastypet ExceptiontjointsortedRtformattNonetfilterR (R2R-t bad_dtypest bad_datesterrorsR8tactualtdesiredR9texRFRHRERGt dtype_msgRJRKtdate_msgtruletmsg((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyRHsZ #         c C`s|jj} |jddgj} g} | } t|trg| D]<} t|j| trI|j| jd k rI| ^qI} | j | } n-t|tr|jd krg} | } nx| D]} d| | sFiles already contain the column name: %s, so the path column cannot use this name. Please set `include_path_column` to a unique name.RcRsR/Rt($t__name__RSR(RftboolR RRRgRptsettmaxRtrangeRt TypeErrortdictR;RRRRtcfilestNotImplementedErrortencodeR ttupleRltcomputetsplitRRRRcRNtfloatR~( R)turlpathRRsRRRR/tassume_missingtstorage_optionstinclude_path_columnR,t reader_nametkwRbt lastskiprowtfirstrowtpath_convertertb_lineterminatortb_outtb_sampletvaluesR4R0RR+tneedtpartstnpartsRrRtR8((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyt read_pandas s        ,   $   "  $'.    ! sQ Read {file_type} files into a Dask.DataFrame This parallelizes the :func:`pandas.{reader}` function in the following ways: - It supports loading many files at once using globstrings: >>> df = dd.{reader}('myfiles.*.csv') # doctest: +SKIP - In some cases it can break up large files: >>> df = dd.{reader}('largefile.csv', blocksize=25e6) # 25MB chunks # doctest: +SKIP - It can read CSV files from external resources (e.g. S3, HDFS) by providing a URL: >>> df = dd.{reader}('s3://bucket/myfiles.*.csv') # doctest: +SKIP >>> df = dd.{reader}('hdfs:///myfiles.*.csv') # doctest: +SKIP >>> df = dd.{reader}('hdfs://namenode.example.com/myfiles.*.csv') # doctest: +SKIP Internally ``dd.{reader}`` uses :func:`pandas.{reader}` and supports many of the same keyword arguments with the same performance guarantees. See the docstring for :func:`pandas.{reader}` for more information on available keyword arguments. Parameters ---------- urlpath : string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. blocksize : str, int or None, optional Number of bytes by which to cut up larger files. Default value is computed based on available physical memory and the number of cores. If ``None``, use a single block for each file. Can be a number like 64000000 or a string like "64MB" collection : boolean, optional Return a dask.dataframe if True or list of dask.delayed objects if False sample : int, optional Number of bytes to use when determining dtypes assume_missing : bool, optional If True, all integer columns that aren't specified in ``dtype`` are assumed to contain missing values, and are converted to floats. Default is False. storage_options : dict, optional Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. include_path_column : bool or str, optional Whether or not to include the path to each particular file. If True a new column is added to the dataframe called ``path``. If str, sets new column name. Default is False. **kwargs Extra keyword arguments to forward to :func:`pandas.{reader}`. Notes ----- Dask dataframe tries to infer the ``dtype`` of each column by reading a sample from the start of the file (or of the first file if it's a glob). Usually this works fine, but if the ``dtype`` is different later in the file (or in other files) this can cause issues. For example, if all the rows in the sample had integer dtypes, but later on there was a ``NaN``, then this would error at compute time. To fix this, you have a few options: - Provide explicit dtypes for the offending columns using the ``dtype`` keyword. This is the recommended solution. - Use the ``assume_missing`` keyword to assume that all columns inferred as integers contain missing values, and convert them to floats. - Increase the size of the sample using the ``sample`` keyword. It should also be noted that this function may fail if a {file_type} file includes quoted strings that contain the line terminator. To get around this you can specify ``blocksize=None`` to not split files into multiple partitions, at the cost of reduced parallelism. c `sRttdddttdtfd }tjd|d||_||_|S(Nic `sFt|d|d|d|d|d|d|d|d|d | |  S( NRRsRRRR/RRR(R( RRRsRRRR/RRRR,(R)(s4lib/python2.7/site-packages/dask/dataframe/io/csv.pytreads  R)t file_type(tAUTO_BLOCKSIZERjRSRltREAD_DOC_TEMPLATERRt__doc__R(R)RRR((R)s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyt make_readers     tread_csvtCSVt read_tablet delimitedtread_fwfs fixed-widthcK`s#|}|j||WdQXdS(N(tto_csv(R2tfilR,tf((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyt _write_csvs c K`sMtrd} d} n d} d} |jd| } t|d|d| d| d|d|j|pci} ttd t} |j}| |d | d |g}t |d kr|rt|d >> df.to_csv('/path/to/data/export-*.csv') # doctest: +SKIP The * will be replaced by the increasing sequence 0, 1, 2, ... :: /path/to/data/export-0.csv /path/to/data/export-1.csv Use a globstring and a ``name_function=`` keyword argument. The name_function function should expect an integer and produce a string. Strings produced by name_function must preserve the order of their respective partition indices. >>> from datetime import date, timedelta >>> def name(i): ... return str(date(2015, 1, 1) + i * timedelta(days=1)) >>> name(0) '2015-01-01' >>> name(15) '2015-01-16' >>> df.to_csv('/path/to/data/export-*.csv', name_function=name) # doctest: +SKIP :: /path/to/data/export-2015-01-01.csv /path/to/data/export-2015-01-02.csv ... You can also provide an explicit list of paths:: >>> paths = ['/path/to/data/alice.csv', '/path/to/data/bob.csv', ...] # doctest: +SKIP >>> df.to_csv(paths) # doctest: +SKIP Parameters ---------- filename : string Path glob indicating the naming scheme for the output files name_function : callable, default None Function accepting an integer (partition index) and producing a string to replace the asterisk in the given filename globstring. Should preserve the lexicographic order of partitions compression : string or None String like 'gzip' or 'xz'. Must support efficient random access. Filenames with extensions corresponding to known compression algorithms (gz, bz2) will be compressed accordingly automatically sep : character, default ',' Field delimiter for the output file na_rep : string, default '' Missing data representation float_format : string, default None Format string for floating point numbers columns : sequence, optional Columns to write header : boolean or list of string, default True Write out column names. If a list of string is given it is assumed to be aliases for the column names header_first_partition_only : boolean, default False If set, only write the header row in the first output file index : boolean, default True Write row names (index) index_label : string or sequence, or False, default None Column label for index column(s) if desired. If None is given, and `header` and `index` are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R nanRep : None deprecated, use na_rep mode : str Python write mode, default 'w' encoding : string, optional A string representing the encoding to use in the output file, defaults to 'ascii' on Python 2 and 'utf-8' on Python 3. compression : string, optional a string representing the compression to use in the output file, allowed values are 'gzip', 'bz2', 'xz', only used when the first argument is a filename line_terminator : string, default '\n' The newline character or character sequence to use in the output file quoting : optional constant from csv module defaults to csv.QUOTE_MINIMAL quotechar : string (length 1), default '"' character used to quote fields doublequote : boolean, default True Control quoting of `quotechar` inside a field escapechar : string (length 1), default None character used to escape `sep` and `quotechar` when appropriate chunksize : int or None rows to write at a time tupleize_cols : boolean, default False write multi_index columns as a list of tuples (if True) or new (expanded format) if False) date_format : string, default None Format string for datetime objects decimal: string, default '.' Character recognized as decimal separator. E.g. use ',' for European data storage_options: dict Parameters passed on to the backend filesystem class. Returns ------- The names of the file written if they were computed right away If not, the delayed tasks associated to the writing of the files twbsutf-8twttencodingRtmodet name_functiontnumRaiiR+t schedulerN(RRSRgR t npartitionsRRRlt to_delayedR(textendtzipRR0(R2tfilenameRRRRRtheader_first_partition_onlyR,tdefault_encodingRRRt to_csv_chunkRyRtdR((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pyRs*x     =(t_Framei(At __future__RRRtioRtwarningsRRRtpsutilt ImportErrorRStnumpyR&tpandasR#tpandas.api.typesRRR R R tbytesR R tbytes.compressionRRRt compatibilityRRRRRtutilsRRRRRjRlR6RR~RtRuntimeWarningtvirtual_memoryttotalt TOTAL_MEMRt CPU_COUNTRRRRRRRRRtcoreRR(((s4lib/python2.7/site-packages/dask/dataframe/io/csv.pytsZ    ("+ U\