B T\3@sddlmZmZmZddlZddlmZddlZddlZddl Z ddl m Z ddl Z ddlZddlmZmZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z ddl!m"Z"ddlm#Z#m$Z$ddlm%Z%m&Z&dZ'ddZ(ddZ)dIddZ*ddZ+dJddZ,ddZ-dd Z.d!d"Z/d#d$Z0d%d&Z1dKd(d)Z2d*d+Z3dLd,d-Z4d.d/Z5d0d1Z6d2d3Z7d4d5d6d7d8d9d:d;hZ8d5d8d9hZ9dMdd?Z;iZdPdGdHZ?ere?j@ej?_@dS)Q)absolute_importdivisionprint_functionN) OrderedDict) LooseVersion) DataFrameSeries)clear_known_categoriesstrip_unknown_categoriesUNKNOWN_CATEGORIES)compress)tokenize)PY3 string_types)delayed)get_fs_token_paths)infer_storage_options)import_requirednatural_sort_key)_get_pyarrow_dtypes_meta_from_dtypes) read_parquet to_parquetc s|d}td}dd|dD}g}x0|D](\}}|rH||rHd}|||fq.Wdd|D}|dd dig}d d|D}|st|}t|fd d|D} nd d|D} t|} || | |fS) aPGet the set of names from the pandas metadata section Parameters ---------- pandas_metadata : dict Should conform to the pandas parquet metadata spec Returns ------- index_names : list List of strings indicating the actual index names column_names : list List of strings indicating the actual column names storage_name_mapping : dict Pairs of storage names (e.g. the field names for PyArrow) and actual names. The storage and field names will differ for index names for certain writers (pyarrow > 0.8). column_indexes_names : list The names for ``df.columns.name`` or ``df.columns.names`` for a MultiIndex in the columns Notes ----- This should support metadata written by at least * fastparquet>=0.1.3 * pyarrow>=0.7.0 Z index_columnsz__index_level_\d+__cSs$g|]}|d|d|dfqS)Z field_namename)get).0xr 8lib/python3.7/site-packages/dask/dataframe/io/parquet.py >sz*_parse_pandas_metadata..columnsNcSsg|]\}}||kr|qSr r )r storage_namerr r r!r"IsZcolumn_indexesrcSsg|] }|dqS)rr )rrr r r!r"Pscsg|]\}}|kr|qSr r )rr$r)index_storage_names2r r!r"\scSsg|]\}}||kr|qSr r )rr$rr r r!r"bs)recompilematchappendrlistsetdict) pandas_metadataZindex_storage_namesZindex_name_xprZpairsZpairs2r$Z real_name index_namescolumn_index_names column_namesstorage_name_mappingr )r%r!_parse_pandas_metadatas(    r2cs|dk }|dk }t}|dkr&t|}nt|tr<|g}t}nt|}|dkrR|}n0|dkrhg}||}nt|trz|g}nt|}|r|s|fdd|DnP|r|s|fdd|Dn0|r|r||trtdn|||fS)amNormalize user and file-provided column and index names Parameters ---------- user_columns : None, str or list of str data_columns : list of str user_index : None, str, or list of str data_index : list of str Returns ------- column_names : list of str index_names : list of str out_type : {pd.Series, pd.DataFrame} NFcsg|]}|kr|qSr r )rr)r.r r!r"sz,_normalize_index_columns..csg|]}|kr|qSr r )rr)r0r r!r"sz3Specified index and column names must not intersect)rr* isinstancerr r+ intersection ValueError)Z user_columnsZ data_columnsZ user_indexZ data_indexZspecified_columnsZspecified_indexout_typer )r0r.r!_normalize_index_columnsjs>     r7cs\ddlt|jjr|nt|dkr\|dk rFj|jjdqt||||dSnTy&j|djdjjdWn,tk rj|djjdYnXt j j ddkr|dkrt d t||\}}  fd d jD} d t|| fd dt| D} | shdf|i} d} | || Sr |dk r |jj} yj} Wn"tk rj} YnX| | kr| | } | d| ddg} n*|dkrtdj| ddt| d} n"|dkrtddt| d} t| dtjrNdd | D} | || S)NrrF) open_withsep)r# categoriesindex _metadataTzzinfer_divisions=True is not supported by the fastparquet engine for datasets that do not contain a global '_metadata' filecs0g|](}j|jsj|s|qSr )apiZfilter_out_statsschemaZfilter_out_cats)rrg) fastparquetfilterspfr r!r"sz%_read_fastparquet..z read-parquet-csFi|]>\}}t||tkjjjjf |fqSr )_read_parquet_row_groupZrow_group_filenamer r?catsdtypes file_scheme)rir@) all_columnsr:fsr.rr6rCr1r r! sz%_read_fastparquet..)NNminmaxzlUnable to infer divisions for index of '{index_name}' because it is not known to be sorted across partitions) index_name)NzDUnable to infer divisions for because no index column was discoveredcSsg|]}t|qSr )pd Timestamp)rdr r r!r"s)rAr3r> ParquetFilelenopenr9_read_fp_multifile ExceptionospathsplitfnNotImplementedError_pf_validation row_groupsr enumerater;rsorted_partitioned_columns TypeErrorr5formatnpZ datetime64)rJfs_tokenpathsr#rBr:r;infer_divisionsmetarNrgsdsk divisionsminmaxr ) rIr:rArBrJr.rr6rCr1r!_read_fastparquets`          rkcsddlm}||j|||jt|jp(g|t|trDt|}|jjr`dd|jjD}ng}t |dkr| }t|ts|g}|jt|jddDn.cSsi|] }||qSr r )rkr r r!rKsz"_pf_validation..rz/File has multiple entries for 'pandas' metadataNc3s|]}|kr|VqdS)Nr )rr)r0r r! 6sz!_pf_validation..csi|]\}}|||qSr )r)rrpv)r1r r!rK9sz&Cannot read DataFrame with MultiIndex.)r:)r;)fastparquet.utilrlr#r*rEr3tuplefmdZkey_value_metadatarSZ _get_indexr2jsonloadsextendr5r7r:rZ_dtypesitemsrrOr Z Categoricalr r;catZset_categoriesrAssertionError)rCr#r;r:rBrlZ pandas_mdr.r/r6rIrFrfrzZcatcolr )r0r1r!r\sf                 r\c sddlm}ddlm}m}m||\} fdd|D} || } ||djd| _t| | _ t ||g\} } } dt || f dd t | D}d t |d }|| |S) zBRead dataset with fastparquet by assuming metadata from first filer)rR) analyse_pathsget_file_scheme join_pathcsg|] }|qSr r )rp)r~r r!r"\sz&_read_fp_multifile..)r8z read-parquet-c s8i|]0\}}t|tkjjf |fqSr )_read_pf_simpler rErG)rrHrX) rIbaser:rJr.rr6rCr1r r!rKfsz&_read_fp_multifile..)Nr)rArRrsr|r}r~rTrG_paths_to_catsrEr\rr^rS)rJrcrdr#r:r;rRr|r}ZfnsZ parsed_pathsschemerf_rNrhrir ) rIrr:rJr.r~rr6rCr1r!rUVs      rUc sddlm} | ||jd} ||dd} x$| jD]} x| jD] }| |_q@Wq4W|| _|| _ || _ | j ||d}|j j dkrrdd|j _nrfdd D|j _fd d |D|_|r||jdS|Sd S) z9Read dataset with fastparquet using ParquetFile machineryr)rR)r8/)r;rcsg|]}||qSr )r)rr)r1r r!r"sz#_read_pf_simple..cs$g|]}|pgkr||qSr )r)rcol)r.r1r r!r"sN)rArRrTreplacelstripr]r# file_pathrGrErZ to_pandasr;nlevelsrrnames)rJrXrr.rI is_seriesr:rErr1rRrCrelpathr@chdfr )r.r1r!ros,         rcCsddlm}m}m}t}t}x|D]}|d}|dkr||} x| D]4\} } || t|| || t| qJWq&xVt | dddD]<\} } d| } || t|| || t| qWq&Wx| D]\} } || }t | t |kr\t}x*|| D]}|||t|qWdd | D}td ||| }t |d krddl}d d | D}|d |qWdd| DS)z2Extract out fields and labels from directory namesr) ex_from_sep val_to_num groupby_typesrhiveNr=zdir%icSs&g|]}t|dkr|D]}|qqS)r)rS)rrpcr r r!r"sz"_paths_to_cats..z)Partition names map to the same value: %srcSsg|] }|dqS)rr )rrr r r!r"sz.)rsrrrrfindall setdefaultr+addr^rYryrSvaluesr5warningswarn)rdrrrrrEZraw_catsrXs partitionsrnvalrHrrrawZconflicts_by_valueZraw_valZ conflictsZ vals_by_typerZexamplesr r r!rs>      rc shddlm} ddlm} ddDtttfsBgd}r^\kr^gfddD| fd d|D}| ||j d }| |_ x2|j D](}x"|j D]}| |d d |_qWqW||_|j|d }|jjdkrr6|j_nr6fddD|j_fddD|_ |r`||j dS|SdS)z9Read a single file with fastparquet, to be used in a taskr)rR)rcSsi|]\}}||qSr r )rrprrr r r!rKsz&_read_parquet_file..Tcsg|]}||qSr )r)rr)name_storage_mappingr r!r"sz&_read_parquet_file..cs g|]\}}|kr||fqSr r )rrprr)r#r r!r"s)r8rr)r#r;r:rcsg|]}||qSr )r)rr)r1r r!r"scs g|]}|kr||qSr )r)rr)r;r1r r!r"sN)fastparquet.apirR collectionsrryr3rtr*rrTrGr]r#rrrrZrr;rrr)rJrrZr;r#seriesr:csdtrr1argsrRrrCr@rrr )r#r;rr1r!_read_parquet_files>          rc  sFddlm} ddlm}ddlm}ddDttt fsNgd}rj\krjgfdd D |fd d |D}| |j ||| \}}|||||||j || d |j jd krr |j _nrfd d D|j _fdd D|_|r>||jdS|SdS)Nr) _pre_allocate)read_row_group_file)rcSsi|]\}}||qSr r )rrprrr r r!rKsz+_read_parquet_row_group..Tcsg|]}||qSr )r)rr)rr r!r"sz+_read_parquet_row_group..cs g|]\}}|kr||fqSr r )rrprr)r#r r!r"s)rTZassignrrcsg|]}||qSr )r)rr)r1r r!r"scs g|]}|kr||qSr )r)rr)r;r1r r!r"s)rrZfastparquet.corerrrryr3rtr*rnum_rowsrTr;rrrr#)rJrZr;r#r@rr:r?rrrr1rrrrrZviewsr )r#r;rr1r!rDs8          rDc Csddlm}m}ddl} t|}t|s0d} n|r~t| jdkr^||||||||j|j } q|||||||j ||j|j } nBt||_ ||j ||gd} || ||j ||d} WdQRX| S)Nr)partition_on_columnsmake_part_filez0.1.4wb) compressionru)Zfastparquet.writerrrrAcopyrSr __version__rTmkdirsr9rjoinr?) rrJrXfilenamerur partition_onrrrArgfilr r r!_write_partition_fastparquet s      rFc  s`ddl} j} | dd} | dksBt| trJd| krJtd|j} |dksf|dkr||j r|| }|j dg}nd}g}|ry| j j j| d}Wnttfk rd}YnX|r|jd krtd nt|j t|j tks tt|jkr$td |j t|j n\t|jj|j ||j jkrrtd t|jt|jAn||j }|j| jj|s| j |}||dd d}| d|krtd|| dn"| jj!|j"f| |d| dfddt#|j$D}t%t&ddfddt'||(D}t%t)||| S)Nrobject_encodingutf8ZinferzM"infer" not allowed as object encoding, because this required data in memory.T)r8r9F)remptyZflatz?Requested file scheme is hive, but existing file scheme is not.z5Appended columns not the same. Previous: {} | New: {}zAppended dtypes differ. {}rMr=zMAppended divisions overlapping with the previous ones. Previous: {} | New: {})r index_colsZignore_columnscsg|]}d|qS)zpart.%i.parquetr )rrH)i_offsetr r!r"dsz&_write_fastparquet..)purec s$g|]\}}||qSr r )rrpart)rrurJrrXwriter r!r"hs)*rArr9popr3r,rr5riknown_divisions reset_indexr#r>rRrTIOErrorrGr+rErar*rOr rFZlocanyryZ iteritemsruwriterZ find_max_partr]r_Z make_metadataZ_metarangeZ npartitionsrrzip to_delayed_write_metadata)rrJrcrX write_indexr)ignore_divisionsrrkwargsrAr9rrirrCrjZold_end filenameswritesr )rrurJrrrXrr!_write_fastparquet(sb     2&     rc Csddl}t|}xft||D]X\}}|dk rt|trTx<|D]} |j| q>Wqx|jD] } || _q\W|j|qW| |dg}|j j |||j dd| |dg}|j j |||j ddS)zc Write Parquet metadata after writing all row groups See Also -------- to_parquet rNr<F)r8Z no_row_groups_common_metadata)r8) rArrr3r*r]r)r#rrrZwrite_common_metadatarT) rrrurXrJr9rArZr@rchunkr r r!rns     rcsddlm}ddlmttr*gndkr8gntt|trRt|}j|||dj dk rddj j D} ng} j } | j dk od| j k} | rt| j dd} t| \} ng| jd d Ddg} fd d| D7t||\}g}fd d }x0jD]&}||}|jdkr<||q.spandasrcSsi|] }||qSr r )rrpr r r!rKsz!_read_pyarrow..csg|]}|kr|qSr r )rr)r0r r!r"scsj|ddS)Nrb)mode)rRrT)rZ)rJpqr r!sz_read_pyarrow..rcSs t|jS)N)rrX)piecer r r!rs)rncs g|]\}}d|kr|qS)rr )rr$r)r.r r!r"scsi|]\}}|||qSr )r)rrprr)r1r r!rKs)Zcolsz read-parquet-c s0i|](\}}t|tkjf|fqSr )_read_pyarrow_parquet_piecer r)rrHr)r:r0datasetrJr.r6 task_namer r!rKs )% bytes.corerpyarrow.parquetparquetr3rr*rtZParquetDatasetrZpartition_namesr?Zto_arrow_schemaZmetadatarvrwdecoder2rr7pieces get_metadataZnum_row_groupsr)rSsortedry_get_pyarrow_divisionsrrr r r#r{rr^r )rJrcrdr#rBr:r;rerrr?Zhas_pandas_metadatar-r/rIZnon_empty_piecesZ_openrrCZdivisions_namesdivisions_namerirFrfZ task_planr ) r:r0rrJr.r6rr1rr! _read_pyarrowsx                  rcCsVtdtdtddd}y||}Wn$tk rLtdj|dYnX||S)a Convert an input time in the specified units to nanoseconds Parameters ---------- val: int Input time value unit : str Time units of `val`. One of 's', 'ms', 'us', 'ns' Returns ------- int Time val in nanoseconds geAg.Ag@@r)rZmsusnszUnsupported time unit '{unit}')unit)intrKeyErrorr5ra)rrZfactorsZfactorr r r!_to_nss rc sddl}ddlm}|dkr2|jtdkr2td|dkrj|dk oN||dk}|dkrn|dkrntdnd}|r|rg}d}x|D]} | |j } | dfdd t j D} y| |} Wntk rd}PYnX| } | j}|jr,|dks||jkr,||j|jf|j}qd}PqW|rd d |D|d d g}||}|j|jr|jjfd d |D}dd |D}|j|krdfdd |D}n*|dkrtdj|ddt|d }n|rdt|d }nd}|S)a{ Compute DataFrame divisions from a list of pyarrow dataset pieces Parameters ---------- pa_pieces : list[pyarrow.parquet.ParquetDatasetPiece] List of dataset pieces. Each piece corresponds to a single partition in the eventual dask DataFrame divisions_name : str|None The name of the column to compute divisions for pa_schema : pyarrow.lib.Schema The pyarrow schema for the dataset infer_divisions : bool or None If True divisions must be inferred (otherwise an exception is raised). If False or None divisions are not inferred Returns ------- list rNTz0.9.0z-infer_divisions=True requires pyarrow >=0.9.0FzDUnable to infer divisions for because no index column was discoveredcsg|]}|jqSr )columnZpath_in_schema)rrH)r@r r!r"Msz*_get_pyarrow_divisions..cSsg|] \}}|qSr r )rZmnZmxr r r!r"asr=rcsg|]}t|qSr )r)rrQ) time_unitr r!r"gscSsg|]}t|qSr )rOrP)rrr r r!r"iszutf-8csg|]}|qSr )rstrip)rrQ)encodingr r!r"oszlUnable to infer divisions for index of '{index_name}' because it is not known to be sorted across partitions)rN)N)NN)pyarrowrrrrr[Zget_field_indexr5rrRZ row_grouprZ num_columnsr;rZ statisticsZ has_min_maxrLr)rMZ field_by_nametypesZ is_timestamptyperstringrarS)Z pa_piecesrZ pa_schemareparZdivisions_name_in_schemaZmin_maxsZlast_maxrrCZrg_pathsZdivisions_col_indexZcol_metaZstatsriZ index_fieldZ divisions_nsr )rr@rr!rsb          rc CsDddl}|j|jdd}|j|||d|d} WdQRX|jtdkrr| } xJ|D]} | | d| | <qVWn*|jtdkr| j|d } n| j|d d } t| j t j  } | s|r| |} nd| r$| j j |kr$| jd d } |r| |} tt| j|} | r| j| d d} | j|d d} |r8| | jdS| |SdS)Nrr)rT)r#rZuse_pandas_metadatafilez0.9.0categoryz0.11.0)r:F)r:Zdate_as_object)dropr)Zaxis)r#r)rrTrXreadrrrZastyper3r;rOZ RangeIndexZ set_indexrrr*r+r# differencerZreindex)rJrr#rrrr:rftablerrzZ has_indexrr r r!rs6     rZrow_group_sizeversionZuse_dictionaryrZuse_deprecated_int96_timestampsZcoerce_timestampsZflavorZ chunk_sizec  s|r td|rtdttrHddttt} t| dkrZ|jrZdj dgt t dd j d gd <fd d t |D} t | S) Nz/`append` not implemented for `engine='pyarrow'`z9`ignore_divisions` not implemented for `engine='pyarrow'`zUnexpected keyword arguments: z%rTzpart.%i.parquetF)rr metadata_pathc s2g|]*\}}||f|r(nqSr r )rrHr) first_kwargsrJrrrXtemplaterrr r!r"sz"_write_pyarrow..)r[r+r_pyarrow_write_table_kwargsr*r`rrr9rr_write_partition_pyarrowrr^r) rrJrcrXrr)rrrmsgrr )rrJrrrXrrrr!_write_pyarrows$  rc Ksddl}ddlm} |jj||d} |rF| j| |f|||d|n(||d} | j| | f|WdQRX|dk r||d*} dd|D} | j| j | f| WdQRXdS)Nr)r)preserve_index)Zpartition_colsrrrcSsi|]\}}|tkr||qSr )_pyarrow_write_metadata_kwargs)rrprrr r r!rKsz,_write_partition_pyarrow..) rrZTableZ from_pandasZwrite_to_datasetrTZ write_tableryZwrite_metadatar?) rrXrJrrrrrrrtrZ kwargs_metar r r!rs  rc Cs|tkrt|S|dkrNxdD]"}yt|Stk r>YqXqWtdnx|dkrvtddttdtd<}|S|dkrtdd}t|jd krtd tt dtd<}|St d |d d S)a8Get the parquet engine backend implementation. Parameters ---------- engine : {'auto', 'fastparquet', 'pyarrow'}, default 'auto' Parquet reader library to use. Default is first installed in this list. Returns ------- A dict containing a ``'read'`` and ``'write'`` function. auto)rArz,Please install either fastparquet or pyarrowrAz`fastparquet` not installed)rrrz`pyarrow` not installedz0.8.0z!PyArrow version >= 0.8.0 requiredzUnsupported engine: "{0}".z4 Valid choices include "pyarrow" and "fastparquet".N) _ENGINES get_engine RuntimeErrorrrkrrrrrr5ra)engineZengrr r r!r s.       r r c Cs"d}yXddl} t|| jjrZ|j| jjkrFtd|j sFt d|j |dksVt dd}Wnt k rpYnX|rt d d } |j d r|j dtd  } n|j } t| d |d \} } }|}nFt |d } t|d |d \} } }t|tr t|dkr t|td}| | | ||||||dS)a Read ParquetFile into a Dask DataFrame This reads a directory of Parquet data into a Dask.dataframe, one file per partition. It selects the index among the sorted columns if any exist. Parameters ---------- path : string, list or fastparquet.ParquetFile Source directory for data, or path(s) to individual parquet files. Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. Alternatively, also accepts a previously opened fastparquet.ParquetFile() columns : string, list or None (default) Field name(s) to read in as columns in the output. By default all non-index fields will be read (as determined by the pandas parquet metadata, if present). Provide a single field name instead of a list to read in the data as a Series. filters : list List of filters to apply, like ``[('x', '>', 0), ...]``. This implements row-group (partition) -level filtering only, i.e., to prevent the loading of some chunks of the data, and only if relevant statistics have been included in the metadata. index : string, list, False or None (default) Field name(s) to use as the output frame index. By default will be inferred from the pandas parquet file metadata (if present). Use False to read all fields as columns. categories : list, dict or None For any fields listed here, if the parquet encoding is Dictionary, the column will be created with dtype category. Use only if it is guaranteed that the column is encoded as dictionary in all row-groups. If a list, assumes up to 2**16-1 labels; if a dict, specify the number of labels expected; if None, will load categories automatically for data written by dask/fastparquet, not otherwise. storage_options : dict Key/value pairs to be passed on to the file-system backend, if any. engine : {'auto', 'fastparquet', 'pyarrow'}, default 'auto' Parquet reader library to use. If only one library is installed, it will use that one; if both, it will use 'fastparquet' infer_divisions : bool or None (default). By default, divisions are inferred if the read `engine` supports doing so efficiently and the `index` of the underlying dataset is sorted across the individual parquet files. Set to ``True`` to force divisions to be inferred in all cases. Note that this may require reading metadata from each file in the dataset, which may be expensive. Set to ``False`` to never infer divisions. Examples -------- >>> df = dd.read_parquet('s3://bucket/my-parquet-data') # doctest: +SKIP See Also -------- to_parquet FrNz.*://zxParquetFile: Path must contain protocol (e.g., s3://...) when using other than the default LocalFileSystem. Path given: )r rAz['engine' should be set to 'auto' or 'fastparquet' when reading from fastparquet.ParquetFileTrArr<r)rstorage_optionsr)rn)r#rBr:r;re)rAr3r>rRrTutilZ default_openr&r(rZr{ ImportErrorr endswithrSrrrr)rXr#rBr:r;rr reZis_ParquetFilerArZurlpathrJrcrdr r r!rs<<       rdefaultTc Ks|pg}t|t|jr"td|dkr4|| d<ndtkrDd| d<t|d} t|d|d\} } }t|d}| || | |f||||d | }| r|d S|S) aStore Dask.dataframe to Parquet files Notes ----- Each partition will be written to a separate file. Parameters ---------- df : dask.dataframe.DataFrame path : string Destination directory for data. Prepend with protocol like ``s3://`` or ``hdfs://`` for remote data. engine : {'auto', 'fastparquet', 'pyarrow'}, default 'auto' Parquet library to use. If only one library is installed, it will use that one; if both, it will use 'fastparquet'. compression : string or dict, optional Either a string like ``"snappy"`` or a dictionary mapping column names to compressors like ``{"name": "gzip", "values": "snappy"}``. The default is ``"default"``, which uses the default compression for whichever engine is selected. write_index : boolean, optional Whether or not to write the index. Defaults to True *if* divisions are known. append : bool, optional If False (default), construct data-set from scratch. If True, add new row-group(s) to an existing data-set. In the latter case, the data-set must exist, and the schema must match the input data. ignore_divisions : bool, optional If False (default) raises error when previous divisions overlap with the new appended divisions. Ignored if append=False. partition_on : list, optional Construct directory-based partitioning by splitting on these fields' values. Each dask partition will result in one or more datafiles, there will be no global groupby. storage_options : dict, optional Key/value pairs to be passed on to the file-system backend, if any. compute : bool, optional If True (default) then the result is computed immediately. If False then a ``dask.delayed`` object is returned for future computation. **kwargs Extra options to be passed on to the specific backend. Examples -------- >>> df = dd.read_csv(...) # doctest: +SKIP >>> to_parquet('/path/to/output/', df, compression='snappy') # doctest: +SKIP See Also -------- read_parquet: Read parquet data to dask.dataframe z#Partitioning on non-existent columnrrZsnappyrr)rrrX)rr)rrN)r+r#r5rr rrcompute)rrXr rrr)rrrrrrrJrcroutr r r!rs$6   r)NNNNN)NNN)NFFNN)NNNNN)NFFN)N)NNNNNr N)r rNFFNNT)AZ __future__rrrr&rrrrvrWZdistutils.versionrZnumpyrbrmrOZcorerr Zutilsr r r Zbytes.compressionrrrZ compatibilityrrrrrZ bytes.utilsrrrrr__all__r2r7rkr\rUrrrrDrrrrrrrrrrrr r rr__doc__r r r r!sr       NG OU -*( D td)  . j O