B T\o @sDddlmZmZmZddlZddlZddlmZddlZddl Z ddl m Z ddl Z ddlZddlmmZddlmZmZmZmZyddlmZWn ek rddlmZYnXyddlmZWnek rdd ZYnXd d lmZd d lm Z d dl!m"Z"m#Z#m$Z$d dl%m&Z&d dl'm(Z(d dl)m*Z*m+Z+m,Z,eej-Z.e.edkZ/e/Z0ddZ1ddZ2dZ3dZ4ddZ5e d[ddZ6dZ7dd Z8d!d"Z9d\d$d%Z:d]d&d'Z;e,d(Zej?fd^d)d*Z@e<=ejAd_d+d,ZBe<=eCd`d-d.ZDe.d/krejEejFejGfZHn ejEejFfZHe,d0ZIeI=eCd1d2ZJeI=ej?d3d4ZKeI=ejAd5d6ZLe Md#e Nd7e Od8e Pd e Qd9e Qd9e Rd9d9d:ZSd;d<ZTd=d>ZUeI=ej>dad?d@ZVdAdBZWdCdDZXdEdFZYdbdGdHZZdcdIdJZ[dddKdLZ\dMdNZ]dedOdPZ^dQdRZ_dSdTZ`dUdVZadfdWdXZbdgdYdZZcdS)h)absolute_importdivisionprint_functionN) LooseVersion)contextmanager)is_categorical_dtype is_scalar is_sparseis_period_dtype)is_datetime64tz_dtype)is_interval_dtypecCsdS)NF)dtyper r 3lib/python3.7/site-packages/dask/dataframe/utils.pysr)make_array_nonempty)is_dask_collection)PY2IteratorMapping)get_deps)get_sync) asciitable is_arraylikeDispatchz 0.24.0rc1cCsDt|d|}tr6tjtjtjtjtjtjtj tj f}nd}t ||S)Nrr ) getattr HAS_INT_NApdZ Int8DtypeZ Int16DtypeZ Int32DtypeZ Int64DtypeZ UInt8DtypeZ UInt16DtypeZ UInt32DtypeZ UInt64Dtype isinstance)trtypesr r ris_integer_na_dtype)s  r#ccst|trt|}t|s"|Vnt|}|}|j}t|rJ| }| |}|j d|dVx4t t|dD] }|j ||||dVqzW|j |ddVdS)aM Shard a DataFrame by ranges on its index Examples -------- >>> df = pd.DataFrame({'a': [0, 10, 20, 30, 40], 'b': [5, 4 ,3, 2, 1]}) >>> df a b 0 0 5 1 10 4 2 20 3 3 30 2 4 40 1 >>> shards = list(shard_df_on_index(df, [2, 4])) >>> shards[0] a b 0 0 5 1 10 4 >>> shards[1] a b 2 20 3 3 30 2 >>> shards[2] a b 4 40 1 >>> list(shard_df_on_index(df, []))[0] # empty case a b 0 0 5 1 10 4 2 20 3 3 30 2 4 40 1 Nrr) r rlistlennparray sort_indexindexrZ as_orderedZ searchsortedilocrange)df divisionsr*indicesir r rshard_df_on_index5s'    r1z?meta : pd.DataFrame, pd.Series, dict, iterable, tuple, optionalaWAn empty ``pd.DataFrame`` or ``pd.Series`` that matches the dtypes and column names of the output. This metadata is necessary for many algorithms in dask dataframe to work. For ease of use, some alternative inputs are also available. Instead of a ``DataFrame``, a ``dict`` of ``{name: dtype}`` or iterable of ``(name, dtype)`` can be provided. Instead of a series, a tuple of ``(name, dtype)`` can be used. If not provided, dask will try to infer the metadata. This may lead to unexpected results, so providing ``meta`` is recommended. For more information, see ``dask.dataframe.utils.make_meta``. c s|sfddS|d}ddd}tjt||dd}d td |}|jrd |jkrn|jd ||_nPd |d d}t d|j\}}| dd\} } d||| |d d|| |_|S)zReplace `$META` in docstring with param description. If pad keyword is provided, will pad description by that number of spaces (default is 8).cs t|fS)N)insert_meta_param_description)f)kwargsr rrsz/insert_meta_param_description..r ZpadN)Zinitial_indentZsubsequent_indentwidthz{0} {1} z$METAzParameters %s----------NzParameters\n[ ]*----------z rz{0}{1}{2} {3}{4} {5}) gettextwrapZwrap_META_DESCRIPTIONformat _META_TYPESjoin__doc__replaceresplit) argsr4r3indentZbodyZdescrZparameter_headerfirstZlastZ parametersrestr )r4rr2{s"     r2Fc csy dVWntk r}zbt\}}}dt|}d}|rJ|d7}|d7}||rdd|ndt||}t|Wdd}~XYnXdS)zReraise errors in this block to show metadata inference failure. Parameters ---------- funcname : str, optional If provided, will be added to the error message to indicate the name of the method that failed. NzMetadata inference failed{0}. zYou have supplied a custom function and Dask is unable to determine the type of output that that function returns. To resolve this please provide a meta= keyword. The docstring of the Dask function you ran should have more information. zOOriginal error is below: ------------------------ {1} Traceback: --------- {2}z in `{0}`) Exceptionsysexc_infor@ traceback format_tbr>repr ValueError)funcnameZudfeexc_type exc_value exc_tracebacktbmsgr r rraise_on_meta_errors   rXZ__UNKNOWN_CATEGORIES__cCsFt|d|}t|tjr$t|jjkSt|tjr:t|jkStddS)zwReturns whether the categories in `x` are known. Parameters ---------- x : Series or CategoricalIndex _metaz#Expected Series or CategoricalIndexN) rr rSeriesUNKNOWN_CATEGORIEScat categoriesCategoricalIndex TypeError)xr r rhas_known_categoriess      racCst|tjtjfr|}t|tjrr|jdk}|r||j}x\|D]$}t||sH||j j gddqHWn.t|tjrt |j rt|s|j j gddt|jtj rt|js|j g|_nt|tj rt|s| g}|S)zReplace any unknown categoricals with empty categoricals. Useful for preventing ``UNKNOWN_CATEGORIES`` from leaking into results. categoryT)inplace)r rrZ DataFramecopydtypesanyr*rar\set_categoriesrrr^)r`Zcat_maskcatscr r rstrip_unknown_categoriess$        rkTcCst|tjtjfr|}t|tjr|jdk}|dkrD||j}n|j|sZt dxL|D]}||j j t gddq`Wn(t|tjrt |jr|j j t gdd|rt|jtjr|j t g|_nt|tjr| t g}|S)aSet categories to be unknown. Parameters ---------- x : DataFrame, Series, Index cols : iterable, optional If x is a DataFrame, set only categoricals in these columns to unknown. By default, all categorical columns are set to unknown categoricals index : bool, optional If True and x is a Series or DataFrame, set the clear known categories in the index as well. rbNz Not all columns are categoricalsT)rc)r rrZrdrerfr*ZlocallrPr\rhr[rrr^)r`Zcolsr*maskrjr r rclear_known_categoriess$         rncCsFt|tr4|dkr4tjttg||djddStjg|||dS)Nrb)namer*r)rror*)r strrrZ Categoricalr[r+)rorr*r r r _empty_seriessrr make_metacCs|jddS)Nr)r+)r`r*r r rmake_meta_pandas srtcCs |ddS)Nrr )r`r*r r rmake_meta_indexsrucsDt|dr|jSt|r$|ddSdk r4tt|tr^tjfdd|DdSt|t rt |dkrt |d|ddSt|t t frt d d |Dstd |tjfd d|Dd d|DdSt|ds |dk r yt|}t|Stk rYnXt|r2t|Std|dS)auCreate an empty pandas object containing the desired metadata. Parameters ---------- x : dict, tuple, list, pd.Series, pd.DataFrame, pd.Index, dtype, scalar To create a DataFrame, provide a `dict` mapping of `{name: dtype}`, or an iterable of `(name, dtype)` tuples. To create a `Series`, provide a tuple of `(name, dtype)`. If a pandas object, names, dtypes, and index should match the desired output. If a dtype or scalar, a scalar of the same dtype is returned. index : pd.Index, optional Any pandas index to use in the metadata. If none provided, a `RangeIndex` will be used. Examples -------- >>> make_meta([('a', 'i8'), ('b', 'O')]) Empty DataFrame Columns: [a, b] Index: [] >>> make_meta(('a', 'f8')) Series([], Name: a, dtype: float64) >>> make_meta('i8') 1 rYNrcs i|]\}}t||d|qS))r*)rr).0rjd)r*r r 8sz$make_meta_object..)r*rrcss$|]}t|tot|dkVqdS)rN)r tupler&)rvr0r r r =sz#make_meta_object..z5Expected iterable of tuples of (name, dtype), got {0}cs i|]\}}t||d|qS))r*)rr)rvrjrw)r*r rrx@scSsg|] \}}|qSr r )rvrjrwr r r Asz$make_meta_object..)columnsr*rz*Don't know how to create metadata from {0})hasattrrYrrsr dictrrditemsryr&rrr%rlrPr>r'r_scalar_from_dtyperJr_nonempty_scalarr_)r`r*rr )r*rmake_meta_objects4     rz0.20.0 meta_nonemptycCs(t|rt|Stdt|jdS)zCreate a nonempty pandas object from the given metadata. Returns a pandas DataFrame, Series, or Index that contains two rows of fake data. z5Expected Index, Series, DataFrame, or scalar, got {0}N)rrr_r>type__name__)r`r r rmeta_nonempty_object\srcsLtjfddtjD}tj|ttjd}j|_|S)Ncs,i|]$\}}tjdd|fd|qS)N)idx)_nonempty_seriesr+)rvr0rj)rr`r rrxmsz+meta_nonempty_dataframe..)r*r|) rr* enumerater|rrdr'Zaranger&)r`dataresr )rr`rmeta_nonempty_dataframejs  rcCst|}|tjkr"tjd|jdS|tkr<|ddg|jdS|tjkrZtjddg|jdS|tjkrd}|jdkrz|dgnd}tj||d|j|j|jdS|tj krtj dd|j|jd S|tj krt dd }|jdkr||dgnd}tj ||d|j|jd S|tjkrbt|jd kr8tjt|j|jd }ntjjd d g|j|jd}tj||jdS|tjkrdd|jD}dd|jD}tj|||jdStdt|jdS)Nr)rorabz 1970-01-01z 1970-01-02)startperiodsfreqtzro)rrrroDr)orderedr$)r]rcSsg|] }t|qSr )_nonempty_index)rvlr r rr{sz#_nonempty_index..cSsg|] }ddgqS)rr )rvr0r r rr{s)levelslabelsnamesz*Don't know how to handle index of type {0})rrZ RangeIndexro_numeric_index_typesIndexZ DatetimeIndexrrZ PeriodIndexZ period_rangeZTimedeltaIndexr' timedelta64r^r&r]rqrrZ from_codes MultiIndexrrr_r>r)rtyprrrrr r rrusB             r z 1970-01-01Zfoo)rVMmSrUOcCsl|jdkr|dS|jdkr.|tddS|jtkrZt|j}|jdkrV||S|Std|dS)N)r0r3urrjr)rrzCan't handle dtype: {0})kindrcomplex_simple_fake_mappingZastyper_r>)ror r rrs     rcCsbt|tjtjtjfr|St|rJt|dr4|jn tt |}t |St d t |j dS)NrzCan't handle meta of type '{0}')r r Timestamp TimedeltaPeriodr'isscalarr}rrrr_r>r)r`rr r rrs rcCs|dkrt|j}|j}t|r.equal_dtypesz>Expected partition to be DataFrame, Series, or Index, got `%s`z,Expected partition of type `%s` but got `%s`z0.23.0Tsortaxisrcs&g|]\}}}||s|||fqSr r )rvcolrr)rr rr{3szcheck_meta..rzPartition type: `%s` %sZColumnZFoundZExpectedz]The columns in the computed data do not match the columns in the provided metadata. %s :%srIrzMetadata mismatch found%s. %sz in `%s`N)rrrrrr_rrr~rrrconcatrfZfillna itertuplesrr'Z array_equalZ nan_to_numr|rrP)r`metarQ numeric_equalerrmsgr4rfZ bad_dtypesr )rrr check_metasB   rcCsLt|}|dkr|jj}|r:|d}|d}d||}nd}d|||S)z+Summarized representation of an Index. Nrr$z , {} to {}rIz{}: {} entries{})r& __class__rr>)rronrtailZsummaryr r r index_summaryNsrcCslddlm}t|drh|dkr,|jdd}t||jrdt|jksRtt||r|j |j ksft|j j |j ksxtt|t j r|j |j j kst|rt||nt||jrPdt|jkstt|t|j t|kstt|j |r(|j |j kst|j |j f|j j |j ks(t|r8t||t|j|||jdnt||jrdt|jksztt|t|jt jstt|jt|j t|kstt|j |rt|j|jt|j j|j|rt||t|j|||jdn\t||jjrNt|s)ZdskrrrZddrWr r rrcsV       "   rc Csy`t|tjrVt|jjt|j@rBddtt|jjD|j_|j |j d}n| }Wnt t t fk rzYnX|S)NcSsg|] }d|qS)z-overlapped-index-name-%dr )rvr0r r rr{sz_maybe_sort..)Zby)r rrdrr*rr|r,r&Z sort_valuestolistr_ IndexErrorrPr))rr r r _maybe_sorts  rc Ks|rlt|t|t|drlt|drltt|jd}tt|jd}||kslt||ft|t|t |||d}t |||d}|s|j dd}|j dd}t|dr| }t|dr| }t |t jrt|}t|}tj||f|nt |t jr@t|}t|}tj||fd|i|n^t |t jr`tj||f|n>||krndSt|rt|stnt||stdS)Nr.r)rrT)Zdrop to_pandasr)assert_divisionsr}rr'Zasarrayr.rrassert_sane_keynamesrZ reset_indexrr rrdrrZassert_frame_equalrZZassert_series_equalrrZisnanZallclose) rrrrZcheck_divisionsZ check_indexr4ZatZbtr r r assert_eqsD      rcCs`t|dr|j}t|tstx,|D]$}t|tr:|d}||r$dSq$Wtdj|ddS)NrrTz/given dask graph doesn't contain label: {label})label)r}rr rrry startswithr>)rrkr r rassert_dask_graphs    rcCst|dsdSt|dsdS|js&dSdd}t|j|}x\t|ddD]H\}}t|rP|||j|kszt || |j|dksPt qPWt|dr||d|jdkst ||d |jdkst dS)Nr.r*cSs8t|tjr|Sy |jdStk r2|jSXdS)Nr)r rrr*Zget_level_valuesAttributeError)r`r r rr*s   zassert_divisions..indexr$r) r}Zknown_divisionsrrZ __dask_keys__rr&minr.rmax)ddfr*Zresultsr0r-r r rrs  " rcCst|dsdSxr|jD]d}xt|tr4|d}q Wt|ttfsHtt|dksXtd|ksdtt s| dd stqWdS)Nrrdr5r) r}rkeysr ryrpbytesrr&rrD isidentifier)rrr r rrs    rcCs>ddddh}|r|dt|tjrtxJtj|jj|jgddD]*\}}}|j|krb|j|ksD||ksDt qDWnt|tj tj fr|jj }|j }|j|kr|j|ks||kst nt |jdr"|jj }t |dst|st t t|}n|j }|j|kr|j|ks:||ks:t nt|jt|ks:t d S) aCheck that the dask metadata matches the result. If `numeric_equal`, integer and floating dtypes compare equal. This is useful due to the implicit conversion of integer to floating upon encountering missingness, which is hard to infer statically.rrrr)r0r3r)rrN)updater rrdrrYrfrrrrZrrr}r'rr)rrrrrrrr r rrs&   &" (rcCsLt|j\}}|r.ttt||ksHtnttt||ksHtdS)N)rrrmapr&valuesr)r`reqZ dependenciesZ dependentsr r rassert_max_deps sr)NF)NT)N)N)N)N)N)NT)N)TTN)TTTT)T)T)dZ __future__rrrrCr<Zdistutils.versionrrKrM contextlibrZnumpyr'ZpandasrZpandas.util.testingutilZtestingrZpandas.api.typesrrr r r ImportErrorZpandas.core.commonr extensionsrbaserZ compatibilityrrrrrZlocalrZutilsrrr __version__rrrr#r1r?r=r2rXr[rarkrnrrrsregisterrZrdrtrruobjectrZ Int64IndexZ Float64IndexZ UInt64IndexrrrrrZbool_ZvoidZ datetime64rZstr_Zunicode_rrrrrrrrrrrrrrrrrr r r rs         8   !     >   ,   . L  0 (