B T\]* @sdZddlmZmZmZddlmZddlmZddl Z ddl m Z dd lmZmZmZdd lmZdd lmZdd lmZydd l mZWnek re jZYnXddZe jZe jZe jZe jZee jZee j Z ee j!Z!ee j"Z"e j#Z#e j$Z$e j%Z%e j&Z&yddl m'Z'm(Z(Wn"ek r>e j'Z'e j(Z(YnXe j)Z)e j*Z*e j+Z+ee, e j-Z-WdQRXe j.Z.ee, e j/Z/WdQRXe j0Z0ee, e j1Z1WdQRXd+ddZ2d,ddZ3ddZ4ddZ5ddZ6ddZ7ddZ8d d!Z9d-d"d#Z:d.d%d&Z;d'd(Z.keepdims_wrapped_callable)NN)r)rr r)rrkeepdims_wrappersr!) nancumprod nancumsumFcsx"tjD]}|kr d|<q W|rLtfddtjD}|ttfddtjD}||ttdjdddS)a Coarsen array by applying reduction to fixed size neighborhoods Parameters ---------- reduction: function Function like np.sum, np.mean, etc... x: np.ndarray Array to be coarsened axes: dict Mapping of axis to coarsening factor Examples -------- >>> x = np.array([1, 2, 3, 4, 5, 6]) >>> coarsen(np.sum, x, {0: 2}) array([ 3, 7, 11]) >>> coarsen(np.max, x, {0: 3}) array([3, 6]) Provide dictionary of scale per dimension >>> x = np.arange(24).reshape((4, 6)) >>> x array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11], [12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23]]) >>> coarsen(np.min, x, {0: 2, 1: 3}) array([[ 0, 3], [12, 15]]) You must avoid excess elements explicitly >>> x = np.array([1, 2, 3, 4, 5, 6, 7, 8]) >>> coarsen(np.min, x, {0: 3}, trim_excess=True) array([1, 4]) rc3s<|]4\}}||r*td|| ntddVqdS)rN)r).0id)rrr szcoarsen..cs&g|]}j|||fqSr)shape)r$r%)rrrr szcoarsen..r )r)rrr enumerater(rZreshape)Z reductionrrZ trim_excessr%ZindZnewshaper)rrrcoarsen^s(  r+csNttrg|jttr8fddt|jD|tddDS)aD Trim boundaries off of array >>> x = np.arange(24).reshape((4, 6)) >>> trim(x, axes={0: 0, 1: 1}) array([[ 1, 2, 3, 4], [ 7, 8, 9, 10], [13, 14, 15, 16], [19, 20, 21, 22]]) >>> trim(x, axes={0: 1, 1: 1}) array([[ 7, 8, 9, 10], [13, 14, 15, 16]]) csg|]}|dqS)r)get)r$r%)rrrr)sztrim..css"|]}t||r| ndVqdS)N)r)r$Zaxrrrr'sztrim..)rrrdictrr)rrr)rrtrims    r.cs||dks tdt||jkr*|Stj|| d}|dkrPt| dnt| |tfddt|jDS)a  Chunk and combine function of topk Extract the k largest elements from a on the given axis. If k is negative, extract the -k smallest elements instead. Note that, unlike in the parent function, the returned elements are not sorted internally. Tr)rNc3s"|]}|krntdVqdS)N)r)r$r%)rk_slicerrr'sztopk..) AssertionErrorabsr(np partitionrrrr)akrrr)rr/rtopks r6cs\|dks tt|||}dtj|d}|dkr<|S|tfddt|jDS)zn Final aggregation function of topk Invoke topk one final time and then sort the results internally. Tr)rc3s*|]"}|krtdddntdVqdS)N)r)r$r%)rrrr'sz!topk_aggregate..)r0r6r2sortrrr)r4r5rrr)rrtopk_aggregates r9cCs||fS)z_ Preparatory step for argtopk Put data together with its original indices in a tuple. r)r4idxrrrargtopk_preprocesssr;cs|dks tdt|trXtt|}tdd|D}tdd|D}n|\}}t||jkrv|Stj|| d}|dkrt | dnt | |t fdd t |j D}t ||t ||fS) a* Chunk and combine function of argtopk Extract the indices of the k largest elements from a on the given axis. If k is negative, extract the indices of the -k smallest elements instead. Note that, unlike in the parent function, the returned elements are not sorted internally. TrcSsg|] \}}|qSrr)r$ai_rrrr)szargtopk..cSsg|]\}}t||jqSr)r2Z broadcast_tor()r$r<Zidxirrrr)s)rNc3s"|]}|krntdVqdS)N)r)r$r%)rr/rrr'szargtopk..)r0rlistr r2Z concatenater1r(Z argpartitionrrrrr) a_plus_idxr5rrr4r:idx2r)rr/rargtopks    rAcsl|dks tt|||\}}dtj|d}t||}|dkrL|S|tfddt|jDS)z Final aggregation function of argtopk Invoke argtopk one final time, sort the results internally, drop the data and return the index only. Tr)rc3s*|]"}|krtdddntdVqdS)Nr7)r)r$r%)rrrr'sz$argtopk_aggregate..)r0rAr2Zargsortrrrr)r?r5rrr4r:r@r)rrargtopk_aggregates  rBcCs,t||||}t||kr(|ddS|S)Nr7)r2arangelen)startstopsteplengthdtypeZresrrrrCsrCcKs|j|f|S)N)astype)rZ astype_dtyperrrrrJ srJCcCs8|dkrt|}||St|}|j|jSdS)NrK)r2ZascontiguousarrayviewZasfortranarrayT)rrIorderrrrrLs    rLcsjtjtdk||dk|jk@}||tfddt|jDS)a Chunk function of `slice_with_int_dask_array_on_axis`. Slice one chunk of x by one chunk of idx. Parameters ---------- x: ndarray, any dtype, any shape i-th chunk of x idx: ndarray, ndim=1, dtype=any integer j-th chunk of idx (cartesian product with the chunks of x) offset: ndarray, shape=(1, ), dtype=int64 Index of the first element along axis of the current chunk of x x_size: int Total size of the x da.Array along axis axis: int normalized axis to take elements from (0 <= axis < x.ndim) Returns ------- x sliced along axis, using only the elements of idx that fall inside the current chunk. rc3s"|]}|krntdVqdS)N)r)r$r%)rr:rrr'@sz,slice_with_int_dask_array..)rJr2int64wherer(rrr)rr:offsetZx_sizer idx_filterr)rr:rslice_with_int_dask_arrays  rSc s|tj}t|dk|t||}d}d}t|xb|D]Z}||k|||k@}t|}t||d|d7||7}|jdkr>||d7}q>W|tfddt |j DS)aT Final aggregation function of `slice_with_int_dask_array_on_axis`. Aggregate all chunks of x by one chunk of idx, reordering the output of `slice_with_int_dask_array`. Note that there is no combine function, as a recursive aggregation (e.g. with split_every) would not give any benefit. Parameters ---------- idx: ndarray, ndim=1, dtype=any integer j-th chunk of idx chunk_outputs: ndarray concatenation along axis of the outputs of `slice_with_int_dask_array` for all chunks of x and the j-th chunk of idx x_chunks: tuple dask chunks of the x da.Array along axis, e.g. ``(3, 3, 2)`` axis: int normalized axis to take elements from (0 <= axis < x.ndim) Returns ------- Selection from all chunks of x for the j-th chunk of idx, in the correct order rrr7c3s"|]}|krntdVqdS)N)r)r$r%)r idx_finalrrr'wsz6slice_with_int_dask_array_aggregate..) rJr2rOrPsumZ zeros_likeZcumsumsizerrr) r:Z chunk_outputsZx_chunksrZx_chunk_offsetZchunk_output_offsetZx_chunkrRZidx_cumr)rrTr#slice_with_int_dask_array_aggregateEs      rW)F)N)N)rK)>__doc__Z __future__rrr functoolsrZtoolzrZnumpyr2rZnpcompatZ compatibilityr r r Zcorer ZutilsrZnumbersrr ImportErrorr!rUZprodminmaxZargminZ nanargminZargmaxZ nanargmaxanyallZnansumZnanprodr"r#ZnanminZnanmaxZmeanAttributeErrorZnanmeanvarZnanvarZstdZnanstdr+r.r6r9r;rArBrCrJrLrSrWrrrrsl       #         9   -