ó ¦–Õ\c@`sqddlmZmZmZddlZddlmZddlZy ddl m Z m Z m Z Wn-e k r“ddlm Z m Z m Z nXddlmZmZmZmZmZddlmZdd lmZd Zd jeƒZd jeƒZd jeƒZd jeƒZdjeeƒZd„Zd„Z d„Z!de"fd„ƒYZ#e$d„Z%dS(i(tabsolute_importtdivisiontprint_functionN(tcount(tconcattmergetuniquei(tArraytasarrayt blockwisetgetitemtapply_infer_dtypei(tHighLevelGraph(tflattens\w+s(?:{0:}(?:,{0:})*,?)?s\({}\)s{0:}(?:,{0:})*s ^{0:}->{1:}$cC`sû|jddƒ}tjt|ƒs<tdj|ƒƒ‚n|jdƒ\}}gtjt|ƒD]}t tjt |ƒƒ^qd}gtjt|ƒD]}t tjt |ƒƒ^q›}t |ƒdkrë|ddkrë|dn|}||fS( sÜ Parse string signatures for a generalized universal function. Arguments --------- signature : string Generalized universal function signature, e.g., ``(m,n),(n,p)->(m,p)`` for ``np.matmul``. Returns ------- Tuple of input and output core dimensions parsed from the signature, each of the form List[Tuple[str, ...]], except for one output. For one output core dimension is not a list, but of the form Tuple[str, ...] t ts Not a valid gufunc signature: {}s->iiÿÿÿÿt,i( treplacetretmatcht _SIGNATUREt ValueErrortformattsplittfindallt _ARGUMENTttuplet_DIMENSION_NAMEtlen(t signaturetin_txttout_txttargtinstouts((s0lib/python2.7/site-packages/dask/array/gufunc.pyt_parse_gufunc_signatures442cC`sTt|ƒ}t|tƒs!dn t|ƒ}|dk rT|dk rTtdƒ‚n|ryt|tƒ rytdƒ‚n|dkr‹|n|g}ttt|ƒƒ}tg|D]}t|ƒdkr³t^q³ƒ} |r| dkrûtdƒ‚nt|ƒ|dg}n||} |dk r¬t|tƒsItdƒ‚n|r¬|d} t| ƒdkrztdƒ‚nx,|D]!} | | krtdƒ‚qqWq¬n|dkr$|dk ròg| D]} | rà|fntƒ^qË}qBg| D]"} tt t| ƒ dƒƒ^qù}nt|tƒsBtd ƒ‚ng|D]$}t|tƒrg|fn|^qI}| dkr§|t|ƒkr§||t|ƒksÉ| dkrØ||t|ƒkrØtd ƒ‚n||}|rî|n,g|D]"}tt t|ƒ dƒƒ^qõ}|| }xYt t ||ƒƒD]B\}\}} t|ƒt| ƒkr=td j |ƒƒ‚q=q=W|sèx¾t t ||ƒƒD]B\}\}}t|ƒt|ƒkrŸtd j |ƒƒ‚qŸqŸWnb|rJ|d}x)|D]!} || krÿtd ƒ‚qÿqÿW|d}g|D] }|^q5}n||fS(sÿ Validates logic of `axes`/`axis`/`keepdims` arguments and normalize them. Refer to [1]_ for details Arguments --------- axes: List of tuples axis: int keepdims: bool input_coredimss: List of Tuple of dims output_coredimss: List of Tuple of dims Returns ------- input_axes: List of tuple of int output_axes: List of tuple of int References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html#optional-keyword-arguments is>Only one of `axis` or `axes` keyword arguments should be givens`axes` has to be of type listis.`keepdims` can only be used for scalar outputss*`axis` argument has to be an integer values9`axis` can be used only, if one core dimension is presents3To use `axis`, all core dimensions have to be equals `axes` argument has to be a listsRThe number of `axes` entries is not equal the number of input and output argumentssvThe number of `axes` entries for argument #{} is not equal the number of respective input core dimensions in signatureswThe number of `axes` entries for argument #{} is not equal the number of respective output core dimensions in signatures7To use `keepdims`, all core dimensions have to be equalN( Rt isinstancetlisttNoneRtfiltertTruetintRtranget enumeratetzipR(taxestaxistkeepdimstinput_coredimsstoutput_coredimsstnintnouttfiltered_core_dimstxtnr_outputs_with_coredimst core_dimstcd0tcdticdtat output_axestocdt input_axestidxtiaxtoaxticd0tiax0t_((s0lib/python2.7/site-packages/dask/array/gufunc.pyt_validate_normalize_axes:sl !1        .214" ; ( (     cE `s˜|jdd"ƒ}|jdd"ƒ}|jdtƒ}|jdd"ƒ}|jdd"ƒ}|jdd"ƒ} |jdtƒ} t|tƒsœtdƒ‚nt|ƒ\} } t| tƒsÃd"n t| ƒ‰|d"kr| rùt j |d |ƒ} n|} t | ||d dˆƒ}nt|t tfƒr{ˆd"krrt|ƒd kr_t d ƒ‚n|}|d }qŸ|}n$ˆd"k r–t dƒ‚n|g}| rÃt j |d |d|ƒ}n|d"krØi}nt|||| | ƒ\}}g|D]}t|ƒ^qý}t| ƒt|ƒkrPt dt| ƒt|ƒfƒng}x˜t||| ƒD]„\}‰}|j‰t ‡fd†ˆDƒƒ‰t ‡fd†ttˆƒ d d ƒDƒƒˆ}|j|ƒ}|j|ƒqiW|}g|D]}|j^qþ}g|D]}|j^q}gt|| ƒD]"\}}t|ƒt|ƒ^q?}|ryt|ƒnd"}gt||| ƒD](\}}}tt|||ƒƒ^q’}t|Œ‰ˆj|ƒg|D])}t d„t|||ƒDƒƒ^qà} gt| | ƒD]\}!}"|!|"^q}#| rSt| dtƒnt ƒ}$i}%i}&x›t|#||ƒD]‡\}'‰}(xut|'ˆ|(ƒD]a\})}*}+|%j|)gƒ},|,j|*ƒ|,|%|)<|&j|)gƒ}-|-j|+ƒ|-|&|)t(j)ˆ|>d|1gƒ}?t*|?ˆd |=d!|<d|9ƒ}@|rÌt|@jƒt+d"ƒft|:ƒt j,f}A|@|A}@nd"gt|@jƒ}Bx7ttt|:ƒ d ƒ|:ƒD]\‰}Cˆ|B|C(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. **kwargs : dict Extra keyword arguments to pass to `func` Returns ------- Single dask.array.Array or tuple of dask.array.Array Examples -------- >>> import dask.array as da >>> import numpy as np >>> def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> mean, std = da.apply_gufunc(stats, "(i)->(),()", a) >>> mean.compute().shape (10, 20) >>> def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> c = da.apply_gufunc(outer_product, "(i),(j)->(i,j)", a, b, vectorize=True) >>> c.compute().shape (10, 20, 30, 40) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html R-R.R/t output_dtypest output_sizest vectorizet allow_rechunks$`signature` has to be of type stringRt apply_gufuncis_Must specify single dtype or list of one dtype for `output_dtypes` for function with one outputisSMust specify tuple of dtypes for `output_dtypes` for function with multiple outputstotypessDAccording to `signature`, `func` requires %d arguments, but %s givenc3`s1|]'}|dkr|n |tˆƒVqdS(iN(R(t.0R;(tshape(s0lib/python2.7/site-packages/dask/array/gufunc.pys :sc3`s!|]}|ˆkr|VqdS(N((RLti(R@(s0lib/python2.7/site-packages/dask/array/gufunc.pys ;scs`s|]}d|VqdS(s __loopdim%d__N((RLtd((s0lib/python2.7/site-packages/dask/array/gufunc.pys Jstkeys1Dimension `'{}'` with different lengths in arrayssÆCore dimension `'{}'` consists of multiple chunks. To fix, rechunk into a single chunk along this dimension or set `allow_rechunk=True`, but beware that this may increase memory usage significantly.cs`s'|]\}}|dkr|VqdS(iN((RLtstc((s0lib/python2.7/site-packages/dask/array/gufunc.pys hss1Dimension `'{}'` with different chunksize presenttdtypet concatenatet-c3`s|]}ˆ|VqdS(N((RLRO(t core_shapes(s0lib/python2.7/site-packages/dask/array/gufunc.pys ‰ss%s_%d-%sc`s=i|]3}ˆr!t|ˆfn|ˆf|dˆ“qS(i(R (RLRP(tcore_chunkindsRNt leaf_nameR3(s0lib/python2.7/site-packages/dask/array/gufunc.pys Žs t dependenciestchunksRMN(i(-tpopR&tFalseR$tstrt TypeErrorR#R%RtnpRHR RRRERR,RMR*t transposetappendRZtmaxtdictRtupdatetgettitemstsettunionRRRR R)R(R t __dask_keys__RRR tfrom_collectionsRtslicetnewaxis(EtfuncRtargstkwargsR-R.R/RFRGRHRIR0R1ttempfuncRKR>R<R;ttransposed_argsR tinput_coredimsttidcttransposed_argt input_shapest input_chunkssRQR9t num_loopdimst max_loopdimstnR:tcore_input_shapestloop_input_dimsstlRRt input_dimsstloop_output_dimst dimsizesst chunksizesstdimst chunksizestdimtsizet chunksizetdimsizest chunksizes_tsizestrelevant_chunksizestargindsttmptloop_output_shapetloop_output_chunkstkeystnamettokent leaf_arrsR=todtRAtcore_output_shapet output_shapet output_chunkstleaf_dsktgraphtleaf_arrtslicesttidcstoatj((RWRVRNR@RXR3RMs0lib/python2.7/site-packages/dask/array/gufunc.pyRJ˜sèd!        " 48A  6,!""   * $%      .   0 , tgufunccB`s eZdZd„Zd„ZRS(sÁ Binds `pyfunc` into ``dask.array.apply_gufunc`` when called. Parameters ---------- pyfunc : callable Function to call like ``func(*args, **kwargs)`` on input arrays (``*args``) that returns an array or tuple of arrays. If multiple arguments with non-matching dimensions are supplied, this function is expected to vectorize (broadcast) over axes of positional arguments in the style of NumPy universal functions [1]_ (if this is not the case, set ``vectorize=True``). If this function returns multiple outputs, ``output_core_dims`` has to be set as well. signature : String, keyword only Specifies what core dimensions are consumed and produced by ``func``. According to the specification of numpy.gufunc signature [2]_ axes: List of tuples, optional, keyword only A list of tuples with indices of axes a generalized ufunc should operate on. For instance, for a signature of ``"(i,j),(j,k)->(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. Returns ------- Wrapped function Examples -------- >>> import dask.array as da >>> import numpy as np >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> gustats = da.gufunc(stats, signature="(i)->(),()", output_dtypes=(float, float)) >>> mean, std = gustats(a) >>> mean.compute().shape (10, 20) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> guouter_product = da.gufunc(outer_product, signature="(i),(j)->(i,j)", output_dtypes=float, vectorize=True) >>> c = guouter_product(a, b) >>> c.compute().shape (10, 20, 30, 40) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html cK`sñ||_|jddƒ|_|jdtƒ|_|jddƒ|_|jddƒ|_|jdtƒ|_|jddƒ|_ |jddƒ|_ |jdtƒ|_ |rÆt d ƒ‚nd j d t|jƒd|jƒ|_dS( NRRHR-R.R/RGRFRIs(Unsupported keyword argument(s) providedsì Bound ``dask.array.gufunc`` func: ``{func}`` signature: ``'{signature}'`` Parameters ---------- *args : numpy/dask arrays or scalars Arrays to which to apply to ``func``. Core dimensions as specified in ``signature`` need to come last. **kwargs : dict Extra keyword arguments to pass to ``func`` Returns ------- Single dask.array.Array or tuple of dask.array.Array Rm(tpyfuncR[R&RR\RHR-R.R/RGRFRIR^RR]t__doc__(tselfRžRo((s0lib/python2.7/site-packages/dask/array/gufunc.pyt__init__s cO`sjt|j|jd|jd|jd|jd|jd|jd|jd|j p`|j dt ƒ||ŽS(NRHR-R.R/RGRFRI( RJRžRRHR-R.R/RGRFRIR[R\(R RnRo((s0lib/python2.7/site-packages/dask/array/gufunc.pyt__call__!s        (t__name__t __module__RŸR¡R¢(((s0lib/python2.7/site-packages/dask/array/gufunc.pyR©sX c `spdddddddh}t|ƒjˆjƒƒrEtdƒ‚n‡‡fd †}d jd ˆƒ|_|S( sk Decorator for ``dask.array.gufunc``. Parameters ---------- signature : String Specifies what core dimensions are consumed and produced by ``func``. According to the specification of numpy.gufunc signature [2]_ axes: List of tuples, optional, keyword only A list of tuples with indices of axes a generalized ufunc should operate on. For instance, for a signature of ``"(i,j),(j,k)->(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. Returns ------- Decorator for `pyfunc` that itself returns a `gufunc`. Examples -------- >>> import dask.array as da >>> import numpy as np >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> @da.as_gufunc("(i)->(),()", output_dtypes=(float, float)) ... def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> mean, std = stats(a) >>> mean.compute().shape (10, 20) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> @da.as_gufunc("(i),(j)->(i,j)", output_dtypes=float, vectorize=True) ... def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> c = outer_product(a, b) >>> c.compute().shape (10, 20, 30, 40) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html RHR-R.R/RGRFRIs(Unsupported keyword argument(s) providedc`st|dˆˆS(NR(R(Rž(RoR(s0lib/python2.7/site-packages/dask/array/gufunc.pyt _as_gufuncƒss Decorator to make ``dask.array.gufunc`` signature: ``'{signature}'`` Parameters ---------- pyfunc : callable Function matching signature ``'{signature}'``. Returns ------- ``dask.array.gufunc`` R(RgtissubsetRŽR^RRŸ(RRot _allowedkeysR¥((RoRs0lib/python2.7/site-packages/dask/array/gufunc.pyt as_gufunc/s P(&t __future__RRRtnumpyR_t itertoolsRRtcytoolzRRRt ImportErrorttoolztcoreRRR R R thighlevelgraphR R RRt_CORE_DIMENSION_LISTRt_INPUT_ARGUMENTSt_OUTPUT_ARGUMENTSRR#RERJtobjectRR&R¨(((s0lib/python2.7/site-packages/dask/array/gufunc.pyts,     (  ^ ÿ†