B T\@s`ddlZddlZddlZddlmZmZddlmZddlm Z ddl m Z ddZ d d Z dS) N)baseutils)unpack_collections)HighLevelGraph) blockwisecs|dd}|dd}|dd}|dd|di}|dd}tt|t|krztd d d t|Dt|d d |dddDt|pd} | rtd| ddlm} m } m } |dkrtd|r| |\} nDt dd t d|Dddd\}t t|j|ddd} x0|D]$\}}t|tsR|f}||<q8Wtt| |ddd}xL|D]D\}t|dr~tdr~|jtkr~td|jfq~Wdd|D}g}g} g}xZ|D]R\}dkr| |}t|\}}||n| ||j}||fqWi}x<|D]0\}}| |}t|\}}|||||<qJW|sd|pt|dtj||||f|f}t|||f||||d|}tj ||| |d }fd!d |D}rxt!|D]\}krt"rDtt#||||<nbtt$j%rxtfd"d#||D||<n.tttfrt||<nt&d$qWt|}| ||||d%S)&a Tensor operation: Generalized inner and outer products A broad class of blocked algorithms and patterns can be specified with a concise multi-index notation. The ``blockwise`` function applies an in-memory function across multiple blocks of multiple inputs in a variety of ways. Many dask.array operations are special cases of blockwise including elementwise, broadcasting, reductions, tensordot, and transpose. Parameters ---------- func : callable Function to apply to individual tuples of blocks out_ind : iterable Block pattern of the output, something like 'ijk' or (1, 2, 3) *args : sequence of Array, index pairs Sequence like (x, 'ij', y, 'jk', z, 'i') **kwargs : dict Extra keyword arguments to pass to function dtype : np.dtype Datatype of resulting array. concatenate : bool, keyword only If true concatenate arrays along dummy indices, else provide lists adjust_chunks : dict Dictionary mapping index to function to be applied to chunk sizes new_axes : dict, keyword only New indexes and their dimension lengths Examples -------- 2D embarrassingly parallel operation from two arrays, x, and y. >>> z = blockwise(operator.add, 'ij', x, 'ij', y, 'ij', dtype='f8') # z = x + y # doctest: +SKIP Outer product multiplying x by y, two 1-d vectors >>> z = blockwise(operator.mul, 'ij', x, 'i', y, 'j', dtype='f8') # doctest: +SKIP z = x.T >>> z = blockwise(np.transpose, 'ji', x, 'ij', dtype=x.dtype) # doctest: +SKIP The transpose case above is illustrative because it does same transposition both on each in-memory block by calling ``np.transpose`` and on the order of the blocks themselves, by switching the order of the index ``ij -> ji``. We can compose these same patterns with more variables and more complex in-memory functions z = X + Y.T >>> z = blockwise(lambda x, y: x + y.T, 'ij', x, 'ij', y, 'ji', dtype='f8') # doctest: +SKIP Any index, like ``i`` missing from the output index is interpreted as a contraction (note that this differs from Einstein convention; repeated indices do not imply contraction.) In the case of a contraction the passed function should expect an iterable of blocks on any array that holds that index. To receive arrays concatenated along contracted dimensions instead pass ``concatenate=True``. Inner product multiplying x by y, two 1-d vectors >>> def sequence_dot(x_blocks, y_blocks): ... result = 0 ... for x, y in zip(x_blocks, y_blocks): ... result += x.dot(y) ... return result >>> z = blockwise(sequence_dot, '', x, 'i', y, 'i', dtype='f8') # doctest: +SKIP Add new single-chunk dimensions with the ``new_axes=`` keyword, including the length of the new dimension. New dimensions will always be in a single chunk. >>> def f(x): ... return x[:, None] * np.ones((1, 5)) >>> z = blockwise(f, 'az', x, 'a', new_axes={'z': 5}, dtype=x.dtype) # doctest: +SKIP If the applied function changes the size of each chunk you can specify this with a ``adjust_chunks={...}`` dictionary holding a function for each index that modifies the dimension size in that index. >>> def double(x): ... return np.concatenate([x, x]) >>> y = blockwise(double, 'ij', x, 'ij', ... adjust_chunks={'i': lambda n: 2 * n}, dtype=x.dtype) # doctest: +SKIP Include literals by indexing with None >>> y = blockwise(add, 'ij', x, 'ij', 1234, None, dtype=x.dtype) # doctest: +SKIP See Also -------- top - dict formulation of this function, contains most logic nameNtokendtype adjust_chunksnew_axes align_arraysTz-Repeated elements not allowed in output indexcSsg|]\}}|dkr|qS)).0kvrr3lib/python3.7/site-packages/dask/array/blockwise.py wszblockwise..cSs"h|]}|dk r|D]}|qqS)Nr)rargarrr yszblockwise..rrrzUnknown dimension)Array unify_chunks normalize_argz"Must specify dtype of output arraycSs g|]\}}|dk r||fqS)Nr)rrirrrrscSs t|dS)Nr)len)Zairrrszblockwise..)keyndim__len__z1Index string %s does not match array dimension %dcSs"i|]\}}|dk r|j|jqS)N) numblocksr)rrindrrr szblockwise..z%s-%s_)r! dependenciesr )r%csg|] }|qSrr)rr)chunkssrrrsc3s|]}VqdS)Nr)rr$)r r"rr szblockwise..z4adjust_chunks values must be callable, int, or tuple)r )'poprset ValueErrortoolzZ frequenciesitemsZcorerrrmax partitiondictzipchunks isinstancetuplelisthasattrrrextendappendrrfuncnamestriprtokenizecore_blockwiserZfrom_collections enumeratecallablemapnumbersZIntegralNotImplementedError)funcZout_indargskwargsoutr r r r newrrrZarraysrrrZargindsr!r%Z argindsstrr collectionsZkwargs2Zgraphr1rr)r r&r"rr sa      ,  (        " rcOstdt||S)Nz.The da.atop function has moved to da.blockwise)warningswarnr)rBrCrrratops rI) r?rGr+rrZdelayedrZhighlevelgraphrrr;rIrrrrs   @