B F.\ @s<dZddlmZmZmZddlZddlZddlZddlm Z m Z y*ddl Z e dZ eddrhe dWnek rd Z YnXdd lmZmZde e d iZeded <dZeeZee ddlZejejd ed <WdQRXee*ddlZd dZejed ed<dZWdQRXeepddlZyddlZej jZ!ej jZ"Wn"ek rrej#Z!ej$Z"YnXddZ%ddZ&e%e&d ed<dZWdQRXee*ddl Z e e jddde jd ed<WdQRXej'(dZ)e)dkre)ekre)Zn e*de)d+e,e-e.efddZ/d"ddZ0d d!ZdS)#zY Record known compressors Includes utilities for determining whether or not to compress )print_functiondivisionabsolute_importN)identitypartialbloscZ releasegilTF)ignoring ensure_bytes)compress decompresszlibcCs t|ttfrt|}t|S)N) isinstance memoryview bytearraybytessnappyr )datar?lib/python3.7/site-packages/distributed/protocol/compression.py_fixed_snappy_decompress-srrcCs>yt|Stk r8t|ttfr2tt|SYnXdS)N) lz4_compress TypeErrorrrrr)rrrr_fixed_lz4_compressGs  rc CsByt|Sttfk r<t|ttfr6tt|SYnXdS)N)lz4_decompress ValueErrorrrrrr)rrrr_fixed_lz4_decompressPs  rlz4)clevelcnamezdistributed.comm.compressionautozyszbyte_sample..Ncsg|]\}}||qSrr)r$startend)r&rrr(s)range enumerateappendminzipjoinmapr )r&r'nZstartsZendsir+partsr)r&r'r byte_samplens  r8@c Cstjd}|dkrt}|s$d|fSt||kr8d|fSt|dkrLd|fSt|}t|}t|d}t|||}t||dt|krd|fSt|t kr|j t|}nt|}trt rt|t krt j ||j ddd }d }n |t |}t|d|kr d|fS||fSdS) aX Maybe compress payload 1. We don't compress small messages 2. We sample the payload in a few spots, compress that, and if it doesn't do any good we return the original 3. We then compress the full original, it it doesn't compress well then we return the original 4. We return the compressed result zdistributed.comm.compressionr!Nlr g?rr)Ztypesizer rr)daskconfiggetdefault_compressionr#int compressionsr8typeritemsizerr r ) ZpayloadZmin_sizeZ sample_sizeZnsamples compressionr ZsamplenbytesZ compressedrrrmaybe_compresss4         rDcCsddt|d|DS)z: Decompress frames according to information in the header cSs g|]\}}t|d|qS)r )r?)r$cframerrrr(szdecompress..rB)r2)headerZframesrrrr sr )r9r9r)1__doc__Z __future__rrrZloggingr"r:ZtoolzrrrZ set_nthreadsr5hasattrZset_releasegil ImportErrorZutilsr r r?r=Z getLogger__name__Zloggerr r r rrrZ lz4.blockblockrrZ LZ4_compressZLZ4_uncompressrrr;r<defaultrr3sortedr4strr8rDrrrrsp                    3