B ÉM^\†žã@s<ddlmZmZmZddlZddlmZddlZddlm Z ddl Z ddlZddlmZddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlmZddlZddlZyddlZWne k r dZYnXddl!Z!ddl"Z"ddl#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*dd l+m,Z,dd l-m.Z.ddl/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;ddlZ>m?Z?ddl@mZddlAmBZBddlCmDZDddlEmFZFddlGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQddlRmSZSmTZTmUZUyddlVZ#Wne k rNYnXe WeX¡ZYdd„ejZj[j\ ]¡DƒZ^e9dd„ƒ _¡e!j`dddd„ƒZae!j`dddd „ƒZbe!j`ddd!d"„ƒZce)jdd#d$„ƒZee!j`d%d&„ƒZfe!j`d'd(„ƒZge!j`d)d*„ƒZhed+d,„ƒZied-d.„ƒZje ke#j:j:¡Zld/d0„Zmd1d2„Znd3d4„Zod5d6„Zpd7d8„Zqd9d:„Zrd;d<„Zsd=d>„Ztd?d@„ZudAdB„ZvdCdD„ZwdÜdFdG„ZxdÝdHdI„ZydÞdJdK„ZzdßdLdM„Z{dàdNdO„Z|dádPdQ„Z}dRdS„Z~efdTdU„Ze €e¡Z‚e ƒ¡Z„GdVdW„dWe…ƒZ†dXdY„Z‡dZd[„Zˆe)jddâd\d]„ƒZ‰dãd_d`„ZŠej‹dakr0eŠdbƒeŒs4t‚ndZŒiZŽe)jddcdd„ƒZdedf„Zdgdh„Z‘didj„Z’edädkdl„ƒZ“e!j`dmdn„ƒZ”e!j`dodp„ƒZ•e!j`dqdr„ƒZ–e!j`dsdt„ƒZ—e!j`dudv„ƒZ/e!j`dwdx„ƒZ˜edådydz„ƒZ™e!j`d{d|„ƒZše!j`d}d~„ƒZ›e!j`dd€„ƒZEedd‚idifdƒd„„ƒZœe)jddæd†d‡„ƒZe)jddçdˆd‰„ƒZždŠd‹„ZŸdèddŽ„Z ddl¡m¢Z¢ddlRmSZSe)jddeSiifd‘d’„ƒZ£e)jdd“d”„ƒZ¤d•d–gd—dŒdeSd‚iiidid^fd˜d™„Z¥e¦fdšd›„Z§dœd„Z¨edždŸ„ƒZ©déd¡d¢„Zªdêd¤d¥„Z«e)jddëd¦d§„ƒZ¬e&d¨d©„ƒZeƒrdªd«„Z®ne!j¯ °d¬¡Z®e)jddìdd®„ƒZ±e)jddde²fd¯d°„ƒZ³e)jddíd²d³„ƒZ´e)jddîd´dµ„ƒZµe)jddïd¶d·„ƒZ¶e)jddðd¸d¹„ƒZ·e)jddñdºd»„ƒZ¸eej¹dfd¼d½„ƒZºed¾d¿„ƒZ»edÀdÁ„ƒZ¼edÂdÃ„ƒZ½edÄdÅ„ƒZ¾ej¿ Àej¿ Áej¿ ÂeÃ¡dÆ¡¡ZÄdÇdÈ„ZÅdÉdÊ„ZÆdËdÌ„ZÇdÍdÎ„ZÈdÏdÐ„ZÉdòdÔdÕ„ZÊdódÖd×„ZËdØdÙ„ZÌdÚdÛ„ZÍdS)ôé)Úprint_functionÚdivisionÚabsolute_importN)Úcontextmanager)Ú timedelta)Úglob)Úsleep)ÚmergeÚmemoizeÚassoc)ÚgenÚqueues)ÚTimeoutError)ÚIOLoopé)Údefault_clientÚ_global_clientsÚClient)ÚPY3ÚEmptyÚWINDOWSÚPY2)Úoffload)Úinitialize_logging)ÚconnectÚrpcÚCommClosedError)Útime)Ú_cleanup_dangling)Úenable_proctitle_on_children)ÚSecurity) ÚignoringÚ log_errorsÚ mp_contextÚget_ipÚget_ipv6ÚDequeHandlerÚreset_logger_locksÚsyncÚiscoroutinefunctionÚthread_state)ÚWorkerÚTOTAL_MEMORYÚ_global_workerscCs$i|]\}}t|tjƒr|j|“qS©)Ú isinstanceÚloggingZLoggerÚlevel)Ú.0ÚnameÚloggerr.r.ú5lib/python3.7/site-packages/distributed/utils_test.pyú Asr6cCsdS)Nr.r.r.r.r5ÚFsr7Zsession)ZscopecCs| d¡ d¡}| d¡|S)NÚdatazfile.pyzprint('hello world!'))ÚmktempÚjoinÚwrite)Útmpdir_factoryÚ local_filer.r.r5Úvalid_python_scriptIs r>cCs(| d¡ d¡}d}| d |¡¡|S)Nr8zdistributed_script.py)zfrom distributed import Clientze = Client('127.0.0.1:8989')zprint(e)Ú )r9r:r;)r<r=Úlinesr.r.r5Úclient_contract_scriptPsrAcCs| d¡ d¡}| d¡|S)Nr8zfile.pyza+1)r9r:r;)r<r=r.r.r5Úinvalid_python_scriptYs rBcCs&x tD]}|ƒ}|jdddqWdS)NF)ÚreportÚ executor_wait)r-Ú_close)Úwr.r.r5Úcleanup_global_workers`s rGc#sNtdd…=t ¡tƒ¦}|j‰t ¡‰ˆ ¡‡‡fdd„}||_|Vyt|t dd| |j¡WnHtk rœ}zt dt|ƒ¡sŒ‚Wdd}~XYntjk r°Yn Xˆ ¡WdQRXtdd…=tƒ}x&ttƒrútdƒtƒ|dksÖt‚qÖWtƒtrBx8t ¡jdd D]$}ttjƒ| ¡WdQRXqWt ¡dS) Ncs"ˆ ¡z ˆƒWdˆ ¡XdS)N)ÚclearÚsetr.)Ú is_stoppedÚ orig_startr.r5Ústartqs zloop..startgà?)Zcallback_timeoutzIOLoop is clos(ed|ing)gš™™™™™¹?é T)Ú recursive)r-rrHÚ pristine_looprLÚ threadingÚEventrIr(rGÚadd_callbackÚstopÚRuntimeErrorÚreÚmatchÚstrrrÚwaitrrÚAssertionErrorrrÚpsutilÚProcessZchildrenr!Z NoSuchProcessÚ terminate)ÚlooprLÚeZchildr.)rJrKr5r]gs: r]c csntƒ^}tj|jdd}d|_| ¡t ¡}| |j¡| ¡|V| |j ¡|j ddWdQRXdS)Nztest IOLoop)Útargetr3Té)Útimeout)rOrPZThreadrLÚdaemonrQrRrIrXrSr:)r]ÚthreadZloop_startedr.r.r5Úloop_in_thread—srdccs(ddl}|j ¡}|V|jdddS)Nr)Zlinger)ÚzmqZContextÚinstanceZdestroy)reÚctxr.r.r5Úzmq_ctx¦s rhccszt ¡t ¡tƒ}| ¡t ¡|ks.t‚z |VWdy|jddWnttfk rbYnXt ¡t ¡XdS)NT)Úall_fds) rZclear_instanceZ clear_currentZmake_currentZcurrentrYÚcloseÚKeyErrorÚ ValueError)r]r.r.r5rO®s rOc #s”ddl}ddlm}| ¡‰iˆ_dˆ_‡fdd„}| d|¡$| d|¡ˆVWdQRXWdQRXx|j ¡D]}| ¡qvW|j ¡dS)Nr)Úremote_magiccsˆS)Nr.r.)Úipr.r5r$Èszmock_ipython..get_ipzIPython.get_ipythonz&distributed._ipython_utils.get_ipython)ÚmockZdistributed._ipython_utilsrmZMockZuser_nsZkernelZpatchZ_clientsÚvaluesZ stop_channelsrH)rormr$Zkcr.)rnr5Úmock_ipythonÀsrqcCs$tjj ¡tjj t t¡¡dS)N)ÚdaskÚconfigrHÚupdateÚcopyÚdeepcopyÚoriginal_configr.r.r.r5Úreset_config×srxcs"tsˆSt ˆ¡‡fdd„ƒ}|S)zŠ A decorator to disable debug facilities during timing-sensitive tests. Warning: this doesn't affect already created IOLoops. c s@tj d¡}|dk rtjd=z ˆ||ŽS|dk r:|tjd<XdS)NÚPYTHONASYNCIODEBUG)ÚosÚenvironÚget)ÚargsÚkwargsZold_asyncio_debug)Úfuncr.r5Úwrappedçs znodebug..wrapped)rÚ functoolsÚwraps)rr€r.)rr5ÚnodebugÜsrƒcCs$tj d¡|_|jdk r tjd=dS)za A setup_module() that you can install in a test module to disable debug facilities. ryN)rzr{r|Ú_old_asyncio_debug)Úmoduler.r.r5Únodebug_setup_moduleõs r†cCs|jdk r|jtjd<dS)ze A teardown_module() that you can install in a test module to reenable debug facilities. Nry)r„rzr{)r…r.r.r5Únodebug_teardown_moduleÿs r‡cCs|dS)Nrr.)Úxr.r.r5Úincsr‰cCs|dS)Nrr.)rˆr.r.r5ÚdecsrŠcCs||S)Nr.)rˆÚyr.r.r5ÚmulsrŒcCs||S)Nr.)rˆr‹r.r.r5ÚdivsrcCs|dkrt|dƒSdSdS)NrrT)Údeep)Únr.r.r5rŽsrŽcCstdƒ‚dS)Nzhello!)rT)rˆr.r.r5ÚthrowssrcCs|dS)Nér.)rˆr.r.r5Údouble#sr’ç{®Gáz”?cCst|ƒ|dS)Nr)r)rˆÚdelayr.r.r5Úslowinc'sr•cCst|ƒ|dS)Nr)r)rˆr”r.r.r5Úslowdec,sr–cCst|ƒd|S)Nr‘)r)rˆr”r.r.r5Ú slowdouble1sr—cCs"ddlm}t|ƒ|ƒ|dS)Nr)Úrandomr)r˜r)rˆZscaler˜r.r.r5Ú randominc6sr™cCst|ƒ||S)N)r)rˆr‹r”r.r.r5Úslowadd<sršcCst|ƒt|ƒS)N)rÚsum)Úseqr”r.r.r5ÚslowsumAsrcOs0| dd¡}t|ƒt|ƒdkr(|dS|SdS)Nr”g{®Gáz”?rr)r|rÚlen)r}r~r”r.r.r5ÚslowidentityFs rŸcCs|ƒ|}x|ƒ|krqWdS)z* Burn CPU for *duration* seconds. Nr.)ZdurationZtimerÚdeadliner.r.r5Úrun_forOs r¡c@seZdZdd„Zdd„ZdS)Ú_ModuleSlotcCs||_||_dS)N)ÚmodnameÚslotname)Úselfr£r¤r.r.r5Ú__init__^sz_ModuleSlot.__init__cCsttj|j|jƒS)N)ÚgetattrÚsysÚmodulesr£r¤)r¥r.r.r5r|bsz_ModuleSlot.getN)Ú__name__Ú __module__Ú__qualname__r¦r|r.r.r.r5r¢]sr¢cs&ttdƒ‰ttƒ‰‡‡‡fdd„}|S)zi Return a function that returns a result (or raises an exception) from *items* at each call. Ú _varying_dictcsNˆ ¡}|ˆ}|tˆƒkr"t‚n(ˆ|}|d|ˆ<t|tƒrF|‚n|SdS)Nr)r|ržÚ IndexErrorr/Ú Exception)ZdctÚirˆ)ÚitemsÚkeyÚslotr.r5rrs zvarying..func)r¢rªÚnextÚ_varying_key_gen)r±rr.)r±r²r³r5Úvaryingfs r¶cCsdd„}|tt|ƒfS)zi Like *varying*, but return the full specification for a map() call on multiple items lists. c_s |||ŽS)Nr.)rr}r~r.r.r5Úapply‡szmap_varying..apply)Úmapr¶)Z itemslistsr·r.r.r5Úmap_varying‚sr¹ccst |¡Vt |d¡‚dS)Nr)rrÚReturn)rˆr”r.r.r5Úgenincsr»TcCs0|rt |¡}t|ddƒ}tƒ}t|||ƒdS)Nz Úexec)ÚtextwrapÚdedentÚcompileÚglobalsr¼)Úcoder¾Únsr.r.r5Úcompile_snippet“s rÃ)ér`zp async def asyncinc(x, delay=0.02): await gen.sleep(delay) return x + 1 c#srytˆ‰Wn>tk rJt ¡‰tˆ<tj‡‡fdd„ƒ}|ƒYnXˆ ¡V}|dkrdt‚n t |¡‚dS)zR Read one message at a time from a comm that reads lists of messages. c3sXxByˆ ¡V}Wntk r&PYnXx|D]}ˆ |¡q.WqWˆ d¡tˆ=dS)N)ÚreadrZ put_nowaitÚ_readone_queues)ZmessagesÚmsg)ÚcommÚqr.r5Úbackground_read´s z readone..background_readN) rÆrkr ÚQueuerÚ coroutiner|rrº)rÈrÊrÇr.)rÈrÉr5Úreadone©s rÍc Ksxddlm}tƒ\}|fddi|—Ž}| d¡}xt|ƒD]}| |j¡q:Wz| ¡Wd|jddXWdQRXdS)Nr)Ú SchedulerÚvalidateTz 127.0.0.1)ri)ÚdistributedrÎrOrLÚrangeÚputÚaddressrj)rÉZnputsr~rÎr]Ú schedulerÚdoner°r.r.r5Ú run_schedulerÉs rÖcs¤ddlm}tƒtƒ‚tƒp}| ¡}||fddi|—Ž‰| ‡fdd„¡| ˆj¡z t j ‡fdd„ƒ}| |¡Wd|jdd XWdQRXWdQRXdS) Nr)r+rÏTcs ˆ d¡S)Nr)Ú_startr.)Úworkerr.r5r7âszrun_worker..c3sˆj ¡VdS)N)Z_closedrXr.)rØr.r5Úwait_until_closedåsz%run_worker..wait_until_closed)ri)rÐr+r'r"rOr|Úrun_syncrÒrÓrrÌrj)rÉÚscheduler_qr~r+r]Úscheduler_addrrÙr.)rØr5Ú run_workerÚsrÝcs–ddlm}tƒztƒh}| ¡}||fddi|—Ž‰| ‡fdd„¡| ˆj¡z| ¡Wd| ˆj ¡|j ddXWdQRXWdQRXdS)Nr)ÚNannyrÏTcs ˆ d¡S)Nr)r×r.)rØr.r5r7õszrun_nanny..)ri)rÐrÞr"rOr|rÚrÒrÓrLrErj)rÉrÛr~rÞr]rÜr.)rØr5Ú run_nannyîsrßc#sZttjƒ‰ˆr$ts$t ¡ttjƒ‰dV‡fdd„‰tj‡‡‡fdd„ƒ}| |¡dS)Ncst dttjƒˆ¡dS)Nz!some RPCs left active by test: %s)ÚpytestÚfailrIrÚactiver.)Ú active_beforer.r5rá szcheck_active_rpc..failc3st‡fdd„ˆˆdVdS)NcstttjƒˆƒdkS)Nr)ržrIrrâr.)rãr.r5r7sz0check_active_rpc..wait..)raÚ fail_func)Úasync_wait_forr.)rãÚactive_rpc_timeoutrár.r5rXszcheck_active_rpc..wait) rIrrârÚgcZcollectrrÌrÚ)r]rærXr.)rãrærár5Úcheck_active_rpcþs rèc cs$tƒ\}}||fVWdQRXdS)N)Úcluster)r]rÔÚworkersr.r.r5Úcluster_fixturesrëcCs|\}}|S)Nr.)rërÔrêr.r.r5ÚssrìcCs|\}}|dS)Nrr.)rërÔrêr.r.r5Úa%srícCs|\}}|dS)Nrr.)rërÔrêr.r.r5Úb+srîc cs.|\}}t|d|d}|VWdQRXdS)NrÓ)r])r)r]rërÔrêÚclientr.r.r5rï1srïc cs.|\}}t|d|d}|VWdQRXdS)NrÓ)r])r)r]rërÔrêrïr.r.r5Úclient_secondary8srðc ks\|ptƒ}t|pid|ƒ}t|p"id|ƒ}tf||dœ|—Ž\}}||fVWdQRXdS)NÚsecurity)Ú worker_kwargsÚscheduler_kwargs)Útls_only_securityrré)ròrórñr~rìrêr.r.r5Útls_cluster_context?s rõc cs(t|d\}}||fVWdQRXdS)N)rñ)rõ)r]rñrÔrêr.r.r5Útls_clusterLsröc cs0|\}}t|d||d}|VWdQRXdS)NrÓ)rñr])r)rör]rñrìrêrïr.r.r5Ú tls_clientRsr÷cCstƒS)N)rôr.r.r.r5rñYsrñr‘Fc#sÀt ¡}tƒx$t ¡D]\}}t |¡ |¡qWtƒt ƒ<}t ||ƒ$|r\t} nt} t ¡} t jt| |df|d}| |¡d|_| ¡g‰xft|ƒD]Z}t ¡} dt ¡}td|tdœ|ƒ}t j| | | f|d}| |¡ˆ || |dœ¡q¦WxˆD]}|d ¡q Wy(x"ˆD]}|djd d |d<q(WWn"tk rjtj d¡‚YnX| ¡‰tƒ}z¦y|d }d| d¡i‰Wnt!k r°i‰YnXt"ˆfˆŽB}x:| #|j$¡}t%|ƒ|krÞPtƒ|d krÂtdƒ‚qÂWWdQRXdˆidd„ˆDƒfVWdt& 'd¡| #‡‡fdd„¡| #‡‡fdd„¡| (¡| )¡| j* )¡| j+ )¡xBˆD]:}|d (¡|d )¡|dj* )¡|dj+ )¡q~W| ,d¡~x$dd„ˆDƒD]}|j,dd qØWt-t.ƒ~~~WdQRXˆdd…=x0t/dƒD]$}t-t0ƒt1 2|¡WdQRXqWXy t3ƒ}Wnt4k rfYn X| )¡WdQRXWdQRXtƒ}x0t5|ƒrºt6dƒtƒ|dksŒt7dƒ‚qŒWdS)Nr)r_r}r~Tz_test_worker-%s)ÚncoresZ local_dirÚmemory_limit)ÚprocÚqueueÚdirrúrûr`)rarÓzWorker failed to start in testrñÚconnection_argsrïzTimeout on cluster creationcSs$g|]}|dt |d¡dœ‘qS)rÓrú)rÓrú)ÚweakrefÚref)r2rFr.r.r5ú ¤szcluster..zClosing out test clustercstdd„ˆDƒdˆdS)NcSsg|]}|d‘qS)rÓr.)r2rFr.r.r5rªsz-cluster....gà?)raÚ rpc_kwargs)Údisconnect_allr.)rrêr.r5r7ªszcluster..cstˆdˆdS)Ngà?)rar)Ú disconnectr.)rÚsaddrr.r5r7sr‘cSsg|]}|d‘qS)rúr.)r2rFr.r.r5r¼sz_test_worker-*g{®Gáz„?z%Workers still around after one second)8rþÚWeakSetrxÚlogging_levelsr±r0Ú getLoggerÚsetLevelrrOrèrßrÝr#rËr[rÖÚaddrbrLrÑÚuuidZuuid4r r,Úappendr|rràZxfailr¯rZget_connection_argsrkrrÚrøržr4Údebugr\rjZ_readerZ_writerr:r!ÚUnboundLocalErrorrÚOSErrorÚshutilZrmtreerrlÚlistrrY)ZnworkersZnannyròræróÚwsr3r1r]Z_run_workerrÛrÔr°rÉÚfnr~rúrØrLrñrìrørFrïr.)rrrêr5ré^s¦ rérÄc #sJˆpi‰tj‡‡fdd„ƒ}ttƒt t|d|ƒ¡VWdQRXdS)Nc 3s@tttƒ,tˆfˆŽ}|jddVWdQRXWdQRXdS)NT)rj)r!ÚEnvironmentErrorrrr\)rF)Úaddrrr.r5Ú do_disconnectØsz!disconnect..do_disconnect)Úseconds)rrÌr!rÚwith_timeoutr)rrarrr.)rrr5rÔs rc#s‡‡fdd„|DƒVdS)Ncsg|]}t|ˆˆƒ‘qSr.)r)r2r)rrar.r5räsz"disconnect_all..r.)Z addressesrarr.)rrar5râsrcCs>y tj d¡stj d¡|ƒ}Wntk r4YnXt|ƒS)Nz --runslowzneed --runslow option to run)ràrsZ getoptionÚmarkÚskipÚAttributeErrorrƒ)rr.r.r5ÚslowçsrrMcs‡fdd„}|S)zn Coroutine test @gen_test(timeout=5) def test_foo(): yield ... # use tornado coroutines cs‡‡fdd„}|S)Nc sLtƒ<}tˆƒrˆ}n t ˆ¡}z|j|ˆdWd| ¡XWdQRXdS)N)ra)rOr)rrÌrÚrS)r]Zcor)rrar.r5Ú test_funcús z&gen_test.._..test_funcr.)rr)ra)rr5Ú_ùs zgen_test.._r.)rarr.)rar5Úgen_testòsr)rÎ)r+c#sötfˆdˆdœ|—Ž‰ˆ |¡}‡‡‡‡‡fdd„t|ƒDƒ}x|D]} |dj| _qDWdd„t||ƒDƒVtƒ} xntˆjƒt|ƒks tdd„ˆj ¡Dƒƒrât d ¡Vtƒ| d krvdd„|DƒVˆj ddVtd ƒ‚qvWt ˆ|f¡‚dS)NT)r]rÏrñc sLg|]D\}}ˆˆjf|d|ˆˆddœt|ƒdkr@tˆ|dƒnˆ—Ž‘qS)rT)rør3rñr]rÏr‘)rÓržr )r2r°Úncore)r+r]rìrñròr.r5rsz!start_cluster..rcSsg|]\}}| |d¡‘qS)r)r×)r2rrFr.r.r5rscss|]}|jdkVqdS)N)rÈ)r2rÈr.r.r5ú sz start_cluster..g{®Gáz„?r`cSsg|]}|jdd‘qS)r)ra)rE)r2rFr.r.r5r"s)ÚfastzCluster creation timeout)rÎrLÚ enumeraterÚziprržrêÚanyZstream_commsrprrrjr¯rº)rørÜr]rñr+róròrÕrêrFrLr.)r+r]rìrñròr5Ú start_clusters" r%c#sBt d¡tjdd„ƒ‰‡fdd„|DƒV| ¡V| ¡dS)NzClosing out test clusterc ss*ttttƒ|jddVWdQRXdS)NF)rC)r!rrrrE)rFr.r.r5Ú end_worker,szend_cluster..end_workercsg|]}ˆ|ƒ‘qSr.r.)r2rF)r&r.r5r1szend_cluster..)r4rrrÌrjrS)rìrêr.)r&r5Úend_cluster(s r')z 127.0.0.1r)z 127.0.0.1r‘z 127.0.0.1c sDddlm‰ttddœˆƒ‰‡‡‡‡‡‡‡‡‡‡ ‡ ‡‡f dd„}|S)Nr)rr`)rùZ death_timeoutcs<tˆƒst ˆ¡‰‡‡‡‡‡‡‡‡‡‡ ‡ ‡‡‡ fdd„}|S)Ncstdd…=t ¡ttjƒ‰tƒtj ddi¡x$t ¡D]\}}t |¡ |¡q._..test_func..coror‘)rar8F)rCrDZrunningcs:g|]2\}}|ˆkrd|jkrd|jkrd|jkr|‘qS)ZThreadedz watch messagezTCP-Executor)r3)r2ÚtÚv)Úactive_threads_startr.r5r£s z=gen_cluster.._..test_func..g{®Gáz„?r`r)Úprofile)*r-rrHrIrPZ_activerxrrrsrr±r0rrrOrèrrÌrÚr§r8rr&Zclear_all_instancesrEr+rjrrrrrÐr3Z call_stackr¨Ú_current_framesrYrr!rr*Zon_event_loop_thread)r3r1r-r/rFrLZbadr3ÚtidrcZcall_stacks)rr+ræÚcheck_new_threadsrïr.rsrrørÔrórñrarò)r2r]rêr5rMsX ,.$ z)gen_cluster.._..test_func)r)rrÌ)rr) rr+rær6rïr.rsrørÔrórñrarò)rr5rIs &jzgen_cluster.._)rÐrr r,) rørÔrarñr+rïróròr.rærsr6rr.) rr+rær6rïr.rsrørÔrórñraròr5Úgen_cluster6s $or7cCs$y |ƒdS|k rdSXdS)NFTr.)rÚexcr.r.r5Úraises»s r9cCs¢| ¡dkržtj d¡r&| tj¡n| tj¡zLtjddkrN| d¡n.t ƒ}x&| ¡dkrzt ƒ|dkrztdƒqVWWdtt ƒ| ¡WdQRXXdS)NÚwinrrÄrMg{®Gáz”?)Zpollr¨ÚplatformÚ startswithZsend_signalÚsignalZCTRL_BREAK_EVENTÚSIGINTÚversion_inforXrrr!rÚkill)rúrLr.r.r5Úterminate_processÃs rAckstj|d<tj|d<tj d¡r*tj|d<d}t|ƒ}tj d¡r^tj tj d|d¡|d<ntj tj d|d¡|d<tj|f|Ž}z*y |VWntk r¬d }‚YnXWdzt |ƒWd| ¡\}}|rtd |ddƒt| ¡ƒtdƒt| ¡ƒXXdS)NÚstdoutÚstderrr:Z creationflagsFZScriptsrÚbinTz+ Print from stderr %s ================= z& Print from stdout ================= )Ú subprocessÚPIPEr¨r;r<ZCREATE_NEW_PROCESS_GROUPrrzÚpathr:ÚprefixÚPopenr¯rAZcommunicateÚprintÚdecode)r}r~Zdump_stdoutrúÚoutÚerrr.r.r5ÚpopenÖs. rNr`cCstt|tƒst‚tƒ|}xV|tƒ}|dkr:td|fƒ‚ytj||d}Wntk r`YqX| ¡PqWdS)NrzFailed to connect to %s)ra) r/ÚtuplerYrrTÚsocketÚcreate_connectionrrj)rÓrar Zsockr.r.r5Ú wait_for_portùs rRçü©ñÒMbP?cCsJtƒ|}x:|ƒsDt|ƒtƒ|kr|dk r2|ƒt d|f¡qWdS)Nz&condition not reached until %s seconds)rrràrá)Ú predicateraräÚperiodr r.r.r5Úwait_for s rVccsNtƒ|}x>|ƒsHt |¡Vtƒ|kr|dk r6|ƒt d|f¡qWdS)Nz&condition not reached until %s seconds)rrrràrá)rTrarärUr r.r.r5rås råcCsŒd}}z\y>t tjtj¡}| d¡| d¡t | ¡dd…¡}Wntk r\dSXdSWd|dk rv| ¡|dk r†| ¡XdS)z‚ Return whether IPv6 is locally functional. This doesn't guarantee IPv6 is properly configured outside of localhost. N)z::rr`r‘FT) rPZAF_INET6ZSOCK_STREAMZbindZlistenrQZgetsocknamerrj)ZservZclir.r.r5Úhas_ipv6s rWcCs|S)Nr.)rr.r.r5Ú requires_ipv67srXz ipv6 requiredccs(|dkrd}t|||dV}| ¡dS)zh Check that it is possible to connect to the distributed *addr* within the given *timeout*. Ngà?)rarý)rÚabort)rrarýrÈr.r.r5Úassert_can_connect>s rZc cs>|dkrd}t |¡t|||dV}| ¡WdQRXdS)zj Check that it is impossible to connect to the distributed *addr* within the given *timeout*. Ngà?)rarý)ràr9rrY)rrarýZexception_classrÈr.r.r5Úassert_cannot_connectKs r[Útcpccs|||f}td||ff|žŽtd|tƒ|ff|žŽg}tƒrr|td||ff|žŽtd|tƒ|ff|žŽg7}|VdS)zT Check that the local *port* is reachable from all IPv4 and IPv6 addresses. z%s://127.0.0.1:%dz %s://%s:%dz %s://[::1]:%dz%s://[%s]:%dN)rZr$rWr%)ÚportrarýÚprotocolr}Úfuturesr.r.r5Ú&assert_can_connect_from_everywhere_4_6Ysr`ccs|||f}td||ff|žŽtd|tƒ|ff|žŽg}tƒrr|td||ff|žŽtd|tƒ|ff|žŽg7}|VdS)zK Check that the local *port* is reachable from all IPv4 addresses. z%s://127.0.0.1:%dz %s://%s:%dz %s://[::1]:%dz%s://[%s]:%dN)rZr$rWr[r%)r]rarýr^r}r_r.r.r5Ú$assert_can_connect_from_everywhere_4ksraccs‚||f}td|f|žŽg}tƒdkrD|tdtƒ|ff|žŽg7}tƒrx|td|f|žŽtdtƒ|ff|žŽg7}|VdS)zR Check that the local *port* is only reachable from local IPv4 addresses. ztcp://127.0.0.1:%dz 127.0.0.1ztcp://%s:%dztcp://[::1]:%dz tcp://[%s]:%dN)rZr$r[rWr%)r]rarýr}r_r.r.r5Úassert_can_connect_locally_4}s rbccsltƒs t‚||f}td|f|žŽtdtƒ|ff|žŽtd|f|žŽtdtƒ|ff|žŽg}|VdS)zK Check that the local *port* is reachable from all IPv6 addresses. ztcp://127.0.0.1:%dztcp://%s:%dztcp://[::1]:%dz tcp://[%s]:%dN)rWrYr[r$rZr%)r]rarýr}r_r.r.r5Ú$assert_can_connect_from_everywhere_6’s rcccs~tƒs t‚||f}td|f|žŽtdtƒ|ff|žŽtd|f|žŽg}tƒdkrt|tdtƒ|ff|žŽg7}|VdS)zR Check that the local *port* is only reachable from local IPv6 addresses. ztcp://127.0.0.1:%dztcp://%s:%dztcp://[::1]:%dz::1z tcp://[%s]:%dN)rWrYr[r$rZr%)r]rarýr}r_r.r.r5Úassert_can_connect_locally_6¢s rdc csœt|tƒrt |¡}|j}|jdd…}|dk r<|j}||_t ¡}t |¡g|jdd…<| |¡z |VWd||jdd…<| |¡|dk r–||_XdS)z*Capture output from the given Logger. N)r/rWr0rr1ZhandlersÚ propagateÚsixÚStringIOÚ StreamHandlerr)r4r1reZ orig_levelZ orig_handlersZorig_propagateZsior.r.r5Úcaptured_loggerµs riccs:t|tjƒst‚|j}t ¡|_z|jVWd||_XdS)z9Capture output from the given logging.StreamHandler. N)r/r0rhrYÚstreamrfrg)ZhandlerZorig_streamr.r.r5Úcaptured_handlerÌs rkc cstddlm}tjj}| ¡}z6| ¡| | ¡¡tj ||¡t|ƒdVWd| ¡| |¡t|ƒXdS)z6 Temporarily change configuration dictionary. r)ÚdefaultsN)rsrlrrrurHrtr)Ú new_configrlrsZorig_configr.r.r5rmÙs rmc cs>tj ¡}tj |¡z dVWdtj ¡tj |¡XdS)N)rzr{rurtrH)ZchangesZ saved_environr.r.r5Únew_environmentís rnc cs’ddl}tj d¡}tjdd\}}z\t |d¡}| | |¡¡WdQRX|tjd<z dVWd|rt|tjd<ntjd=XWdt |¡XdS)zH Temporarily change configuration file to match dictionary *c*. rNZDASK_CONFIGzdask-config)rHrF) Úyamlrzr{r|ÚtempfileZmkstempÚfdopenr;ÚdumpÚremove)r,roZold_fileÚfdrGÚfr.r.r5Únew_config_fileøs rvZtestscCs&tj t|¡}tj |¡s"t|ƒ‚|S)z; Get the path to one of the test TLS certificates. )rzrGr:Ú certs_dirÚexistsrY)ÚfilenamerGr.r.r5Úget_certsrzcCs2tdƒ}tdƒ}d|d|id|id|idœi}|S)z= A functional TLS configuration with our test certs. ztls-ca-cert.pemztls-key-cert.pemZtlsZcert)zca-filerïrÔrØ)rz)Úca_fileZkeycertr,r.r.r5Ú tls_configsr|cCstƒ}d|d<|S)zg A functional TLS configuration with our test certs, disallowing plain TCP communications. Tzrequire-encryption)r|)r,r.r.r5Útls_only_config4sr}c Cs ttƒƒtƒ}WdQRX|S)z: A Security object with proper TLS configuration. N)rmr|r )Úsecr.r.r5Útls_security>src Cs*ttƒƒtƒ}WdQRX|js&t‚|S)zg A Security object with proper TLS configuration and disallowing plain TCP communications. N)rmr}r Zrequire_encryptionrY)r~r.r.r5rôGs rôútls-cert.pemútls-key.pemútls-ca-cert.pemcCs<tjtjjt|ƒd}d|_tj|_| t|ƒt|ƒ¡|S)N)ÚcafileF) ÚsslÚcreate_default_contextÚPurposeZCLIENT_AUTHrzÚcheck_hostnameÚ CERT_REQUIREDÚverify_modeÚload_cert_chain)ÚcertfileÚkeyfiler{rgr.r.r5Úget_server_ssl_contextRs rcCs<tjtjjt|ƒd}d|_tj|_| t|ƒt|ƒ¡|S)N)rƒF) r„r…r†ZSERVER_AUTHrzr‡rˆr‰rŠ)r‹rŒr{rgr.r.r5Úget_client_ssl_context\s rŽc Cstt d¡}y0| |¡\}}||kr8| ||t||ƒf¡Wn4tk rn}zt d||f¡Wdd}~XYnXdS)NÚresourcez.rlimit too low (%s) and can't be increased: %s)ràZimportorskipZ getrlimitZ setrlimitÚmaxr¯r)ÚlimitZdesiredrZsoftZhardr^r.r.r5Úbump_rlimitfs r’cKs&| dddg¡tfdtƒdœ|—ŽS)Nrø)ztls://127.0.0.1r)ztls://127.0.0.1r‘ztls://127.0.0.1)rÔrñ)Ú setdefaultr7rô)r~r.r.r5Úgen_tls_clusterrsr”)r“)r“)r“)r)r“)r“)r“)T)r)NNN)rÄN)rÄN)rM)r`)NrS)NrS)NN)NNr\)NNr\)NN)NN)NN)r€rr‚)r€rr‚)ÎZ __future__rrrÚcollectionsÚ contextlibrruZdatetimerrrçrÚ itertoolsr0Zlogging.configrzrZrUrr=rPrEr¨rpr½rPrrr rþr„ÚImportErrorràrfrrZtoolzr r rZtornadorr Ztornado.genrZtornado.iolooprrïrrrZ compatibilityrrrrZ comm.utilsrrsrZcorerrrZmetricsZprocessrZ proctitlerrñr Zutilsr!r"r#r$r%r&r'r(r)r*rØr+r,r-Z dask.arrayrrªr4ÚrootZmanagerZ loggerDictr±rr-Zfixturer>rArBrÌrGr]rdrhrOrqrvrwrxrƒr†r‡r‰rŠrŒrrŽrr’r•r–r—r™ršrrŸr¡ÚdefaultdictÚintrÚcountrµÚobjectr¢r¶r¹r»rÃr?ZasyncincrYrÆrÍrÖrÝrßrèrërìrírîrðrõrör÷rérrrrrÔrÎr%r'r7r¯r9rArNrRrVrårWrXrrrZrr[r`rarbrcrdÚINFOrirkrmrnrvrGÚabspathr:ÚdirnameÚ__file__rwrzr|r}rrôrrŽr’r”r.r.r.r5Úsh 0 0 t #