ó ‡ˆ\c@s3dZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZddlmZddlmZmZedk rAdd lmZdd lmZdd lmZdd lmZdd lmZdd l mZ!ddl"m#Z#m$Z$ndee ƒfd„ƒYZ%de%fd„ƒYZ&de'fd„ƒYZ(de'fd„ƒYZ)de(e%fd„ƒYZ*de(e)e%fd„ƒYZ+de)e%fd„ƒYZ,de'fd„ƒYZ-de'fd „ƒYZ.d!e/fd"„ƒYZ0dS(#s, Backends for embarrassingly parallel code. iÿÿÿÿN(tABCMetatabstractmethodi(t format_exc(tWorkerInterrupttTransportableException(tmp(twith_metaclasstPY27(t delete_folder(tMemmappingPool(t ThreadPool(tget_memmapping_executor(t TimeoutError(tprocess_executort cpu_counttParallelBackendBasecBsàeZdZeZdZdd„ZdddddgZed„ƒZ edd „ƒZ d dddd „Z d „Z d „Zd„Zd„Zd„Zd„Zed„Zd„Zejd„ƒZed d„ƒZRS(sEHelper abc which defines all methods a ParallelBackend must implementicCs ||_dS(N(t nesting_level(tselfR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt__init__$stOMP_NUM_THREADStOPENBLAS_NUM_THREADStMKL_NUM_THREADStVECLIB_MAXIMUM_THREADStNUMEXPR_NUM_THREADScCsdS(såDetermine the number of jobs that can actually run in parallel n_jobs is the number of workers requested by the callers. Passing n_jobs=-1 means requesting all available workers for instance matching the number of CPU cores on the worker host(s). This method should return a guesstimate of the number of workers that can actually perform work concurrently. The primary use case is to make it possible for the caller to know in how many chunks to slice the work. In general working on larger data chunks is more efficient (less scheduling overhead and better use of CPU cache prefetching heuristics) as long as all the workers have enough work to do. N((Rtn_jobs((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyteffective_n_jobs,scCsdS(sSchedule a func to be runN((Rtfunctcallback((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt apply_async>sicKs||_|j|ƒS(sØReconfigure the backend and return the number of workers. This makes it possible to reuse an existing backend instance for successive independent calls to Parallel with different parameters. (tparallelR(RRRtprefertrequiret backend_args((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt configureBs cCsdS(s;Call-back method called at the beginning of a Parallel callN((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt start_callLscCsdS(s5Call-back method called at the end of a Parallel callN((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt stop_callOscCsdS(s0Shutdown the workers and free the shared memory.N((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt terminateRscCsdS(s Determine the optimal batch sizei((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytcompute_batch_sizeUscCsdS(s1Callback indicate how long it took to run a batchN((Rt batch_sizetduration((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytbatch_completedYscCsgS(s'List of exception types to be captured.((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytget_exceptions\scCsdS(svAbort any running tasks This is called when an exception has been raised when executing a tasks and all the remaining tasks will be ignored and can therefore be aborted to spare computation resources. If ensure_ready is True, the backend should be left in an operating state as future tasks might be re-submitted via that same backend instance. If ensure_ready is False, the implementer of this method can decide to leave the backend in a closed / terminated state as no new task are expected to be submitted to this backend. Setting ensure_ready to False is an optimization that can be leveraged when aborting tasks via killing processes from a local process pool managed by the backend it-self: if we expect no new tasks, there is no point in re-creating new workers. N((Rt ensure_ready((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytabort_everything`scCsLt|ddƒd}|dkr5td|ƒdfStd|ƒdfSdS(söBackend instance to be used by nested Parallel calls. By default a thread-based backend is used for the first level of nesting. Beyond, switch to sequential backend to avoid spawning too many threads on the host. RiiN(tgetattrtSequentialBackendtNonetThreadingBackend(RR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytget_nested_backendxs ccs dVdS(sœContext manager to manage an execution context. Calls to Parallel.retrieve will be made inside this context. By default, this does nothing. It may be useful for subclasses to handle nested parallelism. In particular, it may be required to avoid deadlocks if a backend manages a fixed number of workers, when those workers may be asked to do nested Parallel calls. Without 'retrieval_context' this could lead to deadlock, as all the workers managed by the backend may be "busy" waiting for the nested parallel calls to finish, but the backend has no free workers to execute those tasks. N((R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytretrieval_context…scCsOxH|jD]=}tjj|dƒ}|dkr t|ƒtj|RAt uses_threadstsupports_sharedmemRR.RR0(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR-¤s   tPoolManagerMixincBsGeZdZdZd„Zd„Zd„Zdd„Ze d„Z RS(s,A helper class for managing pool of workers.cCsd|dkrtdƒ‚nEtdks3|dkr7dS|dkr`ttƒd|dƒ}n|S(s?Determine the number of jobs which are going to run in parallelis&n_jobs == 0 in Parallel has no meaningiN(RERR.tmaxR(RR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRÉs  cCs9|jdk r5|jjƒ|jjƒd|_ndS(s#Shutdown the process or thread poolN(t_poolR.tcloseR$(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR$Õs  cCs|jS(s>Used by apply_async to make it possible to implement lazy init(RM(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt _get_poolÜscCs|jƒjt|ƒd|ƒS(sSchedule a func to be runR(RORt SafeFunction(RRR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRàs cCs?|jƒ|r;|jd|jjd|j|jjndS(s@Shutdown the pool and restart a new one with the same parametersRRN(R$R!RRt _backend_args(RR*((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR+ås N( R<R=R>R.RMRR$RORRAR+(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRKÄs   tAutoBatchingMixincBsJeZdZdZdZdZdZd„Zd„Zd„Z d„Z RS( s/A helper class for automagically batching jobs.gš™™™™™É?iigcCs|j|_|j|_dS(N(t_DEFAULT_EFFECTIVE_BATCH_SIZEt_effective_batch_sizet _DEFAULT_SMOOTHED_BATCH_DURATIONt_smoothed_batch_duration(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRþs cCs|j}|j}|dkr‘||jkr‘t||j|ƒ}td|dƒ}||_|jjdkrö|jjd||fƒqöne||jkrð|dkrð|d}||_|jjdkrö|jjd||fƒqön|}||kr|j |_n|S(s Determine the optimal batch sizeiiii s:Batch computation too fast (%.4fs.) Setting batch_size=%d.s:Batch computation too slow (%.4fs.) Setting batch_size=%d.( RTRVtMIN_IDEAL_BATCH_DURATIONtintRLRtverboset_printtMAX_IDEAL_BATCH_DURATIONRU(Rtold_batch_sizetbatch_durationtideal_batch_sizeR&((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR%s2            cCsR||jkrN|j}||jkr0|}nd|d|}||_ndS(s1Callback indicate how long it took to run a batchgš™™™™™é?gš™™™™™É?N(RTRVRU(RR&R't old_durationt new_duration((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR(1s   cCs|j|_|j|_dS(sgReset batch statistics to default values. This avoids interferences with future jobs. N(RSRTRURV(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pytreset_batch_statsAs ( R<R=R>RWR[RSRURR%R(Ra(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRRís  / R/cBs8eZdZeZeZeZddd„Zd„Z RS(sŽA ParallelBackend which will use a thread pool to execute batches in. This is a low-overhead backend but it suffers from the Python Global Interpreter Lock if the called function relies a lot on Python objects. Mostly useful when the execution bottleneck is a compiled extension that explicitly releases the GIL (for instance a Cython loop wrapped in a "with nogil" block or an expensive call to a library such as NumPy). The actual thread pool is lazily initialized: the actual thread pool construction is delayed to the first call to apply_async. ThreadingBackend is used as the default backend for nested calls. icKsL|j|ƒ}|dkr6ttd|jƒƒ‚n||_||_|S(s?Build a process or thread pool and return the number of workersiR(RtFallbackToBackendR-RRt_n_jobs(RRRR ((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR!]s   cCs+|jdkr$t|jƒ|_n|jS(s“Lazily initialize the thread pool The actual pool of worker threads is only initialized at the first call to apply_async. N(RMR.R Rc(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyROhsN( R<R=R>RAR@RIRJR.R!RO(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR/Js   tMultiprocessingBackendcBsAeZdZdZeZd„Zddddd„Zd„Z RS(sA ParallelBackend which will use a multiprocessing.Pool. Will introduce some communication and memory overhead when exchanging input and output data with the with the worker Python processes. However, does not suffer from the Python Global Interpreter Lock. t__JOBLIB_SPAWNED_PARALLEL__cCsÎtdkrdStjƒjrE|dkrAtjdddƒndStjdkrz|dkrvtjdddƒndStt j ƒt j ƒs¸|dkr´tjdddƒndSt t |ƒj|ƒS( s¢Determine the number of jobs which are going to run in parallel. This also checks if we are attempting to create a nested parallel loop. isHMultiprocessing-backed parallel loops cannot be nested, setting n_jobs=1t stackleveliisTMultiprocessing-backed parallel loops cannot be nested, below loky, setting n_jobs=1sVMultiprocessing-backed parallel loops cannot be nested below threads, setting n_jobs=1N(RR.tcurrent_processtdaemontwarningstwarnR t_CURRENT_DEPTHt isinstancet threadingtcurrent_threadt _MainThreadtsuperRdR(RR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRs*       icKs«|j|ƒ}|dkr6ttd|jƒƒ‚nttjj|jdƒƒ}|rit dƒ‚ndtj|jRsRAR@RR.R!R$(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRdss  &t LokyBackendcBsheZdZeZdd d d dd„Zd„Zd d„Ze d d„ƒZ d„Z ed„Z RS( s>Managing pool of workers with loky instead of multiprocessing.ii,cKsd|j|ƒ}|dkr6ttd|jƒƒ‚nt|d|d|j||_||_|S(s9Build a process executor and return the number of workersiRttimeoutRr(RRbR-RR R;t_workersR(RRRRRtidle_worker_timeouttmemmappingexecutor_args((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR!Ñs    cCs×|dkrtdƒ‚n¸tdks3|dkr7dStjƒjrl|dkrhtjdddƒndSttj ƒtj ƒsª|dkr¦tjdddƒndS|dkrÓt t ƒd|dƒ}n|S( s?Determine the number of jobs which are going to run in parallelis&n_jobs == 0 in Parallel has no meaningisRLoky-backed parallel loops cannot be called in a multiprocessing, setting n_jobs=1RfisKLoky-backed parallel loops cannot be nested below threads, setting n_jobs=1N( RERR.RgRhRiRjRlRmRnRoRLR(RR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRàs&      cCsP|jjt|ƒƒ}tj|j|ƒ|_|dk rL|j|ƒn|S(sSchedule a func to be runN( R{tsubmitRPt functoolstpartialtwrap_future_resultR5R.tadd_done_callback(RRRtfuture((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRüs  cCs5y|jd|ƒSWntk r0tƒ‚nXdS(skWrapper for Future.result to implement the same behaviour as AsyncResults.get from multiprocessing.RzN(RGtLokyTimeoutErrorR (RƒRz((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRs cCs9|jdk r+t|jjƒd|_n|jƒdS(N(R{R.Rt _temp_folderRa(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR$ s cCsX|jjdtƒt|jjƒd|_|rT|jd|jjd|jƒndS(sLShutdown the workers and restart a new one with the same parameters t kill_workersRRN( R{tshutdownRARR…R.R!RR(RR*((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR+s  N( R<R=R>RAR@R.R!RRt staticmethodRR$R+(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRyÌs     RFcBseZd„Zd„ZRS(cCs|ƒ|_dS(N(tresults(Rtbatch((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR"scCs|jS(N(R‰(R((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR5's(R<R=RR5(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRF!s RPcBs eZdZd„Zd„ZRS(sWrapper that handles the serialization of exception tracebacks. If an exception is triggered when calling the inner function, a copy of the full traceback is captured to make it possible to serialize it so that it can be rendered in a different Python process. cCs ||_dS(N(R(RR((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyR2scOs“y|j||ŽSWnxtk r0tƒ‚n_tk rŽtrˆtjƒ\}}}t|||ddddƒ}t||ƒ‚q‚nXdS(Ntcontexti t tb_offseti( RtKeyboardInterruptRt BaseExceptionRtsystexc_infoRR(Rtargstkwargste_typete_valuete_tbttext((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyt__call__5s    (R<R=R>RR—(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRP+s RbcBseZdZd„ZRS(s<Raised when configuration should fallback to another backendcCs ||_dS(N(tbackend(RR˜((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRMs(R<R=R>R(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyRbJs(1R>RuR3RRiRmRRBtabcRRt format_stackRt my_exceptionsRRt_multiprocessing_helpersRt_compatRRR.tdiskRtpoolR tmultiprocessing.poolR texecutorR tmultiprocessingR texternals.loky._baseR„texternals.lokyR RRR-tobjectRKRRR/RdRyRFRPt ExceptionRb(((sJlib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyts>        † )]) XU