ó áp7]c@sèddlmZddlmZmZmZddlmZddlm Z ddl Z e d„Z e d„Zdd „Zd „Zd „Ze e e d „Zdd „Zid„Zdefd„ƒYZdefd„ƒYZdS(iÿÿÿÿ(tRegularizedResults(t_calc_nodewise_rowt_calc_nodewise_weightt_calc_approx_inv_cov(tLikelihoodModelResults(tOLSNcCs/|dkrtddƒ‚n|j|jS(sÁestimates the regularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized Returns ------- An array of the paramters for the regularized fit s!_est_regularized_naive currently s#requires that fit_kwds not be None.N(tNonet ValueErrortfit_regularizedtparams(tmodtpnumt partitionstfit_kwds((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_est_regularized_naiveKs  cCs/|dkrtddƒ‚n|j|jS(s¬estimates the unregularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit Returns ------- An array of the parameters for the fit s#_est_unregularized_naive currently s#requires that fit_kwds not be None.N(RRtfitR (R R R R ((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_est_unregularized_naivees  icCsmt|dƒ}t|ƒ}tj|ƒ}x|D]}||7}q2W||:}d|tj|ƒ|k<|S(s joins the results from each run of _est__naive and returns the mean estimate of the coefficients Parameters ---------- params_l : list A list of arrays of coefficients. threshold : scalar The threshold at which the coefficients will be cut. i(tlentnptzerostabs(tparams_lt thresholdtpR t params_mnR ((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt _join_naives    cCs2|jtj|ƒ| }||d|7}|S(scalculates the log-likelihood gradient for the debiasing Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array-like The estimated coefficients for the current partition. alpha : scalar or array-like The penalty weight. If a scalar, the same penalty weight applies to all variables in the model. If a vector, it must have the same length as `params`, and contains a penalty weight for each coefficient. L1_wt : scalar The fraction of the penalty given to the L1 penalty term. Must be between 0 and 1 (inclusive). If 0, the fit is a ridge fit, if 1 it is a lasso fit. score_kwds : dict-like or None Keyword arguments for the score function. Returns ------- An array-like object of the same dimension as params Notes ----- In general: gradient l_k(params) where k corresponds to the index of the partition For OLS: X^T(y - X^T params) i(tscoreRtasarray(R R talphatL1_wtt score_kwdstgrad((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt _calc_grad˜s&cCs?tj|jtj|ƒ|ƒ}|dd…df|jS(súcalculates the weighted design matrix necessary to generate the approximate inverse covariance matrix Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array-like The estimated coefficients for the current partition. hess_kwds : dict-like or None Keyword arguments for the hessian function. Returns ------- An array-like object, updated design matrix, same dimension as mod.exog N(Rtsqrtthessian_factorRRtexog(R R t hess_kwdstrhess((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_calc_wdesign_matÃs$cCss|dkrin|}|dkr*in|}|dkrOtddƒ‚n |d}d|krr|d}nd}|jj\}} ttjd| |ƒƒ} |j|j} t || |||ƒ|} t || |ƒ} g}g}xmt || t |d| | ƒƒD]G}t | ||ƒ}|j|ƒt| |||ƒ}|j|ƒqW| | ||fS(s‚estimates the regularized fitted parameters, is the default estimation_method for class DistributedModel. Parameters ---------- mod : statsmodels model class instance The model for the current partition. mnum : scalar Index of current partition. partitions : scalar Total number of partitions. fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized score_kwds : dict-like or None Keyword arguments for the score function. hess_kwds : dict-like or None Keyword arguments for the Hessian function. Returns ------- A tuple of parameters for regularized fit An array-like object of the fitted parameters, params An array-like object for the gradient A list of array like objects for nodewise_row A list of array like objects for nodewise_weight s$_est_regularized_debiased currently s#requires that fit_kwds not be None.RRigð?N(RRR#tshapetintRtceilRR R R&trangetminRtappendR(R tmnumR R RR$RRtnobsRtp_partR Rtwexogtnodewise_row_ltnodewise_weight_ltidxt nodewise_rowtnodewise_weight((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_est_regularized_debiasedÚs.     +   c Cst|ddƒ}t|ƒ}tj|ƒ}tj|ƒ}g}g}xL|D]D}||d7}||d7}|j|dƒ|j|dƒqQWtj|ƒ}tj|ƒ}||:}|d|9}t||ƒ} || j|ƒ} d| tj| ƒ|k<| S(s†joins the results from each run of _est_regularized_debiased and returns the debiased estimate of the coefficients Parameters ---------- results_l : list A list of tuples each one containing the params, grad, nodewise_row and nodewise_weight values for each partition. threshold : scalar The threshold at which the coefficients will be cut. iiiigð¿(RRRtextendtarrayRtdotR( t results_lRRR Rtgrad_mnR1R2trtapprox_inv_covtdebiased_params((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_join_debiaseds&    c CsY|jjƒ}|j|ƒ|j|||}|j|||jd||j}|S(sßhandles the model fitting for each machine. NOTE: this is primarily handled outside of DistributedModel because joblib can't handle class methods. Parameters ---------- self : DistributedModel class instance An instance of DistributedModel. pnum : scalar index of current partition. endog : array-like endogenous data for current partition. exog : array-like exogenous data for current partition. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_e : dict-like Additional init_kwds to add for each partition. Returns ------- estimation_method result. For the default, _est_regularized_debiased, a tuple. R (t init_kwdstcopytupdatet model_classtestimation_methodR testimation_kwds( tselfR tendogR#R t init_kwds_ettemp_init_kwdstmodeltresults((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt_helper_fit_partitionHs  tDistributedModelc Bs\eZdZddddddddd„Zddddd„Zdd„Zdd„ZRS(sÇ Distributed model class Parameters ---------- partitions : scalar The number of partitions that the data will be split into. model_class : statsmodels model class The model class which will be used for estimation. If None this defaults to OLS. init_kwds : dict-like or None Keywords needed for initializing the model, in addition to endog and exog. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS estimation_method : function or None The method that performs the estimation for each partition. If None this defaults to _est_regularized_debiased. estimation_kwds : dict-like or None Keywords to be passed to estimation_method. join_method : function or None The method used to recombine the results from each partition. If None this defaults to _join_debiased. join_kwds : dict-like or None Keywords to be passed to join_method. results_class : results class or None The class of results that should be returned. If None this defaults to RegularizedResults. results_kwds : dict-like or None Keywords to be passed to results class. Attributes ---------- partitions : scalar See Parameters. model_class : statsmodels model class See Parameters. init_kwds : dict-like See Parameters. init_kwds_generator : generator or None See Parameters. estimation_method : function See Parameters. estimation_kwds : dict-like See Parameters. join_method : function See Parameters. join_kwds : dict-like See Parameters. results_class : results class See Parameters. results_kwds : dict-like See Parameters. Examples -------- Notes ----- c Cs||_|dkr!t|_n ||_|dkrBi|_n ||_|dkrct|_n ||_|dkr„i|_n ||_|dkr¥t|_ n ||_ |dkrÆi|_ n ||_ |dkrçt |_ n ||_ | dkri|_ n | |_ dS(N(R RRRCR@R6RDRER?t join_methodt join_kwdsRt results_classt results_kwds( RFR RCR@RDRERNRORPRQ((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyt__init__­s2                        t sequentialc Cs¹|dkri}n|dkr9|j|||ƒ}n7|dkr`|j||||ƒ}ntd|ƒ‚|j||j}|jdgdg|j}|j|||j S(sePerforms the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like or None Keywords needed for the model fitting. parallel_method : str type of distributed estimation to be used, currently "sequential", "joblib" and "dask" are supported. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. RStjoblibs.parallel_method: %s is currently not supportediN( Rtfit_sequentialt fit_joblibRRNRORCR@RPRQ( RFtdata_generatorR tparallel_methodtparallel_backendtinit_kwds_generatorR:R tres_mod((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRÜs         c CsÁg}|dkr]x¨t|ƒD]7\}\}}t|||||ƒ}|j|ƒqWn`tt||ƒƒ} xH| D]@\}\\}}} t|||||| ƒ}|j|ƒqyW|S(s*Sequentially performs the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. N(Rt enumerateRLR,tzip( RFRWR RZR:R RGR#RKttup_genRH((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRUs     c ssddlm}|tˆjƒ\}‰}|dkrn|dkrn|‡‡‡fd†t|ƒDƒƒ}n|dk r¾|dkr¾|-|‡‡‡fd†t|ƒDƒƒ}WdQXn±|dkr|dk rtt||ƒƒ} |‡‡‡fd†| Dƒƒ}n_|dk ro|dk rott||ƒƒ} |'|‡‡‡fd†| Dƒƒ}WdQXn|S(s©Performs the distributed estimation in parallel using joblib Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. iÿÿÿÿ(t parallel_funcc3s3|])\}\}}ˆˆ|||ˆƒVqdS(N((t.0R RGR#(tfR RF(sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pys csc3s3|])\}\}}ˆˆ|||ˆƒVqdS(N((R`R RGR#(RaR RF(sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pys isNc3s<|]2\}\\}}}ˆˆ|||ˆ|ƒVqdS(N((R`R RGR#R@(RaR RF(sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pys osc3s<|]2\}\\}}}ˆˆ|||ˆ|ƒVqdS(N((R`R RGR#R@(RaR RF(sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pys vs(tstatsmodels.tools.parallelR_RLR RR\R]( RFRWR RYRZR_tpartn_jobsR:R^((RaR RFsFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRVDs&N(t__name__t __module__t__doc__RRRRRURV(((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRMms> ,9 /tDistributedResultscBs eZdZd„Zd„ZRS(sS Class to contain model results Parameters ---------- model : class instance class instance for model used for distributed data, this particular instance uses fake data and is really only to allow use of methods like predict. params : array parameter estimates from the fit model. cCstt|ƒj||ƒdS(N(tsuperRhRR(RFRJR ((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRRscOs|jj|j|||ŽS(shCalls self.model.predict for the provided exog. See Results.predict. Parameters ---------- exog : array-like NOT optional The values for which we want to predict, unlike standard predict this is NOT optional since the data in self.model is fake. args, kwargs : Some models can take additional arguments or keywords, see the predict method of the model for the details. Returns ------- prediction : ndarray, pandas.Series or pandas.DataFrame See self.model.predict (RJtpredictR (RFR#targstkwargs((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRjs(ReRfRgRRRj(((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyRh}s (tstatsmodels.base.elastic_netRt(statsmodels.stats.regularized_covarianceRRRtstatsmodels.base.modelRt#statsmodels.regression.linear_modelRtnumpyRRRRRR R&R6R?RLtobjectRMRh(((sFlib/python2.7/site-packages/statsmodels/base/distributed_estimation.pyts  E    + @ . $ÿ