B Zi@sdZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd d ZdddZddZGdddeZdddZdS)zkInfluence and Outlier Measures Created on Sun Jan 29 11:16:09 2012 Author: Josef Perktold License: BSD-3 )lzip) defaultdictN)OLS)cache_readonly) multipletests)maybe_unwrap_resultsbonf皙?FcCs@ddlm}|dkr$t|jjdd}t|dd}|dkrPt|}td|jj|j } |rt |  ddd} | | } |dk rt || }|jd} |jt | | d } t| ||d } t j| | | df}|dk r|dddf|k}||}ntd}|dk r`_ r rg?)rKrrrCr:rA)rGrLrVrWr.r.r/dffitsUszOLSInfluence.dffitscCsF|jj|j}|t|jdddf}|tt|jj}|S)z](cached attribute) dfbetas uses results from leave-one-observation-out loop N)r*paramsparams_not_obsirrCrUZdiagZnormalized_cov_params)rGdfbetasr.r.r/r\rszOLSInfluence.dfbetascCst|jdS)z(cached attribute) error variance for all LOOO regressions This is 'mse_resid' from each auxiliary regression. uses results from leave-one-observation-out loop rD)rr _res_looo)rGr.r.r/rU}szOLSInfluence.sigma2_not_obsicCst|jdS)z(cached attribute) parameter estimates for all LOOO regressions uses results from leave-one-observation-out loop rZ)rr r])rGr.r.r/r[szOLSInfluence.params_not_obsicCst|jdS)z(cached attribute) determinant of cov_params of all LOOO regressions uses results from leave-one-observation-out loop det_cov_params)rr r])rGr.r.r/det_cov_params_not_obsisz$OLSInfluence.det_cov_params_not_obsicCsP|j}|jd|j}||d|9}ddlm}|j||j|jj}||fS)zX(cached attribute) Cooks distance uses original results, no nobs loop rr r)r ) rKrTr:rr fr#r*r!)rGrLZcooks_d2r Zpvalsr.r.r/cooks_distances  zOLSInfluence.cooks_distancecCs|jtj|j}|S)z(cached attribute) covariance ratio between LOOO and original This uses determinant of the estimate of the parameter covariance from leave-one-out estimates. requires leave one out loop for observations )r_rlinalgdetr* cov_params)rG cov_ratior.r.r/res zOLSInfluence.cov_ratiocCs|jjd|jS)z(cached attribute) estimate of variance of the residuals :: sigma2 = sigma2_OLS * (1 - hii) where hii is the diagonal of the hat matrix r )r*rDrK)rGr.r.r/ resid_vars zOLSInfluence.resid_varcCs t|jS)z(cached attribute) estimate of standard deviation of the residuals See Also -------- resid_var )rrCrf)rGr.r.r/ resid_stds zOLSInfluence.resid_stdr7Tc Cs|dkr,|j}t||r ||S|jjj}nHy|j||Wntk rRYnX|j|}i}|jdd|f}|jjd}t ||k}|jdd|f}t || } |r| ||<| S)aregression results from LOVO auxiliary regression with cache The result instances are stored, which could use a large amount of memory if the datasets are large. There are too many combinations to store them all, except for small problems. Parameters ---------- drop_idx : int index of exog that is dropped from the regression endog_idx : 'endog' or int If 'endog', then the endogenous variable of the result instance is regressed on the exogenous variables, excluding the one at drop_idx. If endog_idx is an integer, then the exog with that index is regressed with OLS on all other exogenous variables. (The latter is the auxiliary regression for the variance inflation factor.) this needs more thought, memory versus speed not yet used in any other parts, not sufficiently tested r7Nr ) rFhasattrr*rr7rEKeyErrorr3r4rr<rr8) rGZdrop_idxZ endog_idxZstoreZstoredr=r:r-r>r9r.r.r/ _ols_xnotis&    zOLSInfluence._ols_xnotic Csddlm}|jjj}|j}||j}tt}xN|D]F\}}x<|D]4} | ||dd|f } ||  t | | qBWq4W|S)aoregress endog on exog without one of the variables This uses a k_vars loop, only attributes of the OLS instance are stored. Parameters ---------- attributes : list of strings These are the names of the attributes of the auxiliary OLS results instance that are stored and returned. not yet used r) LeaveOneOutN) #statsmodels.sandbox.tools.cross_valrkr*rr7r3r:rlistrBr8appendr) rGZ attributesrkr7r3cv_iterZres_looinidxoutidxZattres_ir.r.r/_get_drop_varis    zOLSInfluence._get_drop_varic Csddlm}dd}|j}|j}tj|jtjd}tj|jtjd}tj|jtjd}||j}xH|D]@\} } | || ||  } | j || <| j || <|| || <qfWt |||dS)a9collect required results from the LOOO loop all results will be attached. currently only 'params', 'mse_resid', 'det_cov_params' are stored regresses endog on exog dropping one observation at a time this uses a nobs loop, only attributes of the OLS instance are stored. r)rkcSstj|S)N)rrbrcrd)r9r.r.r/-sz(OLSInfluence._res_looo..)Zdtype)rZrDr^)rlrkr7r3rZzerosr4floatrArBr8rZrDdict) rGrkZget_det_cov_paramsr7r3rZrDr^rorprqrrr.r.r/r]!s    zOLSInfluence._res_loooc Cszddlm}|jjj}|j}dd|jD}|t|jd|j |j |j d|j |j dd|d}||j||d}||S)a Creates a DataFrame with all available influence results. Returns ------- frame : DataFrame A DataFrame with all results. Notes ----- The resultant DataFrame contains six variables in addition to the DFBETAS. These are: * cooks_d : Cook's Distance defined in `Influence.cooks_distance` * standard_resid : Standardized residuals defined in `Influence.resid_studentized_internal` * hat_diag : The diagonal of the projection, or hat, matrix defined in `Influence.hat_matrix_diag` * dffits_internal : DFFITS statistics using internally Studentized residuals defined in `Influence.dffits_internal` * dffits : DFFITS statistics using externally Studentized residuals defined in `Influence.dffits` * student_resid : Externally Studentized residuals defined in `Influence.resid_studentized_external` r)rcSsg|] }d|qS)Zdfb_r.).0ir.r.r/ _sz.OLSInfluence.summary_frame..)Zcooks_dZstandard_residZhat_diagrXrrY)r)rr)r%rr*rrr ZxnamesrvrarTrKrXrrYr\join)rGrrr Z beta_labelsZ summary_dataZdfbetar.r.r/ summary_frame@s   zOLSInfluence.summary_frame%6.3fc Csdt|jfd|jfd|jjfd|jdfd|jfd|jfd|j dfd |j fd |j dfg }t |\}}t |}||_dd lm}m}dd lm}dd lm}||} ||} dg|g|jdd| d<|||| | dS)acreate a summary table with all influence and outlier measures This does currently not distinguish between statistics that can be calculated from the original regression results and for which a leave-one-observation-out loop is needed Returns ------- res : SimpleTable instance SimpleTable instance with the results, can be printed Notes ----- This also attaches table_data to the instance. Zobsr7z fitted valuezCook's drzstudent. residualzhat diagzdffits internalzext.stud. residualrY) SimpleTabledefault_html_fmt)fmt_base)deepcopyz%4dr data_fmts)headerstxt_fmthtml_fmt)rr<rAr7r*r5rarTrKrXrrYrr6Z table_datastatsmodels.iolib.tabler}r~!statsmodels.iolib.tableformattingrcopyrr4) rGZ float_fmtZ table_rawcolnamesrr}r~rrfmtfmt_htmlr.r.r/ summary_tableqs(        zOLSInfluence.summary_table)N)r7T)r|)r __module__ __qualname____doc__rHrrKrMrNrOrQrTrrSrXrYr\rUr[r_rarerfrgrjrsr]r{rr.r.r.r/r@s2   !        4 1r@cCsddlm}ddlm}t|}t|j|j}|j |d|j }t |j |||j ||g}|||d}|\} } } t | | f} t|jd|j} t t|jd|jj|j ||dddf|dddf| dddf| dddf|j| |j|jdg }|}dd d d d d ddddddg }|}ddlm}m}ddlm}ddlm}||}||}dgdg|jdd|d<|||||d}|||fS)a Generate summary table of outlier and influence similar to SAS Parameters ---------- alpha : float significance level for confidence interval Returns ------- st : SimpleTable instance table with results that can be printed data : ndarray calculated measures and statistics for the table ss2 : list of strings column_names for table (Note: rows of table are observations) r)r )wls_prediction_stdg@)rr NZObszDep Var PopulationzPredicted ValuezStd Error Mean PredictzMean ci 95% lowzMean ci 95% uppzPredict ci 95% lowzPredict ci 95% uppZResidualzStd Error ResidualzStudent ResidualzCook's D)r}r~)r)rz%4dz%6.3fr)rrr)rr Z&statsmodels.sandbox.regression.predstdrr@rrCrKrDr"Zisfr!r6r5r<rArr7r+rTrarr}r~rrrrr4)r9rr rr)Zpredict_mean_seZtppfZpredict_mean_ciZtmpZ predict_seZpredict_ci_lowZpredict_ci_uppZ predict_ciZresid_seZtable_smrZss2rr}r~rrrrstr.r.r/rsJ       r)rr NFN)r1)r )rZstatsmodels.compat.pythonr collectionsrZnumpyrZ#statsmodels.regression.linear_modelrZstatsmodels.tools.decoratorsrZstatsmodels.stats.multitestrZstatsmodels.tools.toolsrr0r;r?objectr@rr.r.r.r/s        O %0s