p7]c@sdZddlmZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZd d eeed Zd dZdZdefdYZdefdYZdefdYZd dZdefdYZdS(skInfluence and Outlier Measures Created on Sun Jan 29 11:16:09 2012 Author: Josef Perktold License: BSD-3 i(t defaultdictN(tlzip(t_plot_influence_doc(tOLS(t multipletests(tcache_readonly(tmaybe_unwrap_resultstbonfg?cCsddlm}|dkr7t|jjdd}nt|dd}|dkrzt|}td|jj n|j } |rt j | j ddd} | | } |dk rt j|| }qn|jd} |jjt j | | d} t| d |d |} t j| | | df}|dk rs|dddf|k}||}n td}|dk rdd lm}||d d d|dgdt j||S|S(sU Outlier Tests for RegressionResults instances. Parameters ---------- model_results : RegressionResults Linear model results method : str - `bonferroni` : one-step correction - `sidak` : one-step correction - `holm-sidak` : - `holm` : - `simes-hochberg` : - `hommel` : - `fdr_bh` : Benjamini/Hochberg - `fdr_by` : Benjamini/Yekutieli See `statsmodels.stats.multitest.multipletests` for details. alpha : float familywise error rate labels : None or array_like If `labels` is not None, then it will be used as index to the returned pandas DataFrame. See also Returns below order : bool Whether or not to order the results by the absolute value of the studentized residuals. If labels are provided they will also be sorted. cutoff : None or float in [0, 1] If cutoff is not None, then the return only includes observations with multiple testing corrected p-values strictly below the cutoff. The returned array or dataframe can be empty if there are no outlier candidates at the specified cutoff. Returns ------- table : ndarray or DataFrame Returns either an ndarray or a DataFrame if labels is not None. Will attempt to get labels from model_results if available. The columns are the Studentized residuals, the unadjusted p-value, and the corrected p-value according to method. Notes ----- The unadjusted p-value is stats.t.sf(abs(resid), df) where df = df_resid - 1. i(tstatst row_labelst get_influences=model_results object %s does not have a get_influence method.Niitalphatmethod(t DataFrametcolumnst student_residtunadj_ps(p)tindex(tscipyRtNonetgetattrtmodeltdataRtAttributeErrort __class__t__name__tresid_studentized_externaltnptabstargsorttasarraytdf_residtttsfRtc_tslicetpandasR (t model_resultsR R tlabelstordertcutoffRtinfltresultstresidtidxtdfRtadj_pRtmaskR ((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt outlier_tests8.    "   "     icCs|d}|jjjd}tj|j|ddddf}tj|jj|f}t|jj|j }tj |d|jd|}|j |S(sRamsey's RESET specification test for linear models This is a general specification test, for additional non-linear effects in a model. Notes ----- The test fits an auxiliary OLS regression where the design matrix, exog, is augmented by powers 2 to degree of the fitted values. Then it performs an F-test whether these additional terms are significant. If the p-value of the f-test is below a threshold, e.g. 0.1, then this indicates that there might be additional non-linear effects in the model and that the linear model is mis-specified. References ---------- http://en.wikipedia.org/wiki/Ramsey_RESET_test iNi( RtexogtshapeRtvandert fittedvaluest column_stackRtendogtfitteyetf_test(trestdegreeR'tk_varsty_fitted_vanderR1tres_auxtr_matrix((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt reset_ramseyis + cCsx|jd}|dd|f}tj||k}|dd|f}t||jj}dd|}|S(s~variance inflation factor, VIF, for one exogenous variable The variance inflation factor is a measure for the increase of the variance of the parameter estimates if an additional variable, given by exog_idx is added to the linear regression. It is a measure for multicollinearity of the design matrix, exog. One recommendation is that if VIF is greater than 5, then the explanatory variable given by exog_idx is highly collinear with the other explanatory variables, and the parameter estimates will have large standard errors because of this. Parameters ---------- exog : ndarray design matrix with all explanatory variables, as for example used in regression exog_idx : int index of the exogenous variable in the columns of exog Returns ------- vif : float variance inflation factor Notes ----- This function does not save the auxiliary regression. See Also -------- xxx : class for regression diagnostics TODO: doesn't exist yet References ---------- http://en.wikipedia.org/wiki/Variance_inflation_factor iNg?(R2RtarangeRR7trsquared(R1texog_idxR<tx_iR/tx_notit r_squared_itvif((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytvariance_inflation_factors' t_BaseInfluenceMixincBsleZdZd ddddd dZejidd6e_d d d dZdd d d d d ZRS( s=common methods between OLSInfluence and MLE/GLMInfluence g?tcooksi0g?c Ksz|dkr-t|do'd|jk}nddlm}||j|d|d|d|d|d |d ||} | S( Nt_cachetres_loooi(t_influence_plottexternalR t criteriontsizet plot_alphatax(RthasattrRKt$statsmodels.graphics.regressionplotsRMR*( tselfRNR RORPRQRRtkwargsRMR:((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytplot_influences ! ttextra_params_doccKsmddlm}|j|\}}|dkr:d}nt|j} tj| } |j| |||dkrtj | tj } ntj ||k} dtj | } |j j jj} | dkrtj| } n|jtj| d| t| |t| | d|}idd 6d d 6}|j|||jd ||j|||S( Ni(tutilss Index Plottalliitlargeitfontsizetblacktcolort Observation(tstatsmodels.graphicsRZt create_mpl_axRtlenR6RRAtscattertonestbool_RR*RRR t annotate_axestwhereRt set_ylabelt set_xlabelt set_title(RUtytylabelt thresholdttitleRRtkwdsRZtfigtnobsRt large_pointstpsizeR&tfont((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt _plot_indexs.      c KsN|}|dkrd}n|dkrZ|jdd|f}d|jjj|} n|jdr|jd}d} n|jds|jd r|j}d } nt|jdr|jd}d} nO|jd r|j}d } n.t ||}|dk r||}n|} |j || d |d|d||} | S(s9index plot for influence attributes Parameters ---------- y_var : string Name of attribute or shortcut for predefined attributes that will be plotted on the y-axis. threshold : None or float Threshold for adding annotation with observation labels. Observations for which the absolute value of the y_var is larger than the threshold will be annotated. Set to a negative number to label all observations or to a large number to have no annotation. title : string If provided, the title will replace the default "Index Plot" title. ax : matplolib axis instance The plot will be added to the `ax` if provided, otherwise a new figure is created. idx : None or integer Some attributes require an additional index to select the y-var. In dfbetas this refers to the column indes. kwds : optional keywords Keywords will be used in the call to matplotlib scatter function. R[tdfbetaNs DFBETA for tcookisCook's distancethattlevers!Leverage (diagonal of hat matrix)R+s Internally Studentized ResidualsRnRoRR( RtdfbetasR*Rt exog_namest startswithtcooks_distancethat_matrix_diagtresid_studentizedRRv( RUty_varRnRoRRR,RpRORlRmRq((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt plot_indexs2              N( Rt __module__t__doc__RRWRtformatRvR(((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRIs   t MLEInfluencecBseZdZd d d d d d dZedZedZedZedZ edZ edZ edZ ed Z ed Zd ZRS( s: Local Influence and outlier measures (experimental) This currently subclasses GLMInfluence instead of the other way. No common superclass yet. This is another version before checking what is common Parameters ---------- results : instance of results class This only works for model and results classes that have the necessary helper methods. other arguments are only to override default behavior and are used instead of the corresponding attribute of the results class. By default resid_pearson is used as resid. Attributes ---------- hat_matrix_diag (hii) : This is the generalized leverage computed as the local derivative of fittedvalues (predicted mean) with respect to the observed response for each observation. d_params : Change in parameters computed with one Newton step using the full Hessian corrected by division by (1 - hii). dbetas : change in parameters divided by the standard error of parameters from the full model results, ``bse``. cooks_distance : quadratic form for change in parameters weighted by ``cov_params`` from the full model divided by the number of variables. It includes p-values based on the F-distribution which are only approximate outside of linear Gaussian models. resid_studentized : In the general MLE case resid_studentized are computed from the score residuals scaled by hessian factor and leverage. This does not use ``cov_params``. d_fittedvalues : local change of expected mean given the change in the parameters as computed in ``d_params``. d_fittedvalues_scaled : same as d_fittedvalues but scaled by the standard errors of a predicted mean of the response. params_one : is the one step parameter estimate computed as ``params`` from the full sample minus ``d_params``. Notes ----- MLEInfluence produces the same results as GLMInfluence (verified for GLM Binomial and Gaussian). There will be some differences for non-canonical links or if a robust cov_type is used. Warning: This does currently not work for constrained or penalized models, e.g. models estimated with fit_constrained or fit_regularized. This has not yet been tested for correctness when offset or exposure are used, although they should be supported by the code. status: experimental, This class will need changes to support different kinds of models, e.g. extra parameters in discrete.NegativeBinomial or two-part models like ZeroInflatedPoisson. cCs4t||_}|jjj\|_|_|dk r@|n |jj|_|dk ra|n |jj|_|dk r|n|j |_ |dk r|n|j |_ |dk r|n |j |_ |jj |_|jjj|jj|_|jjj|jj|_|dk r0||_ndS(N(RR*RR1R2RrR<RR6t resid_pearsonR+tscalet cov_paramsRt model_classthessiantparamst score_obst_hat_matrix_diag(RUR*R+R6R1RRR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt__init__fs!! cCs|t|dr|jS|jjj|jj}|jjj|jj}|tjj |j |j j j d}|S(sxDiagonal of the generalized leverage This is the analogue of the hat matrix diagonal for general MLE. Ri( RSRR*Rt_deriv_mean_dparamsRt_deriv_score_obs_dendogRtlinalgtsolveRtTtsum(RUtdmu_dptdsdyth((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRzs ,cCsV|jjd|j}tjj|j|jj}|d|jdddfS(sChange in parameter estimates This uses one-step approximation of the parameter change to deleting one observation. iiN( RRRRRRRRR(RUtso_notitbeta_i((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytd_paramsscCs|j|jj}|S(sScaled change in parameter estimates The one-step change of parameters in d_params is rescaled by dividing by the standard error of the parameter estimate given by results.bse. (RR*tbse(RUR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR{scCs|jj|jS(sParameter estimate based on one-step approximation This the one step parameter estimate computed as ``params`` from the full sample minus ``d_params``. (R*RR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt params_onescCsy|jtjj|j|jjjjd}||j:}ddlm }|j j ||j|j j }||fS(sCook's distance and p-values Based on one step approximation d_params and on results.cov_params Cook's distance divides by the number of explanatory variables. p-values are based on the F-distribution which are only approximate outside of linear Gaussian models. Warning: The definition of p-values might change if we switch to using chi-square distribution instead of F-distribution, or if we make it dependent on the fit keyword use_t. ii(R(RRRRRRRR<RRtfR!R*R(RUtcooks_d2Rtpvals((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR~s  !cCs[|jjj|jj}|jjj|jj}|tj|tjd|jS(sScore residual divided by sqrt of hessian factor experimental, agrees with GLMInfluence for Binomial and Gaussian. no reference for this i(R*Rt score_factorRthessian_factorRtsqrtR(RUR!thf((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRscCs |jjS(N(R*tget_prediction(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt_get_predictionscCs>tj|jj}|jjj|}||jjdS(s-Change in expected response, fittedvalues Local change of expected mean given the change in the parameters as computed in d_params. Notes ----- This uses one-step approximation of the parameter change to deleting one observation ``d_params``. i(RRR*RRRRR(RURtderiv((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytd_fittedvaluess cCs|j|jjS(s Change in fittedvalues scaled by standard errors This uses one-step approximation of the parameter change to deleting one observation ``d_params``, and divides by the standard errors for the predicted mean provided by results.get_prediction. (RRtse_mean(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytd_fittedvalues_scaleds c Csddlm}|jjj}|j}g|jD]}d|^q2}|td|jdd|j d|j d|j d |}||j d |d |}|j |S( s Creates a DataFrame with influence results. Returns ------- frame : pandas DataFrame A DataFrame with selected results for each observation. The index will be the same as provided to the model. Notes ----- The resultant DataFrame contains six variables in addition to the ``dfbetas``. These are: * cooks_d : Cook's Distance defined in ``cooks_distance`` * standard_resid : Standardized residuals defined in `resid_studentizedl` * hat_diag : The diagonal of the projection, or hat, matrix defined in `hat_matrix_diag` * dffits_internal : DFFITS statistics using internally Studentized residuals defined in `d_fittedvalues_scaled` i(R tdfb_tcooks_ditstandard_residthat_diagtdffits_internalRR(R$R R*RRR txnamestdictR~RRRR{tjoin(RUR RR tit beta_labelst summary_dataRw((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt summary_frames         N(RRRRRRRRR{RR~RRRtpropertyRR(((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR(s<     t OLSInfluencecBs|eZdZdZedZedZedZedZedZ edZ edZ ed Z dd Zed Zed Zed ZedZedZedZedZedZedZedZedZdedZdZedZdZddZ RS(s\class to calculate outlier and influence measures for OLS result Parameters ---------- results : RegressionResults currently assumes the results are from an OLS regression Notes ----- One part of the results can be calculated without any auxiliary regression (some of which have the `_internal` postfix in the name. Other statistics require leave-one-observation-out (LOOO) auxiliary regression, and will be slower (mainly results with `_external` postfix in the name). The auxiliary LOOO regression only the required results are stored. Using the LOO measures is currently only recommended if the data set is not too large. One possible approach for LOOO measures would be to identify possible problem observations with the _internal measures, and then run the leave-one-observation-out only with observations that are possible outliers. (However, this is not yet available in an automized way.) This should be extended to general least squares. The leave-one-variable-out (LOVO) auxiliary regression are currently not used. cCst||_|jjj\|_|_|jj|_|jj|_|j|_|jj |_ |j |_ i|_ i|_dS(N(RR*RR1R2RrR<R6R+RRt mse_residRtaux_regression_exogtaux_regression_endog(RUR*((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR>s   cCs |j|jjjjjdS(sDiagonal of the hat_matrix for OLS Notes ----- temporarily calculated here, this should go to model class i(R1R*Rt pinv_wexogRR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRMscCs|j}|jd|S(sPRESS residuals i(RR+(RUthii((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt resid_pressWs cCs|j}|j|d|S(sInfluence measure matches the influence measure that gretl reports u * h / (1 - h) where u are the residuals and h is the diagonal of the hat_matrix i(RR+(RUR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt influence^s cCs|j}|d|S(sFactor of diagonal of hat_matrix used in influence this might be useful for internal reuse h / (1 - h) i(R(RUR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pythat_diag_factoris cCstj|j|jS(s0Error sum of squares of PRESS residuals (RtdotR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt ess_presssscCs|jS(sStudentized residuals using variance from OLS alias for resid_studentized_internal for compatibility with MLEInfluence this uses sigma from original estimate and does not require leave one out loop (tresid_studentized_internal(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRyscCs|jddS(sStudentized residuals using variance from OLS this uses sigma from original estimate does not require leave one out loop tsigmaN(tget_resid_studentized_externalR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRscCs"tj|j}|jd|S(sStudentized residuals using LOOO variance this uses sigma from leave-one-out estimates requires leave one out loop for observations R(RRtsigma2_not_obsiR(RUt sigma_looo((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRscCsL|j}|dkr0|j}tj|}n|j|tjd|S(scalculate studentized residuals Parameters ---------- sigma : None or float estimate of the standard deviation of the residuals. If None, then the estimate from the regression results is used. Returns ------- stzd_resid : ndarray studentized residuals Notes ----- studentized residuals are defined as :: resid / sigma / np.sqrt(1 - hii) where resid are the residuals from the regression, sigma is an estimate of the standard deviation of the residuals, and hii is the diagonal of the hat_matrix. iN(RRRRRR+(RURRt sigma2_est((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRs    cCsj|j}|jd|j}||d|9}ddlm}|jj||j|jj}||fS(sECooks distance uses original results, no nobs loop iii(R( RRR<RRRR!R*R(RURRRR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR~s  !cCsR|j}|jtj|d|}dtj|jd|j}||fS(sdffits measure for influence of an observation based on resid_studentized_internal uses original results, no nobs loop iig?(RRRRR<Rr(RURtdffits_tdffits_threshold((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRs !cCsR|j}|jtj|d|}dtj|jd|j}||fS(sdffits measure for influence of an observation based on resid_studentized_external, uses results from leave-one-observation-out loop It is recommended that observations with dffits large than a threshold of 2 sqrt{k / n} where k is the number of parameters, should be investigated. Returns ------- dffits: float dffits_threshold : float References ---------- `Wikipedia `_ iig?(RRRRR<Rr(RURRR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytdffitss !cCs_|jj|j}|tj|jdddf:}|tjtj|jj:}|S(sJdfbetas uses results from leave-one-observation-out loop N( R*Rtparams_not_obsiRRRRtdiagtnormalized_cov_params(RUR{((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR{s&"cCs|jj|j}|S(sJdfbetas uses results from leave-one-observation-out loop (R*RR(RURw((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRwscCstj|jdS(serror variance for all LOOO regressions This is 'mse_resid' from each auxiliary regression. uses results from leave-one-observation-out loop R(RRt _res_looo(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRscCstj|jdS(soparameter estimates for all LOOO regressions uses results from leave-one-observation-out loop R(RRR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRscCstj|jdS(stdeterminant of cov_params of all LOOO regressions uses results from leave-one-observation-out loop tdet_cov_params(RRR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pytdet_cov_params_not_obsi scCs&|jtjj|jj}|S(scovariance ratio between LOOO and original This uses determinant of the estimate of the parameter covariance from leave-one-out estimates. requires leave one out loop for observations (RRRtdetR*R(RUt cov_ratio((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyR(s cCs|jd|jS(sestimate of variance of the residuals :: sigma2 = sigma2_OLS * (1 - hii) where hii is the diagonal of the hat matrix i(RR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt resid_var6s cCstj|jS(snestimate of standard deviation of the residuals See Also -------- resid_var (RRR(RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt resid_stdDs R6c Cs|dkr>|j}t||r,||S|jjj}nSy|j||Wntk rdnX|j|}i}|jdd|f}|jjd}t j ||k}|jdd|f}t ||j } |r| ||cCs|jj|jS(sexperimental code (R*RR (RU((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyt_diff_fittedvalues_onesc CsJddlm}d}|jjj}|jjj}|jjj}|jd}|jd}|jd}} |jd} } |jdd} t |d d rt } nt } t j |jd t j}t j |jd t j}t j |jd t j}||j}x|D]\}}|dk rM||} n| dk rf| |} n| dk r| ||d/RXt freq_weightst var_weightstoffsettexposuretn_trialstfamilyt initializeRt start_paramsR tnewtonRRRRN(RRR*RR6R1t_get_init_kwdstpopRRSRtFalseRRR2RRrRR(tnR7RRRR(RURRR6R1t init_kwdsR#R$R%toffset_R&t exposure_R't is_binomialRRRRRRtmod_iR((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRsN           (RRRRRRRR~RRRR R!R(((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyRys)  (Rt collectionsRtnumpyRtstatsmodels.compat.pythonRt)statsmodels.graphics._regressionplots_docRt#statsmodels.regression.linear_modelRtstatsmodels.stats.multitestRtstatsmodels.tools.decoratorsRtstatsmodels.tools.toolsRRR.R0R@RHtobjectRIRRRR(((sClib/python2.7/site-packages/statsmodels/stats/outliers_influence.pyts&  Q $ 0k O