p7]c @s{dZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZdd d ed Zd efd YZdefdYZeZeje_defdYZeZeje_dedZdd eedZddeed dZdedZdZedZ dedZ!defdYZ"e"Z#e#j$je#_dZ%ddZ&ddZ'dZ(d Z)d!Z*dd"d#d$Z+d%Z,d d&Z-d'Z.e/d(krwd)gZ0d)e0krej1j2d*Z3e4ee3d+e4ee3d,Z5d,Z6ej7e5Z8xWe9e5d-D]EZ:d.ej1j2e6j;Z3ee3d/d0d d1dd e8e:e8Z?x5d3d4d5d6gD]!Z@e4e@e?eAe5e@q$We4d7e4d8e4d9e4d:e4d;eBgd:jCd<D]ZDeeEeDjCd=^qZFeBgd;jCd<D]ZDeeEeDjCd=^qZGd>d?e6d@e6d/ZHe4eHee3d/d0d d1ddAd-\ZIZJZJZKe4e8eHkj<e4eKjLjMjNdBe4e3dBe4ej=e8dCejO dDdEdFdGdHejOge4e?e5ejPd3d4d5d6dIgjQeAnd,Z6ejRe6d/fZ3ejSe6dJe3ddd-f| | _q>n|}t||ddd|dfj}|j}|j}||j}tjj||}|r|| _|| _||||| fS||||fSdS(sLagrange Multiplier tests for autocorrelation This is a generic Lagrange Multiplier test for autocorrelation. I don't have a reference for it, but it returns Engle's ARCH test if x is the squared residual array. A variation on it with additional exogenous variables is the Breusch-Godfrey autocorrelation test. Parameters ---------- resid : ndarray, (nobs,) residuals from an estimation, or time series maxlag : int highest lag to use autolag : None or string If None, then a fixed number of lags given by maxlag is used. store : bool If true then the intermediate results are also returned Returns ------- lm : float Lagrange multiplier test statistic lmpval : float p-value for Lagrange multiplier test fval : float fstatistic for F test, alternative version of the same test based on F test for the parameter restriction fpval : float pvalue for F test resstore : instance (optional) a class instance that holds intermediate results. Only returned if store=True See Also -------- het_arch acorr_breusch_godfrey acorr_ljung_box ig(@gY@ig@Nttrimtbothtaiccss$|]\}}|j|fVqdS(N(Rc(t.0tktv((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pys tstbiccss$|]\}}|j|fVqdS(N(Rg(RdReRf((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pys vss(autolag can only be None, 'AIC' or 'BIC'(RCRRPRRQRUtceiltpowertdiffRtc_tonesRtrangeRR#tlowerRSRRtresultstfvaluetf_pvaluetrsquaredRRXR*tresolstusedlag(RR R R RR6txdifftxdalltxshorttresstoreRotmlagtbestict icbestlagticbestRtRstfvaltfpvaltlmtlmpval((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pytacorr_lm/sP*   0% "  7%% % " /     c Cs&t|dd|d|d|d|S(sEngle's Test for Autoregressive Conditional Heteroscedasticity (ARCH) Parameters ---------- resid : ndarray residuals from an estimation, or time series maxlag : int highest lag to use autolag : None or string If None, then a fixed number of lags given by maxlag is used. store : bool If true then the intermediate results are also returned ddof : int Not Implemented Yet If the residuals are from a regression, or ARMA estimation, then there are recommendations to correct the degrees of freedom by the number of parameters that have been estimated, for example ddof=p+a for an ARMA(p,q) (need reference, based on discussion on R finance mailinglist) Returns ------- lm : float Lagrange multiplier test statistic lmpval : float p-value for Lagrange multiplier test fval : float fstatistic for F test, alternative version of the same test based on F test for the parameter restriction fpval : float pvalue for F test resstore : instance (optional) a class instance that holds intermediate results. Only returned if store=True Notes ----- verified agains R:FinTS::ArchTest iR R R R(R(R$R R R Rtddof((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pythet_archs*cCstj|j}|jj}|jd}|dkrmtjdtj|ddd}t |}ntj tj ||f}t |dddf|dd}|jd}tj tj|df|f}|| }tj||f}|jd} |rt} nt||j} | jtj|| | |} | j} | j}tj| d } tj|d }|| j}tjj||}|r| | _|| _||| || fS||| |fSdS( sBreusch Godfrey Lagrange Multiplier tests for residual autocorrelation Parameters ---------- results : Result instance Estimation results for which the residuals are tested for serial correlation nlags : int Number of lags to include in the auxiliary regression. (nlags is highest lag) store : bool If store is true, then an additional class instance that contains intermediate results is returned. Returns ------- lm : float Lagrange multiplier test statistic lmpval : float p-value for Lagrange multiplier test fval : float fstatistic for F test, alternative version of the same test based on F test for the parameter restriction fpval : float pvalue for F test resstore : instance (optional) a class instance that holds intermediate results. Only returned if store=True Notes ----- BG adds lags of residual to exog in the design matrix for the auxiliary regression with residuals as endog, see Greene 12.7.1. References ---------- Greene Econometrics, 5th edition ig(@gY@ig@NRaRb(((RRPR$RR RRQttruncRiRUt concatenatetzerosRRkRlRERRR#tf_testteyeRpR1tsqueezeRrRRXR*RsRt(RoRNR Rtexog_oldR6RvRwR tk_varsRxRstftR}R~RR((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pytacorr_breusch_godfreys8*   '% "   "     c Cstj|}tj|d}|j\}}t||j}|j}|j}||j} | tj j | |d||fS(sBreusch-Pagan Lagrange Multiplier test for heteroscedasticity The tests the hypothesis that the residual variance does not depend on the variables in x in the form :math: \sigma_i = \sigma * f(\alpha_0 + \alpha z_i) Homoscedasticity implies that $\alpha=0$ Parameters ---------- resid : array-like For the Breusch-Pagan test, this should be the residual of a regression. If an array is given in exog, then the residuals are calculated by the an OLS regression or resid on exog. In this case resid should contain the dependent variable. Exog can be the same as x. TODO: I dropped the exog option, should I add it back? exog_het : array_like This contains variables that might create data dependent heteroscedasticity. Returns ------- lm : float lagrange multiplier statistic lm_pvalue :float p-value of lagrange multiplier test fvalue : float f-statistic of the hypothesis that the error variance does not depend on x f_pvalue : float p-value for the f-statistic Notes ----- Assumes x contains constant (for counting dof and calculation of R^2). In the general description of LM test, Greene mentions that this test exaggerates the significance of results in small or moderately large samples. In this case the F-statistic is preferrable. *Verification* Chisquare test statistic is exactly (<1e-13) the same result as bptest in R-stats with defaults (studentize=True). Implementation This is calculated using the generic formula for LM test using $R^2$ (Greene, section 17.6) and not with the explicit formula (Greene, section 11.4.3). The degrees of freedom for the p-value assume x is full rank. References ---------- http://en.wikipedia.org/wiki/Breusch%E2%80%93Pagan_test Greene 5th edition Breusch, Pagan article ii( RRPRRR#RpRqRrRRXR*( R$texog_hetRRLR6tnvarsRsR}R~R((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pythet_breuschpagans=   cCs@tj|}tj|}|jdkr<tdn|j\}}tj|\}}|dd|f|dd|f}|j\}} | ||dd|kstt|d|j} | j } | j } || j } | j tj j|dksttjj| | j }| || | fS(sJWhite's Lagrange Multiplier Test for Heteroscedasticity Parameters ---------- resid : array_like residuals, square of it is used as endogenous variable exog : array_like possible explanatory variables for variance, squares and interaction terms are included in the auxilliary regression. resstore : instance (optional) a class instance that holds intermediate results. Only returned if store=True Returns ------- lm : float lagrange multiplier statistic lm_pvalue :float p-value of lagrange multiplier test fvalue : float f-statistic of the hypothesis that the error variance does not depend on x. This is an alternative test variant not the original LM test. f_pvalue : float p-value for the f-statistic Notes ----- assumes x contains constant (for counting dof) question: does f-statistic make sense? constant ? References ---------- Greene section 11.4.1 5th edition p. 222 now test statistic reproduces Greene 5th, example 11.3 is5x should have constant and at least one more variableNg@i(RRPtndimRRt triu_indicestAssertionErrorRR#RpRqRrtdf_modeltlinalgt matrix_rankRRXR*(R$R tretresRRLR6tnvars0ti0ti1RRsR}R~RR((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pyt het_white\s &*"   %cCstj|}tj|}|j\}}|dkrF|d}n+d|krq|dkrqt||}ntj|dd|f}||}||ddf}t|| || j}t||||j} |j| j} | dkr-t j j | |j | j } d} n.d| } t j j | | j |j } d} |rt } d| _| | _| | _| j |j f| _|| _| | _| | _|| _d | | | f| _| S| | fSdS( sUtest whether variance is the same in 2 subsamples Parameters ---------- y : array_like endogenous variable x : array_like exogenous variable, regressors idx : integer column index of variable according to which observations are sorted for the split split : None or integer or float in intervall (0,1) index at which sample is split. If 0tNii( RRRQRRERRR#RRRrRXR*( R$R tfuncRtexog_auxR6RtlstftestRtlm_pval((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pyt linear_lms"  1* cCstj|}tj|}|jdkr<tdntj|jd\}}tj|dd|f|dd|fdd}d}d}|||jd}tjj |dd} tj | j tj |k} |ddtj | df}||} | tj| } tj|j| } |tj|d d}|| dddf9}|jj|}| jtjj|| }|jd}tjj||}|||fS( s6 White's Two-Moment Specification Test Parameters ---------- resid : array_like OLS residuals exog : array_like OLS design matrix Returns ------- stat : float test statistic pval : float chi-square p-value for test statistic dof : int degrees of freedom Notes ----- Implements the two-moment specification test described by White's Theorem 2 (1980, p. 823) which compares the standard OLS covariance estimator with White's heteroscedasticity-consistent estimator. The test statistic is shown to be chi-square distributed. Null hypothesis is homoscedastic and correctly specified. Assumes the OLS design matrix contains an intercept term and at least one variable. The intercept is removed to calculate the test statistic. Interaction terms (squares and crosses of OLS regressors) are added to the design matrix to calculate the test statistic. Degrees-of-freedom (full rank) = nvar + nvar * (nvar + 1) / 2 Linearly dependent columns are removed to avoid singular matrix error. Reference --------- White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroscedasticity. Econometrica, 48: 817-838. See also het_white. is2X should have a constant and at least one variableNig+=gvIh%<=tmodetrtaxis(RRPRRRRtdeletetvarRtqrR+tdiagonalR(twheretmeanR%R&RQtsolveRRXR*(R$R RteRRtatoltrtolttolRtmasktsqetsqmndevsRtdevxtbtstattdofR?((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pyt spec_whites,/9$$  c Cs|jd}ttd|ddd}||j}tj||j}xt|D]z}tj|dd||f|ddd||fj}d||d}|tj|||j}q^Wtjjtj|j|} tj| tj|| } | S( s Did not run yet from regstats2 :: if idx(29) % HAC (Newey West) L = round(4*(nobs/100)^(2/9)); % L = nobs^.25; % as an alternative hhat = repmat(residuals',p,1).*X'; xuux = hhat*hhat'; for l = 1:L; za = hhat(:,(l+1):nobs)*hhat(:,1:nobs-l)'; w = 1 - l/(L+1); xuux = xuux + w*(za+za'); end d = struct; d.covb = xtxi*xuux*xtxi; iigY@ig"@Nig?( RRUtroundR&RR%RmRtinv( R$RR6RNthhattxuuxtlagtzatwtxtxitcovbNW((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pyt _neweywestcov0s " E!!cCs|jj}|jj}|j\}}tjtj||f}tjtj|}tjtj|}tjtj|} || } || } tj| j| } tj| j| } tj j | | }|||d'gio,gfffff&)R itbinsg g\(\ g333333gp= ףgp= ף?g333333?g4@g)\(?g?i Rg$@RR(aRBt __future__Rtstatsmodels.compat.pythonRRRtnumpyRRRt#statsmodels.regression.linear_modelRtstatsmodels.tsa.stattoolsRRtstatsmodels.tsa.tsatoolsRRQRRtobjectRRt compare_coxRDt compare_jR`RRRRRRRthet_goldfeldquandtR@RRRRRRRR!R(R)RtexamplestrandomtrandnRRtnreplR6RtmcresRmtiiRWRt histogramtsortt mcressorttratioRUtdictRRRtcrvdgtcrvdtcrit_5lags0p05tadfstatt_RxRsRR tinfRRRlRRtrandRLRtfptfoRCtresgqR#RR$t cov_paramstHC0_setcov_HC0(((sClib/python2.7/site-packages/statsmodels/sandbox/stats/diagnostic.pyts W  O  Le  -R H ;Y{  " 1 O " @ 1 Y    '     ::  -56#EEE    E