ó áp7]c@s¼dZddlmZddlmZddlmZddlZd„Z d„Z d„Z e d „Z e d „Zdde d „Zdd „Zd d„Zd„Zd d„Zdefd„ƒYZd„Zdefd„ƒYZdefd„ƒYZedkr¸eƒZejƒejƒejƒddlj Z!dddgdZ"dddgZ#dd gZ$ee#e$ƒZ%e&e%ƒej'd!ƒZ(ej'd!ƒd"Z)ej'd!ƒd#d#Z*de"krôee)ƒ\Z+Z,ee*ƒ\Z-Z.nde"kr3ej/e)d$d%d&ƒZ)ej/e*d$d%d&ƒZ*ne)j0d&Z1ee)ƒ\Z+Z,ee*ƒ\Z-Z.ee+e-d'd ƒZ2ee+e-d'd(ƒZ3ee+e-d'd)ƒZ4e&ej5e2j6e2ƒej7e2j8d&ƒƒkj9ƒƒd*d+gdZ:d,d-gd&Z;e;ej<j=e1ƒZ>e:ej'dd.ƒZ?e2e?j8dƒe>Z@e3e?j8dƒe>ZAe4e?j8dƒe>ZBe!jCe@e2ƒjDƒZEe!jCe@e4ƒjDƒZFeEjGZHeFjGZIe!jCeBe2ƒjDƒZJe!jCeBe4ƒjDƒZKeJjGZLeKjGZMejNjOe4e2d/dƒd&ZPejNjOe2e4d/dƒd&ZQe&ej5eQeMƒeLƒe&ej5ePeHƒeIƒee2e4ƒZRe&ejSejTe2eRjUe4ƒƒƒƒe&ejSejTe4eRjVe2ƒƒƒƒe&ejSejTeMeRjWeLƒƒƒƒejSejTeIeRjWeHƒƒƒee#e$ƒ\ZXZYZZZ[Z\Z]e&d0ƒe&d1ƒe&eZƒe&eJj^eYƒj_ƒe&d2ƒe&d3ƒe&e\ƒe&eJj^e[ƒj_ƒeJj`e[ƒejaejbd5ƒejcd#ƒfƒZdeFjed4ZfeEj^eRjUedƒƒj_Zge&ejSejTefegƒƒƒeeBe)e*ƒZhe&ehjiƒj_ƒe&ehjiƒjjƒe&ehjkƒjlƒe&ehjkƒjjƒe&ehjmdƒd&j_ƒe&ehjmd#ƒd&j_ƒe&ehjnƒƒndS(6sòfunctions to work with contrasts for multiple tests contrast matrices for comparing all pairs, all levels to reference level, ... extension to 2-way groups in progress TwoWay: class for bringing two-way analysis together and try out various helper functions Idea for second part - get all transformation matrices to move in between different full rank parameterizations - standardize to one parameterization to get all interesting effects. - multivariate normal distribution - exploit or expand what we have in LikelihoodResults, cov_params, f_test, t_test, example: resols_dropf_full.cov_params(C2) - connect to new multiple comparison for contrast matrices, based on multivariate normal or t distribution (Hothorn, Bretz, Westfall) iÿÿÿÿ(tprint_function(t assert_equal(tzipNcCsxg}xbt|ƒD]T}xKt|d|ƒD]6}tj|ƒ}d||>> x = np.array(['F', 'F', 'M', 'M', 'F', 'F', 'M', 'M', 'F', 'F', 'M', 'M'], dtype='|S1') >>> dummy_1d(x, varname='gender') (array([[1, 0], [1, 0], [0, 1], [0, 1], [1, 0], [1, 0], [0, 1], [0, 1], [1, 0], [1, 0], [0, 1], [0, 1]]), ['gender_F', 'gender_M']) islevel_%dNs_%s( RRtmaxRtarangetastypetinttuniqueR(RtvarnameR R&t grouplabels((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyR+¬s . -<'tfullcCs¶|dkr^|dd…dd…df|dd…ddd…fj|jddƒ}nT|dkrt|dd…dd…f|dd…dd…fƒ}tjtj|jdtƒ|dd…dd…f|dd…dd…f|fƒ}n°|dkr¦t|dd…dd…f|dd…dd…fƒ}tjtj|jdtƒ|dd…dd…f|dd…dd…f|fƒ}n tdƒ‚|S( sŸdummy variable from product of two dummy variables Parameters ---------- d1, d2 : ndarray two dummy variables, assumes full set for methods 'drop-last' and 'drop-first' method : {'full', 'drop-last', 'drop-first'} 'full' returns the full product, encoding of intersection of categories. The drop methods provide a difference dummy encoding: (constant, main effects, interaction effects). The first or last columns of the dummy variable (i.e. levels) are dropped to get full rank dummy matrix. Returns ------- dummy : ndarray dummy variable for product, see method RJNiiÿÿÿÿs drop-lasts drop-firstismethod not recognized( RtreshapetshapeR,RRRRFt ValueError(R>R@tmethodR6td12rltd12r((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyR,ãs R ;] ;] c Cs |j\}}tjtj|ddƒdkƒ\}}tjtj|ddƒdkƒ\}}tj|ƒ}tjdg|f|kjƒ s¼tj||dgf|kjƒ rËtdƒ‚ntjdg|df}tj|d|gf} || fS(s¬start and endpoints of groups in a sorted dummy variable array helper function for nested categories Examples -------- >>> d1 = np.array([[1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 1, 0], [0, 1, 0], [0, 1, 0], [0, 0, 1], [0, 0, 1], [0, 0, 1], [0, 0, 1]]) >>> dummy_limits(d1) (array([0, 4, 8]), array([ 4, 8, 12])) get group slices from an array >>> [np.arange(d1.shape[0])[b:e] for b,e in zip(*dummy_limits(d1))] [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8, 9, 10, 11])] >>> [np.arange(d1.shape[0])[b:e] for b,e in zip(*dummy_limits(d1))] [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8, 9, 10, 11])] taxisiiiÿÿÿÿsdummy variable is not sorted(RLRtnonzerotdiffRDR)tallRM( tdtnobstnvarststart1tcol1tend1tcol1_tcctstarttend((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt dummy_limitss**#'cCsË|dkr|St|ƒ\}}t|ƒ\}}tj||ƒ}tj||ƒ}||k} || } || } |dkrt|dd…dd…f|dd…dd…fƒ} tjtj|jdtƒ|dd…dd…f|dd…| ffƒ} n§|dkr²t|dd…dd…f|dd…dd…fƒ}tjtj|jdtƒ|dd…dd…f|dd…| ffƒ} n tdƒ‚| | | fS( sÙunfinished and incomplete mainly copy past dummy_product dummy variable from product of two dummy variables Parameters ---------- d1, d2 : ndarray two dummy variables, d2 is assumed to be nested in d1 Assumes full set for methods 'drop-last' and 'drop-first'. method : {'full', 'drop-last', 'drop-first'} 'full' returns the full product, which in this case is d2. The drop methods provide an effects encoding: (constant, main effects, subgroup effects). The first or last columns of the dummy variable (i.e. levels) are dropped to get full rank encoding. Returns ------- dummy : ndarray dummy variable for product, see method RJs drop-lastNiÿÿÿÿis drop-firstismethod not recognized( R_Rtin1dR,RRRLRFRM(R>R@RNRXRZtstart2tend2tfirsttlasttequalt col_dropft col_droplROR6RP((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt dummy_nested4s"     ;T ;T tDummyTransformcBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(sConversion between full rank dummy encodings y = X b + u b = C a a = C^{-1} b y = X C a + u define Z = X C, then y = Z a + u contrasts: R_b b = r R_a a = R_b C a = r where R_a = R_b C Here C is the transform matrix, with dot_left and dot_right as the main methods, and the same for the inverse transform matrix, C^{-1} Note: - The class was mainly written to keep left and right straight. - No checking is done. - not sure yet if method names make sense cCsHtjj||ddƒd|_tjj||ddƒd|_dS(s\C such that d1 C = d2, with d1 = X, d2 = Z should be (x, z) in arguments ? trcondiÿÿÿÿiN(Rtlinalgtlstsqt transf_matrixtinvtransf_matrix(tselfR>R@((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt__init__„s"cCstj|j|ƒS(s b = C a (RtdotRm(Rota((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pytdot_leftŒscCstj||jƒS(s z = x C (RRqRm(RoR((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt dot_right‘scCstj|j|ƒS(s a = C^{-1} b (RRqRn(Rotb((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt inv_dot_left–scCstj||jƒS(s x = z C^{-1} (RRqRn(Rotz((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt inv_dot_right›s(t__name__t __module__t__doc__RpRsRtRvRx(((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyRics     cCsqtj|ƒ}|jd}tdƒgdg|dtdƒg}|d||jdƒd|jdƒS(sngroupmeans using dummy variables Parameters ---------- x : array_like, ndim data array, tested for 1,2 and 3 dimensions d : ndarray, 1d dummy variable, needs to have the same length as x in axis 0. Returns ------- groupmeans : ndarray, ndim-1 means for each group along axis 0, the levels of the groups are the last axis Notes ----- This will be memory intensive if there are many levels in the categorical variable, i.e. many columns in the dummy variable. In this case it is recommended to use a more efficient version. ii.igð?N(.N(RtasarraytndimRRtsum(RRURWtsli((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt groupmean_d¤s +tTwoWaycBsGeZdZdd„Zd„Zd„Zd„Zd„Zd„Z RS(sýa wrapper class for two way anova type of analysis with OLS currently mainly to bring things together Notes ----- unclear: adding multiple test might assume block design or orthogonality This estimates the full dummy version with OLS. The drop first dummy representation can be recovered through the transform method. TODO: add more methods, tests, pairwise, multiple, marginal effects try out what can be added for userfriendly access. missing: ANOVA table cCs¾|jd|_|dkr+d}d}n |\}}t||ƒ\|_|_\}}t||ƒ\|_|_\} } |jd|_} | jd|_ } t || ƒ} | \}}}}}}| \|_ |_ |_ |_|_}t|| ddƒ}t|| ddƒ}t||ƒ|_|jd|_||_tj||ƒjƒ|_|jj|_|jj|jƒ|_d| d| d|_|j|j|_dS(NiRrRuiRNRJs drop-first(RLRVRR+R>t d1_labelsR@t d2_labelstnlevel1tnlevel2RBt prod_labeltC1tC1_labeltC2tC2_labelR,Rit transformRWtexogtsmtOLStfittresolstparamsRvt params_dropftstart_interactiont n_interaction(Rotendogtfactor1tfactor2tvarnamestvname1tvname2R>R‚R@RƒR„R…trestprodlabR‡tC1labR‰tC2labR?tdp_fulltdp_dropf((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyRpÛs,   %%' cCs\|j}tjtj||j|fƒtj|ƒfƒ}|jj|ƒ}||_|S(s7contrast/restriction matrix for no interaction ( R”RthstackRRWRR‹RxtR_nointer_transf(Rotniat R_nointerR¢((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt r_nointerýs  4 cCsn|j}tjtj||j|fƒtj|ƒfƒ}|jj|ƒ}||_|j j |ƒ}|S(s1ttests for no-interaction terms are zero ( R”RR¡RRWRR‹RxR¢Rtt_test(RoR£R¤R¢tt_res((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pytttest_interactions  4 cCs|jƒ}|jj|ƒS(s1ttests for no-interaction terms are zero (R¥Rtf_test(RoR¢((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pytftest_interactions cCsH|dkr(|jj|jƒ|jfS|jj|jƒ|jfSdS(Ni(RR¦R‡RˆR‰RŠ(Rot factorind((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pytttest_conditional_effects cCsxddlm}|jj|j|jƒ}|j}|j}d}tddg|jƒ}||||d|d|ƒS(Niÿÿÿÿ(t SimpleTables!Estimated Coefficients by factorst data_fmtss%#10.4gttitlettxt_fmt( tstatsmodels.iolibR­R‘RKR„R…R‚Rƒtdict(RoR­t params_arrtstubstheadersR¯t table_fmt((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt summary_coeffs  N( RyRzR{RRpR¥R¨RªR¬R·(((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyRÇs "  tTestContrastToolscBs,eZd„Zd„Zd„Zd„ZRS(cCsÇdddg|_ddg|_tjdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgg ƒ|_dS(Nta0ta1ta2tb0tb1ii(tv1nametv2nameRRR>(Ro((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyRp0s          cCs÷tjddddddddddddg ddƒ}tjddgddgddgddgddgddgddgddgddgddgddgddgg ƒddg}}t|d d ƒ\}}t||ƒt||ƒdS( NtFtMtdtypes|S1iitgender_Ftgender_MRHtgender(RRR+R(RoRRUR&tres_dt res_labels((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyt test_dummy_1d@s"0            c Csbt|j|jƒ}dgd}ddddddg|dRRR(RoRute((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyttest_dummy_limitscs(RyRzRpRÈRÔRÖ(((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyR¸.s   t__main__tsmalltlargeiR¹RºR»R¼R½i iiiRQiRNs drop-lasts drop-firstgð?g{®Gáz„?gü©ñÒMbP?gš™™™™™¹?iRjs" tvalues for no effect of factor 1s/each test is conditional on a level of factor 2s" tvalues for no effect of factor 2s/each test is conditional on a level of factor 1iþÿÿÿ(ii(oR{t __future__Rt numpy.testingRtstatsmodels.compat.pythonRtnumpyRR RRtFalseRR'RRBR+R,R_RhtobjectRiR€RR¸RytttRÔRÈRÖtstatsmodels.apitapiRtexamplesR¾R¿RÑtprintRDtytx1tx2R>R‚R@RƒtrepeatRLRVtdd_fulltdd_dropltdd_dropfRqtTtdiagR~RTt effect_sizet noise_scaletrandomtrandntnoisetbetat ydata_fullt ydata_droplt ydata_dropfRŽRtresols_full_fulltresols_full_dropfR‘t params_f_ft params_f_dftresols_dropf_fulltresols_dropf_dropft params_df_ft params_df_dfRkRlttr_ofttr_fot transf_f_dfRCtabsRxRtRvRœR‡RR‰RžR?R¦ttvaluet cov_paramsR¡RRtR_nointttvaluest inter_directt inter_transfttwR¨tpvalueRªtfvalueR¬R·(((sGlib/python2.7/site-packages/statsmodels/sandbox/stats/contrast_tools.pyts¼     B 7 % , /A #g=          7    ))!       '  î