p7]c3@sdZddlmZmZmZddlmZddlZddl m Z ddl m ZddZ dZd dd Zd Zd Zd ZddZddZdddZddZddZddZddddZddddZedkrdGHdGHdddddgZddd d!d"gZxeD]Z ee GHqSWxeD]Z ee GHqoWeeGHeeGHd#d$d%d&d'd(dd)d*d+d,d-d.g Z!e j"d/e j#d0e j$d1ej%d2dd3Z&e j'e&ee&e j"d/e j#d4e j$d1ej%d2dd5Z&e j'e&eeee&de&ej(d2d2d6d7gdd8dd8dd8gdd9dd8dd:ggZ)e)j*d2Z+e)j*dZ,ee+Z-ee,Z.ee)Z/ee+e,e)Z0ee,e+e)Z1edej2ej3e+e,Z4edej2ej3e,e+Z5ee+e,e)Z6d;GHe-e.e/e0e1e4e5e6fGHd<GHej(d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKd9dLdMdNdOdLdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_dQd`dadbdcdddedfdgdhdidjdkg2ZeeZ7HdlGHdmGHdnGHdoGHdpGHdqe+GHdrGHde+dZ8dse8GHee8de8gZ9dte9GHdue9e8ej:de0fGHdvGHdwe8e-dej:d7fGHdxGHdyGHej(dzdzd6gdd{dd{dd{gdd:dd{dd|ggZ;e;GHd}GHd~ej<d2ee;dee;dgdej:d7GHdGHej(dddgdddgdddggZ=ndS(s: Information Theoretic and Entropy Measures References ---------- Golan, As. 2008. "Information and Entropy Econometrics -- A Review and Synthesis." Foundations And Trends in Econometrics 2(1-2), 1-145. Golan, A., Judge, G., and Miller, D. 1996. Maximum Entropy Econometrics. Wiley & Sons, Chichester. i(trangetlziptlmap(tstatsN(tpyplot(t logsumexpcCs|dkrt|Stj|}t|j}d||<|jd|}tjtj||j |j d|}||}|S(s- Compute the log of the sum of exponentials log(e^{a_1}+...e^{a_n}) of a Avoids numerical overflow. Parameters ---------- a : array-like The vector to exponentiate and sum axis : int, optional The axis along which to apply the operation. Defaults is None. Returns ------- sum(log(exp(a))) Notes ----- This function was taken from the mailing list http://mail.scipy.org/pipermail/scipy-user/2009-October/022931.html This should be superceded by the ufunc when it is finished. itaxisN( tNonet sp_logsumexptnptasarraytlisttshapetmaxtlogtexptreshapetsum(taRtshpta_maxtstlse((s;lib/python2.7/site-packages/statsmodels/sandbox/infotheo.pyR-s   1 cCsctj|}tjtj|d sWtj|dk sWtj|dk r[tStSdS(sC Checks to see if `X` is a proper probability distribution iiN(R R tallcloseRtalltFalsetTrue(tX((s;lib/python2.7/site-packages/statsmodels/sandbox/infotheo.pyt _isproperdistQsHtefc CsGt|}|dkr3tjtj|}n|dkrbtj|tj||}n|dkrCtj|tj |}tj||}tj |\}}tj |}d}|d} |||d 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. s+px is not a proper probability distributioniitinfN( RR7tfloatR=R3tstrtlowerR RHRR R(R:talphaR;tmeasuretgenent((s;lib/python2.7/site-packages/statsmodels/sandbox/infotheo.pyt renyientropyks     '  tTcCsdS(s Generalized cross-entropy measures. Parameters ---------- px : array-like Discrete probability distribution of random variable X py : array-like Discrete probability distribution of random variable Y pxpy : 2d array-like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. N((R:R@RARLR;RM((s;lib/python2.7/site-packages/statsmodels/sandbox/infotheo.pytgencrossentropytt__main__sQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesiss Table 3.1g?gS㥛?g;On?g'1Z?gK?gMbp?gh㈵>g-C6?gMbP?g{Gz?g?g333333?g?g333333?gffffff?g?g?g?iot Informationt ProbabilityiitEntropyieg?ig"@g2@g@s Table 3.3sdiscretize functionsg3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@gL0@g3333336@g333333@g;@ǧA@g-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@g @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@s0Example in section 3.6 of Golan, using table 3.3s'Bounding errors using Fano's inequalitys"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)sor, a weaker inequalitysP_{e} >= [H(X|Y) - 1]/log(K)s P(x) = %ss?X = 3 has the highest probability, so this is the estimate Xhats1The probability of error Pe is 1 - p(X=3) = %0.4gsH(Pe) = %0.4g and K=3s-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gsor using the weaker inequalitys'Pe = %0.4g >= [H(X) - 1]/log(K) = %0.4gs>Consider now, table 3.5, where there is additional informations.The conditional probabilities of P(X|Y=y) are gg@g@s2The probability of error given this information issPe = [H(X|Y) -1]/log(K) = %0.4gs+such that more information lowers the errorgV-?gV-?gw/?g(\?g+?g%C?gzG?gPn?(>t__doc__tstatsmodels.compat.pythonRRRtscipyRtnumpyR t matplotlibRtpltt scipy.miscRRRRR1R3R5R6R=R>RCRDRERFRORQt__name__RtYR0tptsubplottylabeltxlabeltlinspacetxtplottarraytwRR:R@tH_XtH_YtH_XYt H_XgivenYt H_YgivenXR4R<tD_YXtD_XYtI_XYtdiscXtpetH_peR9tw2tmeant markovchain(((s;lib/python2.7/site-packages/statsmodels/sandbox/infotheo.pyt s  $ (   #   ( .3     -      &O   ""***'     K@