B Z(B4@sdZddlmZmZmZddlmZddlZddl m Z ddl m ZdddZ d d Zdd d ZddZddZddZdddZdddZdddZdddZdddZddd Zdd#d$Zdd&d'Zed(kred)ed*d+d+d+d+d+gZd,d-d.d/d0gZxeD]Z eee qWxeD]Z eee q Weeeeeed1d2d3d4d5d6d+d7d8d9d:d;de $d?e%dd!d@Z&e 'e&ee&e "d=e #dAe $d?e%dd!dBZ&e 'e&eeee&d!e&e(dddCgdDdDdDgdEdDdFggZ)e)*dZ+e)*d!Z,ee+Z-ee,Z.ee)Z/ee+e,e)Z0ee,e+e)Z1edej2e3e+e,Z4edej2e3e,e+Z5ee+e,e)Z6edGee-e.e/e0e1e4e5e6edHe(dIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\dYd]d^d_d`dadbdcdddedfdgdhdidjdkdld^dmdndodpdqdrdsdtdudvdwdxg2ZeeZ7eedyedzed{ed|ed}ed~e+edd!e+dZ8ede8ee8d!e8gZ9ede9ede9e8e:de0fedede8e-d!e:dfedede(dddgdCdCdCgdFdCd} || | |kr|||| <q|| } |d7}|||| <qW|S)z Discretize `X` Parameters ---------- bins : int, optional Number of bins. Default is floor(sqrt(N)) method : string "ef" is equal-frequency binning "ew" is equal-width binning Examples -------- NrZewrr) lenr ZfloorZsqrtZceilrZrankdatarminZfastsortZzerosr) rmethodZnbinsZnobsZdiscretewidthZsvecZivecZbinnumbaseirrr discretize[s(  r cCst|t|S)z There is a one-to-one transformation of the entropy value from a log base b to a log base a : H_{b}(X)=log_{b}(a)[H_{a}(X)] Returns ------- log_{b}(a) )r r)rbrrr logbasechanges r"cCsttjd|S)z$ Converts from nats to bits )r"r e)rrrr natstobitssr%cCstdtj|S)z$ Converts from bits to nats r#)r"r r$)rrrr bitstonatssr&r#cCsht|}t|dkr&t|dks.tdtt|t| }|dkr`td||S|SdS)aD This is Shannon's entropy Parameters ----------- logbase, int or np.e The base of the log px : 1d or 2d array_like Can be a discrete probability distribution, a 2d joint distribution, or a sequence of probabilities. Returns ----- For log base 2 (bits) given a discrete distribution H(p) = sum(px * log2(1/px) = -sum(pk*log2(px)) = E[log2(1/p(X))] For log base 2 (bits) given a joint distribution H(px,py) = -sum_{k,j}*w_{kj}log2(w_{kj}) Notes ----- shannonentropy(0) is defined as 0 rrz&px does not define proper distributionr#N)r r r ValueErrorr nan_to_numlog2r")pxlogbaseentropyrrrshannonentropys r-cCs\t|}t|dkr&t|dks.td|dkrLtd| t|St| SdS)z Shannon's information Parameters ---------- px : float or array-like `px` is a discrete probability distribution Returns ------- For logbase = 2 np.log2(px) rrz&px does not define proper distributionr#N)r r rr'r"r))r*r+rrr shannoninfos  r.c Cst|rt|std|dkr0t|s0td|dkrDt||}t|tt||}|dkrn|Std||SdS)a Return the conditional entropy of X given Y. Parameters ---------- px : array-like py : array-like pxpy : array-like, optional If pxpy is None, the distributions are assumed to be independent and conendtropy(px,py) = shannonentropy(px) logbase : int or np.e Returns ------- sum_{kj}log(q_{j}/w_{kj} where q_{j} = Y[j] and w_kj = X[k,j] z1px or py is not a proper probability distributionNz&pxpy is not a proper joint distribtionr#)rr'r outerrr(r)r")r*pypxpyr+condentrrr condentropys r3cCs`t|rt|std|dkr0t|s0td|dkrDt||}t||dt||||dS)aC Returns the mutual information between X and Y. Parameters ---------- px : array-like Discrete probability distribution of random variable X py : array-like Discrete probability distribution of random variable Y pxpy : 2d array-like The joint probability distribution of random variables X and Y. Note that if X and Y are independent then the mutual information is zero. logbase : int or np.e, optional Default is 2 (bits) Returns ------- shannonentropy(px) - condentropy(px,py,pxpy) z1px or py is not a proper probability distributionNz&pxpy is not a proper joint distribtion)r+)rr'r r/r-r3)r*r0r1r+rrr mutualinfos r4cCs`t|rt|std|dkr0t|s0td|dkrDt||}t||||dt||dS)aa An information theoretic correlation measure. Reflects linear and nonlinear correlation between two random variables X and Y, characterized by the discrete probability distributions px and py respectively. Parameters ---------- px : array-like Discrete probability distribution of random variable X py : array-like Discrete probability distribution of random variable Y pxpy : 2d array-like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) Returns ------- mutualinfo(px,py,pxpy,logbase=logbase)/shannonentropy(py,logbase=logbase) Notes ----- This is also equivalent to corrent(px,py,pxpy) = 1 - condent(px,py,pxpy)/shannonentropy(py) z1px or py is not a proper probability distributionNz&pxpy is not a proper joint distribtion)r+)rr'r r/r4r-)r*r0r1r+rrrcorrents r5cCsdt|rt|std|dkr0t|s0td|dkrDt||}t||||dt||||dS)ak An information theoretic covariance measure. Reflects linear and nonlinear correlation between two random variables X and Y, characterized by the discrete probability distributions px and py respectively. Parameters ---------- px : array-like Discrete probability distribution of random variable X py : array-like Discrete probability distribution of random variable Y pxpy : 2d array-like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) Returns ------- condent(px,py,pxpy,logbase=logbase) + condent(py,px,pxpy, logbase=logbase) Notes ----- This is also equivalent to covent(px,py,pxpy) = condent(px,py,pxpy) + condent(py,px,pxpy) z1px or py is not a proper probability distributionNz&pxpy is not a proper joint distribtion)r+)rr'r r/r2)r*r0r1r+rrrcovent=s r6rRcCst|stdt|}|dkrBt|}|dkr>td||S|Sdt|ks\|tjkrnt t | S||}t | }|dkrdd||Sdd|td||SdS)as Renyi's generalized entropy Parameters ---------- px : array-like Discrete probability distribution of random variable X. Note that px is assumed to be a proper probability distribution. logbase : int or np.e, optional Default is 2 (bits) alpha : float or inf The order of the entropy. The default is 1, which in the limit is just Shannon's entropy. 2 is Renyi (Collision) entropy. If the string "inf" or numpy.inf is specified the min-entropy is returned. measure : str, optional The type of entropy measure desired. 'R' returns Renyi entropy measure. 'T' returns the Tsallis entropy measure. Returns ------- 1/(1-alpha)*log(sum(px**alpha)) In the limit as alpha -> 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. z+px is not a proper probability distributionrr#infN) rr'floatr-r"stringlowerr r8rrr)r*alphar+measureZgenentrrr renyientropyisr>TcCsdS)a Generalized cross-entropy measures. Parameters ---------- px : array-like Discrete probability distribution of random variable X py : array-like Discrete probability distribution of random variable Y pxpy : 2d array-like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. Nr)r*r0r1r<r+r=rrrgencrossentropysr@__main__zQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesisz Table 3.1g?gS㥛?g;On?g'1Z?gK?gMbp?gh㈵>g-C6?gMbP?g{Gz?g?g333333?g?g333333?gffffff?g?g?g?oZ InformationZ ProbabilityiZEntropyegUUUUUU?gqq?gqq?gUUUUUU?z Table 3.3zdiscretize functionsg3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@g2@gL0@g3333336@g333333@g;@ǧA@g-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@g @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@z0Example in section 3.6 of Golan, using table 3.3z'Bounding errors using Fano's inequalityz"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)zor, a weaker inequalityzP_{e} >= [H(X|Y) - 1]/log(K)z P(x) = %sz?X = 3 has the highest probability, so this is the estimate Xhatz1The probability of error Pe is 1 - p(X=3) = %0.4gzH(Pe) = %0.4g and K=3z-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gzor using the weaker inequalityz'Pe = %0.4g >= [H(X) - 1]/log(K) = %0.4gz>Consider now, table 3.5, where there is additional informationz.The conditional probabilities of P(X|Y=y) are gg?z2The probability of error given this information iszPe = [H(X|Y) -1]/log(K) = %0.4gz+such that more information lowers the errorgV-?gV-?gw/?g(\?g+?g%C?gzG?gPn?)N)rN)r#)r#)Nr#)r#)r#)r#)rr#r7)rr#r?)>__doc__Zstatsmodels.compat.pythonrrrZscipyrZnumpyr Z matplotlibrZpltZ scipy.miscrr rr r"r%r&r-r.r3r4r5r6r>r@__name__printrYrpZsubplotZylabelZxlabelZlinspacexZplotZarraywrr*r0ZH_XZH_YZH_XYZ H_XgivenYZ H_YgivenXr$r,ZD_YXZD_XYZI_XYZdiscXZpeZH_per)Zw2ZmeanZ markovchainrrrr s    $ (  #   ( , 3            "         "6