ó ù`]c@ s:dZddlmZddlZddlmZmZddlm Z ddl m Z d„Z ej Zd„Zd Zydd lmZWnek r©d „ZnXd Zd ZdZe eƒdefd„ƒYƒZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdS(sÌ Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. iÿÿÿÿ(tdivisionN(tABCMetatabstractmethod(treduce(t add_metaclasscC stj|dƒS(Ng@(t_mathtlog(tx((s7lib/python2.7/site-packages/nltk/metrics/association.pyttcC std„|ƒS(NcS s||S(N((Rty((s7lib/python2.7/site-packages/nltk/metrics/association.pyRR (R(ts((s7lib/python2.7/site-packages/nltk/metrics/association.pyRR g#B’ ¡œÇ;(t fisher_exactcO s t‚dS(N(tNotImplementedError(t_argst_kwargs((s7lib/python2.7/site-packages/nltk/metrics/association.pyR !siiþÿÿÿtNgramAssocMeasurescB sÅeZdZdZeed„ƒƒZeed„ƒƒZed„ƒZ ed„ƒZ ed„ƒZ ed„ƒZ ed„ƒZ ed „ƒZed „ƒZed „ƒZed „ƒZRS( s¿ An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. icG stdƒ‚dS(s>Calculates values of a contingency table from marginal values.s?The contingency table is not availablein the general ngram caseN(R (t marginals((s7lib/python2.7/site-packages/nltk/metrics/association.pyt _contingencyGscG stdƒ‚dS(sACalculates values of contingency table marginals from its values.s?The contingency table is not availablein the general ngram caseN(R (t contingency((s7lib/python2.7/site-packages/nltk/metrics/association.pyt _marginalsOsc# sƒtˆƒ}gtˆjƒD]‰dˆ>^q}xJttˆƒƒD]6‰t‡‡‡fd†|Dƒƒ|ˆjdVqEWdS(s3Calculates expected values for a contingency table.ic3 s>|]4‰t‡‡‡fd†tdˆjƒDƒƒVqdS(c3 s-|]#}|ˆ@ˆˆ@krˆ|VqdS(N((t.0R(tconttitj(s7lib/python2.7/site-packages/nltk/metrics/association.pys bsiN(tsumtranget_n(R(tclsRR(Rs7lib/python2.7/site-packages/nltk/metrics/association.pys bsN(RRRtlent_product(RRtn_alltbits((RRRs7lib/python2.7/site-packages/nltk/metrics/association.pyt_expected_valuesWs & cG s|t|tS(s Scores ngrams by their frequency(tNGRAMtTOTAL(R((s7lib/python2.7/site-packages/nltk/metrics/association.pytraw_freqhscG s9|tt|tƒ|t|jd|ttdS(sScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. igà?(R"RtUNIGRAMSR#Rt_SMALL(RR((s7lib/python2.7/site-packages/nltk/metrics/association.pyt student_tms!cG s;|j|Œ}|j|ƒ}td„t||ƒDƒƒS(sZScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. cs s+|]!\}}||d|tVqdS(iN(R&(Rtobstexp((s7lib/python2.7/site-packages/nltk/metrics/association.pys ~s(RR!Rtzip(RRRtexps((s7lib/python2.7/site-packages/nltk/metrics/association.pytchi_sqwscO s&|t|jddƒt|tƒS(sÂScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. tpoweri(R"tgetRR%(Rtkwargs((s7lib/python2.7/site-packages/nltk/metrics/association.pytmi_like€scG s5t|t|t|jdƒtt|tƒƒS(s^Scores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. i(t_log2R"R#RRR%(RR((s7lib/python2.7/site-packages/nltk/metrics/association.pytpmiŠs#cG s<|j|Œ}|jtd„t||j|ƒƒDƒƒS(sOScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4. cs s1|]'\}}|t||ttƒVqdS(N(t_lnR&(RR(R)((s7lib/python2.7/site-packages/nltk/metrics/association.pys ™s(RRRR*R!(RRR((s7lib/python2.7/site-packages/nltk/metrics/association.pytlikelihood_ratio“s cG sAt|tƒ|t|jd}|tt|t|ƒdS(s1Scores ngrams using the Poisson-Stirling measure.i(RR%R#RR"R1(RRR)((s7lib/python2.7/site-packages/nltk/metrics/association.pytpoisson_stirlings#cG s%|j|Œ}|dt|d ƒS(s&Scores ngrams using the Jaccard index.iiÿÿÿÿ(RR(RRR((s7lib/python2.7/site-packages/nltk/metrics/association.pytjaccard£s(t__name__t __module__t__doc__Rt staticmethodRRRt classmethodR!R$R'R,R0R2R4R5R6(((s7lib/python2.7/site-packages/nltk/metrics/association.pyR1s     tBigramAssocMeasurescB s}eZdZdZed„ƒZed„ƒZed„ƒZed„ƒZ ed„ƒZ ed„ƒZ ed„ƒZ RS( s€ A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_ii counts (w1, w2), i.e. the bigram being scored n_ix counts (w1, *) n_xi counts (*, w2) n_xx counts (*, *), i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx icC s<|\}}||}||}|||||||fS(sECalculates values of a bigram contingency table from marginal values.((tn_iit n_ix_xi_tupletn_xxtn_ixtn_xitn_oitn_io((s7lib/python2.7/site-packages/nltk/metrics/association.pyRÈs   cC s'|||||f||||fS(sACalculates values of contingency table marginals from its values.((R=RBRCtn_oo((s7lib/python2.7/site-packages/nltk/metrics/association.pyRÐscc sTt|ƒ}xAtdƒD]3}||||dA||||dA|VqWdS(s3Calculates expected values for a contingency table.iiiN(RR(RR?R((s7lib/python2.7/site-packages/nltk/metrics/association.pyR!Õs cG sO|j|Œ\}}}}||||d||||||||S(sdScores bigrams using phi-square, the square of the Pearson correlation coefficient. i(R(RRR=RCRBRD((s7lib/python2.7/site-packages/nltk/metrics/association.pytphi_sqÝscC s)|\}}||j|||f|ƒS(sƒScores bigrams using chi-square, i.e. phi-sq multiplied by the number of bigrams, as in Manning and Schutze 5.3.3. (RE(RR=R>R?R@RA((s7lib/python2.7/site-packages/nltk/metrics/association.pyR,ès cG sI|j|Œ\}}}}t||g||ggddƒ\}}|S(sºScores bigrams using Fisher's Exact Test (Pedersen 1996). Less sensitive to small counts than PMI or Chi Sq, but also more expensive to compute. Requires scipy. t alternativetless(RR (RRR=RCRBRDtoddstpvalue((s7lib/python2.7/site-packages/nltk/metrics/association.pytfisherðs*cC s|\}}d|||S(s(Scores bigrams using Dice's coefficient.i((R=R>R?R@RA((s7lib/python2.7/site-packages/nltk/metrics/association.pytdiceüs ( R7R8R9RR:RRR!R;RER,RJRK(((s7lib/python2.7/site-packages/nltk/metrics/association.pyR<ªs  tTrigramAssocMeasurescB s2eZdZdZed„ƒZed„ƒZRS(sÄ A collection of trigram association measures. Each association measure is provided as a function with four arguments:: trigram_score_fn(n_iii, (n_iix, n_ixi, n_xii), (n_ixx, n_xix, n_xxi), n_xxx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_iii counts (w1, w2, w3), i.e. the trigram being scored n_ixx counts (w1, *, *) n_xxx counts (*, *, *), i.e. any trigram icC s°|\}}}|\}}} ||} ||} ||} | || | } ||| | }||| | }||| | | | ||}|| | | | |||fS(sÔCalculates values of a trigram contingency table (or cube) from marginal values. >>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) ((tn_iiit n_iix_tuplet n_ixx_tupletn_xxxtn_iixtn_ixitn_xiitn_ixxtn_xixtn_xxitn_oiitn_ioitn_iiotn_ooitn_oiotn_iootn_ooo((s7lib/python2.7/site-packages/nltk/metrics/association.pyRs   "c G sv|\}}}}}}}}|||||||f||||||||||||ft|ƒfS(s»Calculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) (R( RRMRWRXRZRYR[R\R]((s7lib/python2.7/site-packages/nltk/metrics/association.pyR+s(R7R8R9RR:RR(((s7lib/python2.7/site-packages/nltk/metrics/association.pyRLstQuadgramAssocMeasurescB s2eZdZdZed„ƒZed„ƒZRS(s3 A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_iiii counts (w1, w2, w3, w4), i.e. the quadgram being scored n_ixxi counts (w1, *, *, w4) n_xxxx counts (*, *, *, *), i.e. any quadgram ic"C sÎ|\}}}}|\} } } } } }|\}}}}||}||}||}| |||}| |||}| |||}||||||||}||}||||}| |||}||||||||}| |||}||||||||}||||||||} |||||||||||||||| }!||||||||||||||| |!fS(sXCalculates values of a quadgram contingency table from marginal values. (("tn_iiiit n_iiix_tuplet n_iixx_tuplet n_ixxx_tupletn_xxxxtn_iiixtn_iixitn_ixiitn_xiiitn_iixxtn_ixixtn_ixxitn_xixitn_xxiitn_xiixtn_ixxxtn_xixxtn_xxixtn_xxxitn_oiiitn_ioiitn_iioitn_ooiitn_oioitn_iooitn_oooitn_iiiotn_oiiotn_ioiotn_ooiotn_iiootn_oiootn_ioootn_oooo((s7lib/python2.7/site-packages/nltk/metrics/association.pyRTsD   " """Bc G s›|\}}}}}}}}} } } } } }}}|| }||}||}||}||| | }||| | }||||}||||}||||}||| | }|||| || | |}|||| || | |}|||| || | | }||||||||}t|ƒ}|||||f||||||f||||f|fS(sCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) (R( RR_RrRsRuRtRvRwRxRyRzR{R|R}R~RR€RdReRfRgRhRiRjRkRlRmRnRoRpRqR((s7lib/python2.7/site-packages/nltk/metrics/association.pyRs*6    """" (R7R8R9RR:RR(((s7lib/python2.7/site-packages/nltk/metrics/association.pyR^>sq#nt||ƒ}|jdƒsq|j||ƒ}nt|||ƒq#WdS(sAConstructs a ContingencyMeasures given a NgramAssocMeasures classt Contingencyt__t_N(t __class__R7tdirt startswithtgetattrt_make_contingency_fntsetattr(tselftmeasurestktv((s7lib/python2.7/site-packages/nltk/metrics/association.pyt__init__»sc s.‡‡fd†}ˆj|_ˆj|_|S(s‡From an association measure function, produces a new function which accepts contingency table values as its arguments. c sˆˆj|ŒŒS(N(R(R(RŒtold_fn(s7lib/python2.7/site-packages/nltk/metrics/association.pytresÌs(R9R7(RŒRR‘((RŒRs7lib/python2.7/site-packages/nltk/metrics/association.pyR‰Æs  (R7R8R9RR:R‰(((s7lib/python2.7/site-packages/nltk/metrics/association.pyR¶s (R9t __future__RtmathRtabcRRt functoolsRtsixRR1RR3RR&t scipy.statsR t ImportErrorR"R%R#tobjectRR<RLR^R(((s7lib/python2.7/site-packages/nltk/metrics/association.pyt s,       xY;x