B >?[]>@sdZddlmZddlZddlmZmZddlm Z ddl m Z ddZ ej Zd dZd Zydd lmZWnek rd d ZYnXdZdZdZe eGdddeZGdddeZGdddeZGdddeZGdddeZdS)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. )divisionN)ABCMetaabstractmethod)reduce) add_metaclasscCs t|dS)Ng@)_mathlog)xr 7lib/python3.7/site-packages/nltk/metrics/association.pysr cCstdd|S)NcSs||S)Nr )r yr r r r sz..)r)sr r r r sg#B ;) fisher_exactcOstdS)N)NotImplementedError)Z_argsZ_kwargsr r r r!src@seZdZdZdZeeddZeeddZe ddZ ed d Z e d d Z e d dZ eddZe ddZe ddZe ddZe ddZdS)NgramAssocMeasuresa An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. rcGs tddS)z>Calculates values of a contingency table from marginal values.z?The contingency table is not availablein the general ngram caseN)r) marginalsr r r _contingencyGszNgramAssocMeasures._contingencycGs tddS)zACalculates values of contingency table marginals from its values.z?The contingency table is not availablein the general ngram caseN)r) contingencyr r r _marginalsOszNgramAssocMeasures._marginalsc#s`t}ddtjD}x>ttD].tfdd|D|jdVq*WdS)z3Calculates expected values for a contingency table.cSsg|] }d|>qS)r ).0ir r r [sz7NgramAssocMeasures._expected_values..c3s2|]*tfddtdjDVqdS)c3s&|]}|@@kr|VqdS)Nr )rr )contrjr r bsz@NgramAssocMeasures._expected_values...N)sumrange_n)r)clsrr)rr rbsz6NgramAssocMeasures._expected_values..rN)r r!r"len_product)r#rn_allbitsr )r#rrr _expected_valuesWsz#NgramAssocMeasures._expected_valuescGs|t|tS)z Scores ngrams by their frequency)NGRAMTOTAL)rr r r raw_freqhszNgramAssocMeasures.raw_freqcGs6|tt|t|t|jd|ttdS)zScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. rg?)r)r%UNIGRAMSr*r"_SMALL)r#rr r r student_tmszNgramAssocMeasures.student_tcGs,|j|}||}tddt||DS)zZScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. css&|]\}}||d|tVqdS)rN)r-)robsexpr r r r~sz,NgramAssocMeasures.chi_sq..)rr(r zip)r#rrZexpsr r r chi_sqws  zNgramAssocMeasures.chi_sqcOs |t|ddt|tS)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. Zpower)r)getr%r,)rkwargsr r r mi_likeszNgramAssocMeasures.mi_likecGs.t|t|t|jdtt|tS)z^Scores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. r)_log2r)r*r"r%r,)r#rr r r pmiszNgramAssocMeasures.pmicGs.|j|}|jtddt|||DS)zOScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4. css*|]"\}}|t||ttVqdS)N)_lnr-)rr/r0r r r rsz6NgramAssocMeasures.likelihood_ratio..)rr"r r1r()r#rrr r r likelihood_ratios z#NgramAssocMeasures.likelihood_ratiocGs:t|t|t|jd}|tt|t|dS)z1Scores ngrams using the Poisson-Stirling measure.r)r%r,r*r"r)r7)r#rr0r r r poisson_stirlingsz#NgramAssocMeasures.poisson_stirlingcGs"|j|}|dt|ddS)z&Scores ngrams using the Jaccard index.rNr)rr )r#rrr r r jaccards zNgramAssocMeasures.jaccardN)__name__ __module__ __qualname____doc__r" staticmethodrrr classmethodr(r+r.r2r6r8r:r;r<r r r r r1s   rc@sheZdZdZdZeddZeddZeddZe d d Z e d d Z e d dZ eddZ dS)BigramAssocMeasuresa A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_ii counts (w1, w2), i.e. the bigram being scored n_ix counts (w1, *) n_xi counts (*, w2) n_xx counts (*, *), i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx rcCs0|\}}||}||}|||||||fS)zECalculates values of a bigram contingency table from marginal values.r )n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_ior r r rsz BigramAssocMeasures._contingencycCs"|||||f||||fS)zACalculates values of contingency table marginals from its values.r )rDrIrJn_oor r r rszBigramAssocMeasures._marginalsccsLt|}x>tdD]2}||||dA||||dA|VqWdS)z3Calculates expected values for a contingency table.rrN)r r!)rrFrr r r r(sz$BigramAssocMeasures._expected_valuescGsF|j|\}}}}||||d||||||||S)zdScores bigrams using phi-square, the square of the Pearson correlation coefficient. r)r)r#rrDrJrIrKr r r phi_sqszBigramAssocMeasures.phi_sqcCs|\}}|||||f|S)zScores bigrams using chi-square, i.e. phi-sq multiplied by the number of bigrams, as in Manning and Schutze 5.3.3. )rM)r#rDrErFrGrHr r r r2szBigramAssocMeasures.chi_sqcGs2|j|\}}}}t||g||ggdd\}}|S)zScores bigrams using Fisher's Exact Test (Pedersen 1996). Less sensitive to small counts than PMI or Chi Sq, but also more expensive to compute. Requires scipy. Zless)Z alternative)rr)r#rrDrJrIrKZoddsZpvaluer r r fisherszBigramAssocMeasures.fishercCs|\}}d|||S)z(Scores bigrams using Dice's coefficient.rr )rDrErFrGrHr r r diceszBigramAssocMeasures.diceN)r=r>r?r@r"rArrr(rBrMr2rNrOr r r r rCs     rCc@s,eZdZdZdZeddZeddZdS)TrigramAssocMeasuresa A collection of trigram association measures. Each association measure is provided as a function with four arguments:: trigram_score_fn(n_iii, (n_iix, n_ixi, n_xii), (n_ixx, n_xix, n_xxi), n_xxx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_iii counts (w1, w2, w3), i.e. the trigram being scored n_ixx counts (w1, *, *) n_xxx counts (*, *, *), i.e. any trigram r3cCs|\}}}|\}}} ||} ||} ||} | || | } ||| | }||| | }||| | | | ||}|| | | | |||fS)zCalculates values of a trigram contingency table (or cube) from marginal values. >>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r )n_iiiZ n_iix_tupleZ n_ixx_tupleZn_xxxZn_iixZn_ixiZn_xiiZn_ixxZn_xixZn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooor r r rs   z!TrigramAssocMeasures._contingencyc Gs`|\}}}}}}}}|||||||f||||||||||||ft|fS)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) )r ) rrQrRrSrUrTrVrWrXr r r r+szTrigramAssocMeasures._marginalsN)r=r>r?r@r"rArrr r r r rPs rPc@s,eZdZdZdZeddZeddZdS)QuadgramAssocMeasuresa3 A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: n_iiii counts (w1, w2, w3, w4), i.e. the quadgram being scored n_ixxi counts (w1, *, *, w4) n_xxxx counts (*, *, *, *), i.e. any quadgram rLc"Cs|\}}}}|\} } } } } }|\}}}}||}||}||}| |||}| |||}| |||}||||||||}||}||||}| |||}||||||||}| |||}||||||||}||||||||} |||||||||||||||| }!||||||||||||||| |!fS)zXCalculates values of a quadgram contingency table from marginal values. r )"n_iiiiZ n_iiix_tupleZ n_iixx_tupleZ n_ixxx_tupleZn_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooor r r rTsD      @z"QuadgramAssocMeasures._contingencyc GsV|\}}}}}}}}} } } } } }}}|| }||}||}||}||| | }||| | }||||}||||}||||}||| | }|||| || | |}|||| || | |}|||| || | | }||||||||}t|}|||||f||||||f||||f|fS)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) )r ) rrZrirjrlrkrmrnrorprqrrrsrtrurvrwr[r\r]r^r_r`rarbrcrdrerfrgrhr&r r r rs*$      z QuadgramAssocMeasures._marginalsN)r=r>r?r@r"rArrr r r r rY>s .res)r@r=)rrrr )rrr rsz(ContingencyMeasures._make_contingency_fnN)r=r>r?r@rrArr r r r rxs rx)r@Z __future__rZmathrabcrr functoolsrZsixrr7rr9r%r-Z scipy.statsr ImportErrorr)r,r*objectrrCrPrYrxr r r r  s,   xY;x