ó ù`]c@ sûdZddlmZmZddlmZddlmZmZddl m Z m Z ej defd„ƒYƒZ ej defd „ƒYƒZej d efd „ƒYƒZd efd „ƒYZdefd„ƒYZdefd„ƒYZdS(uLanguage Modelsiÿÿÿÿ(tdivisiontunicode_literals(tcompat(t LanguageModelt Smoothing(t KneserNeyt WittenBelltMLEcB seZdZdd„ZRS(ubClass for providing MLE ngram model scores. Inherits initialization from BaseNgramModel. cC s|j|ƒj|ƒS(uÄReturns the MLE score for a word given a context. Args: - word is expcected to be a string - context is expected to be something reasonably convertible to a tuple (tcontext_countstfreq(tselftwordtcontext((s-lib/python2.7/site-packages/nltk/lm/models.pytunmasked_scoresN(t__name__t __module__t__doc__tNoneR (((s-lib/python2.7/site-packages/nltk/lm/models.pyRstLidstonecB s#eZdZd„Zdd„ZRS(u«Provides Lidstone-smoothed scores. In addition to initialization arguments from BaseNgramModel also requires a number by which to increase the counts, gamma. cO s&tt|ƒj||Ž||_dS(N(tsuperRt__init__tgamma(R Rtargstkwargs((s-lib/python2.7/site-packages/nltk/lm/models.pyR)scC sH|j|ƒ}||}|jƒ}||j|t|jƒ|jS(utAdd-one smoothing: Lidstone or Laplace. To see what kind, look at `gamma` attribute on the class. (RtNRtlentvocab(R R R tcountst word_countt norm_count((s-lib/python2.7/site-packages/nltk/lm/models.pyR -s  N(RRRRRR (((s-lib/python2.7/site-packages/nltk/lm/models.pyR!s tLaplacecB seZdZd„ZRS(uwImplements Laplace (add one) smoothing. Initialization identical to BaseNgramModel because gamma is always 1. cO s tt|ƒjd||ŽdS(Ni(RRR(R RR((s-lib/python2.7/site-packages/nltk/lm/models.pyR@s(RRRR(((s-lib/python2.7/site-packages/nltk/lm/models.pyR9stInterpolatedLanguageModelcB s#eZdZd„Zdd„ZRS(uuLogic common to all interpolated language models. The idea to abstract this comes from Chen & Goodman 1995. cK s_t|tƒst‚|jdiƒ}tt|ƒj||||j|j||_ dS(Nuparams( t issubclassRtAssertionErrortpopRRRRRt estimator(R t smoothing_clstorderRtparams((s-lib/python2.7/site-packages/nltk/lm/models.pyRJscC sM|s|jj|ƒS|jj||ƒ\}}|||j||dƒS(Ni(R#t unigram_scoret alpha_gammaR (R R R talphaR((s-lib/python2.7/site-packages/nltk/lm/models.pyR PsN(RRRRRR (((s-lib/python2.7/site-packages/nltk/lm/models.pyRDs tWittenBellInterpolatedcB seZdZd„ZRS(u.Interpolated version of Witten-Bell smoothing.cK s tt|ƒjt||dS(N(RR*RR(R R%R((s-lib/python2.7/site-packages/nltk/lm/models.pyRZs(RRRR(((s-lib/python2.7/site-packages/nltk/lm/models.pyR*WstKneserNeyInterpolatedcB seZdZdd„ZRS(u-Interpolated version of Kneser-Ney smoothing.gš™™™™™¹?cK s-tt|ƒjt|di|d6|dS(NR&udiscount(RR+RR(R R%tdiscountR((s-lib/python2.7/site-packages/nltk/lm/models.pyRas(RRRR(((s-lib/python2.7/site-packages/nltk/lm/models.pyR+^sN(Rt __future__RRtnltkRt nltk.lm.apiRRtnltk.lm.smoothingRRtpython_2_unicode_compatibleRRRRR*R+(((s-lib/python2.7/site-packages/nltk/lm/models.pyts