ó ù`]c@ s7dZddlmZmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZydd lmZWn,ek rÐddlZejd „ZnXe eƒd efd „ƒYƒZd„Zd„Zdd„Ze eƒdefd„ƒYƒZdS(uLanguage Model Interface.iÿÿÿÿ(tdivisiontunicode_literalsN(tABCMetatabstractmethod(tbisect(t add_metaclass(t NgramCounter(t log_base2(t Vocabulary(t accumulatecc s_t|ƒ}yt|ƒ}Wntk r0dSX|Vx"|D]}|||ƒ}|Vq=WdS(uReturn running totalsN(titertnextt StopIteration(titerabletfunctitttotaltelement((s*lib/python2.7/site-packages/nltk/lm/api.pyR s   t SmoothingcB s5eZdZd„Zed„ƒZed„ƒZRS(uëNgram Smoothing Interface Implements Chen & Goodman 1995's idea that all smoothing algorithms have certain features in common. This should ideally allow smoothing algoritms to work both with Backoff and Interpolation. cC s||_||_dS(uä :param vocabulary: The Ngram vocabulary object. :type vocabulary: nltk.lm.vocab.Vocabulary :param counter: The counts of the vocabulary items. :type counter: nltk.lm.counter.NgramCounter N(tvocabtcounts(tselft vocabularytcounter((s*lib/python2.7/site-packages/nltk/lm/api.pyt__init__2s cC s tƒ‚dS(N(tNotImplementedError(Rtword((s*lib/python2.7/site-packages/nltk/lm/api.pyt unigram_score<scC s tƒ‚dS(N(R(RRtcontext((s*lib/python2.7/site-packages/nltk/lm/api.pyt alpha_gamma@s(t__name__t __module__t__doc__RRRR(((s*lib/python2.7/site-packages/nltk/lm/api.pyR)s cC st|ƒt|ƒS(u0Return average (aka mean) for sequence of items.(tsumtlen(titems((s*lib/python2.7/site-packages/nltk/lm/api.pyt_meanEscC s#t|tjƒr|Stj|ƒS(N(t isinstancetrandomtRandom(tseed_or_generator((s*lib/python2.7/site-packages/nltk/lm/api.pyt_random_generatorJscC sy|stdƒ‚nt|ƒt|ƒkr<tdƒ‚ntt|ƒƒ}|d}|jƒ}|t|||ƒS(u`Like random.choice, but with weights. Heavily inspired by python 3.6 `random.choices`. u"Can't choose from empty populationu3The number of weights does not match the populationiÿÿÿÿ(t ValueErrorR"tlistR R&R(t populationtweightstrandom_generatort cum_weightsRt threshold((s*lib/python2.7/site-packages/nltk/lm/api.pyt_weighted_choicePs  t LanguageModelcB s€eZdZd d d„Zd d„Zd d„Zed d„ƒZd d„Z d„Z d„Z d„Z d d d d „Z RS( uKABC for Language Models. Cannot be directly instantiated itself. cC sI||_|dkrtƒn||_|dkr<tƒn||_dS(u}Creates new LanguageModel. :param vocabulary: If provided, this vocabulary will be used instead of creating a new one when training. :type vocabulary: `nltk.lm.Vocabulary` or None :param counter: If provided, use this object to count ngrams. :type vocabulary: `nltk.lm.NgramCounter` or None :param ngrams_fn: If given, defines how sentences in training text are turned to ngram sequences. :type ngrams_fn: function or None :param pad_fn: If given, defines how senteces in training text are padded. :type pad_fn: function or None N(tordertNoneRRRR(RR3RR((s*lib/python2.7/site-packages/nltk/lm/api.pyRgs c s[ˆjs7|dkr$tdƒ‚nˆjj|ƒnˆjj‡fd†|DƒƒdS(ueTrains the model on a text. :param text: Training text as a sequence of sentences. u:Cannot fit without a vocabulary or text to create it from.c3 s!|]}ˆjj|ƒVqdS(N(Rtlookup(t.0tsent(R(s*lib/python2.7/site-packages/nltk/lm/api.pys †sN(RR4R*tupdateR(Rttexttvocabulary_text((Rs*lib/python2.7/site-packages/nltk/lm/api.pytfitzs    cC s4|j|jj|ƒ|r-|jj|ƒndƒS(u©Masks out of vocab (OOV) words and computes their model score. For model-specific logic of calculating scores, see the `unmasked_score` method. N(tunmasked_scoreRR5R4(RRR((s*lib/python2.7/site-packages/nltk/lm/api.pytscoreˆscC s tƒ‚dS(uÏScore a word given some optional context. Concrete models are expected to provide an implementation. Note that this method does not mask its arguments with the OOV label. Use the `score` method for that. :param str word: Word for which we want the score :param tuple(str) context: Context the word is in. If `None`, compute unigram score. :param context: tuple(str) or None :rtype: float N(R(RRR((s*lib/python2.7/site-packages/nltk/lm/api.pyR<’scC st|j||ƒƒS(u‡Evaluate the log score of this word in this context. The arguments are the same as for `score` and `unmasked_score`. (RR=(RRR((s*lib/python2.7/site-packages/nltk/lm/api.pytlogscore£scC s)|r|jt|ƒd|S|jjS(u²Helper method for retrieving counts for a given context. Assumes context has been checked and oov words in it masked. :type context: tuple(str) or None i(RR"tunigrams(RR((s*lib/python2.7/site-packages/nltk/lm/api.pytcontext_counts«scC s5dtg|D] }|j|d|d ƒ^q ƒS(u©Calculate cross-entropy of model for given evaluation text. :param Iterable(tuple(str)) text_ngrams: A sequence of ngram tuples. :rtype: float iÿÿÿÿ(R$R>(Rt text_ngramstngram((s*lib/python2.7/site-packages/nltk/lm/api.pytentropy¶scC std|j|ƒƒS(uŽCalculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text, so the arguments are the same. g@(tpowRC(RRA((s*lib/python2.7/site-packages/nltk/lm/api.pyt perplexityÁsic  sN|dkrgn t|ƒ}t|ƒ}|dkrt|ƒˆjkr]|ˆj dn|‰ˆjˆjjˆƒƒ}xNˆrÎ| rÎtˆƒdkrªˆdng‰ˆjˆjjˆƒƒ}qWt|ƒ}t |t ‡‡fd†|Dƒƒ|ƒSg}x=t |ƒD]/}|j ˆj ddd||d|ƒƒqW|S(uÜGenerate words from the model. :param int num_words: How many words to generate. By default 1. :param text_seed: Generation can be conditioned on preceding context. :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. :return: One (str) word or a list of words generated from model. Examples: >>> from nltk.lm import MLE >>> lm = MLE(2) >>> lm.fit([[("a", "b"), ("b", "c")]], vocabulary_text=['a', 'b', 'c']) >>> lm.fit([[("a",), ("b",), ("c",)]]) >>> lm.generate(random_seed=3) 'a' >>> lm.generate(text_seed=['a']) 'b' ic3 s!|]}ˆj|ˆƒVqdS(N(R=(R6tw(RR(s*lib/python2.7/site-packages/nltk/lm/api.pys ðst num_wordst text_seedt random_seedN(R4R+R)R"R3R@RR5tsortedR1ttupletrangetappendtgenerate(RRGRHRIR.tsamplest generatedt_((RRs*lib/python2.7/site-packages/nltk/lm/api.pyRNÉs(  '" &  N(RRR R4RR;R=RR<R>R@RCRERN(((s*lib/python2.7/site-packages/nltk/lm/api.pyR2_s    (R t __future__RRR&tabcRRRtsixRtnltk.lm.counterRt nltk.lm.utilRtnltk.lm.vocabularyRt itertoolsR t ImportErrortoperatortaddtobjectRR$R)R4R1R2(((s*lib/python2.7/site-packages/nltk/lm/api.pyts(