B >?[ @sdZddlmZmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZydd lmZWn*ek rddlZejfd d ZYnXe eGd ddeZddZddZdddZe eGdddeZdS)zLanguage Model Interface.)divisionunicode_literalsN)ABCMetaabstractmethod)bisect) add_metaclass) NgramCounter) log_base2) Vocabulary) accumulateccsRt|}y t|}Wntk r(dSX|Vx|D]}|||}|Vq6WdS)zReturn running totalsN)iternext StopIteration)iterablefuncittotalelementr*lib/python3.7/site-packages/nltk/lm/api.pyr s   r c@s0eZdZdZddZeddZeddZdS) Smoothinga#Ngram Smoothing Interface Implements Chen & Goodman 1995's idea that all smoothing algorithms have certain features in common. This should ideally allow smoothing algoritms to work both with Backoff and Interpolation. counter represents the number of counts for ngrams cCs||_||_dS)N)vocabcounts)self vocabularycounterrrr__init__4szSmoothing.__init__cCs tdS)N)NotImplementedError)rwordrrr unigram_score8szSmoothing.unigram_scorecCs tdS)N)r)rrcontextrrr alpha_gamma<szSmoothing.alpha_gammaN)__name__ __module__ __qualname____doc__rrrr!rrrrr)s  rcCst|t|S)z0Return average (aka mean) for sequence of items.)sumlen)itemsrrr_meanAsr)cCst|tjr|St|S)N) isinstancerandomZRandom)Zseed_or_generatorrrr_random_generatorFs r,cCsV|s tdt|t|kr$tdtt|}|d}t|}|t|||S)z`Like random.choice, but with weights. Heavily inspired by python 3.6 `random.choices`. z"Can't choose from empty populationz3The number of weights does not match the population) ValueErrorr'listr r,r+r)Z populationZweights random_seedZ cum_weightsrZ thresholdrrr_weighted_choiceLs  r1c@sheZdZdZdddZdddZdddZedd d Zdd d Z d dZ ddZ ddZ dddZ dS) LanguageModelzKABC for Language Models. Cannot be directly instantiated itself. NcCs2||_|dkrtn||_|dkr(tn||_dS)a}Creates new LanguageModel. :param vocabulary: If provided, this vocabulary will be used instead of creating a new one when training. :type vocabulary: `nltk.lm.Vocabulary` or None :param counter: If provided, use this object to count ngrams. :type vocabulary: `nltk.lm.NgramCounter` or None :param ngrams_fn: If given, defines how sentences in training text are turned to ngram sequences. :type ngrams_fn: function or None :param pad_fn: If given, defines how senteces in training text are padded. :type pad_fn: function or None N)orderr rrr)rr3rrrrrrcszLanguageModel.__init__cs@js"|dkrtdj|jfdd|DdS)zeTrains the model on a text. :param text: Training text as a sequence of sentences. Nz:Cannot fit without a vocabulary or text to create it from.c3s|]}j|VqdS)N)rlookup).0Zsent)rrr sz$LanguageModel.fit..)rr.updater)rtextZvocabulary_textr)rrfitvs  zLanguageModel.fitcCs$||j||r|j|ndS)zMasks out of vocab (OOV) words and computes their model score. For model-specific logic of calculating scores, see the `unmasked_score` method. N)unmasked_scorerr4)rrr rrrscoreszLanguageModel.scorecCs tdS)aScore a word given some optional context. Concrete models are expected to provide an implementation. Note that this method does not mask its arguments with the OOV label. Use the `score` method for that. :param str word: Word for which we want the score :param tuple(str) context: Context the word is in. If `None`, compute unigram score. :param context: tuple(str) or None :rtype: float N)r)rrr rrrr:szLanguageModel.unmasked_scorecCst|||S)zEvaluate the log score of this word in this context. The arguments are the same as for `score` and `unmasked_score`. )r r;)rrr rrrlogscoreszLanguageModel.logscorecCs"|r|jt|d|S|jjS)zHelper method for retrieving counts for a given context. Assumes context has been checked and oov words in it masked. :type context: tuple(str) or None )rr'Zunigrams)rr rrrcontext_countsszLanguageModel.context_countscsdtfdd|DS)zCalculate cross-entropy of model for given evaluation text. :param Iterable(tuple(str)) text_ngrams: A sequence of ngram tuples. :rtype: float r-cs$g|]}|d|ddqS)r-N)r<)r5Zngram)rrr sz)LanguageModel.entropy..)r))r text_ngramsr)rrentropyszLanguageModel.entropycCstd||S)zCalculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text, so the arguments are the same. g@)powrA)rr@rrr perplexityszLanguageModel.perplexityr=cs|dkr gnt|}|dkrt|jkr>|j ddn|j}x:r|stdkrvddngj}qVWt|}t|tfdd|D|Sg}x*t |D]}| j d|||dqW|S)aGenerate words from the model. :param int num_words: How many words to generate. By default 1. :param text_seed: Generation can be conditioned on preceding context. :param random_seed: If provided, makes the random sampling part of generation reproducible. :return: One (str) word or a list of words generated from model. Examples: >>> from nltk.lm import MLE >>> lm = MLE(2) >>> lm.fit([[("a", "b"), ("b", "c")]], vocabulary_text=['a', 'b', 'c']) >>> lm.fit([[("a",), ("b",), ("c",)]]) >>> lm.generate(random_seed=3) 'a' >>> lm.generate(text_seed=['a']) 'b' Nr=c3s|]}|VqdS)N)r;)r5w)r rrrr6sz)LanguageModel.generate..) num_words text_seedr0) r/r'r3r>rr4sortedr1tuplerangeappendgenerate)rrErFr0ZsamplesZ generated_r)r rrrKs&" zLanguageModel.generate)NN)N)N)N)N)r=NN)r"r#r$r%rr9r;rr:r<r>rArCrKrrrrr2[s       r2)N)r%Z __future__rrr+abcrrrZsixrZnltk.lm.counterrZ nltk.lm.utilr Znltk.lm.vocabularyr itertoolsr ImportErroroperatoraddobjectrr)r,r1r2rrrrs(