ó ù`]c@sdZddlmZddlZddlmZmZddlmZddl m Z yddl m Z Wn!e k rddl m Z nXe d„ƒZejeƒd „ƒZyeWnek rØeZnXejeƒd „ƒZe jd efd „ƒYƒZdS( uLanguage Model Vocabularyiÿÿÿÿ(tunicode_literalsN(tCountertIterable(tchain(tcompat(tsingledispatchcCstdjt|ƒƒƒ‚dS(Nu2Unsupported type for looking up in vocabulary: {0}(t TypeErrortformatttype(twordstvocab((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt_dispatched_lookupscst‡fd†|DƒƒS(ucLook up a sequence of words in the vocabulary. Returns an iterator over looked up words. c3s|]}t|ˆƒVqdS(N(R (t.0tw(R (s1lib/python2.7/site-packages/nltk/lm/vocabulary.pys (s(ttuple(R R ((R s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt_!scCs||kr|S|jS(u$Looks up one word in the vocabulary.(t unk_label(twordR ((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt_string_lookup3st VocabularycBs–eZdZdddd„Zed„ƒZd„Zd„Zd„Z d„Z d „Z d „Z d „Z ejd d kr‹d„Znd„ZRS(u´ Stores language model vocabulary. Satisfies two common language modeling requirements for a vocabulary: - When checking membership and calculating its size, filters items by comparing their counts to a cutoff value. - Adds a special "unknown" token which unseen words are mapped to. >>> words = ['a', 'c', '-', 'd', 'c', 'a', 'b', 'r', 'a', 'c', 'd'] >>> from nltk.lm import Vocabulary >>> vocab = Vocabulary(words, unk_cutoff=2) Tokens with counts greater than or equal to the cutoff value will be considered part of the vocabulary. >>> vocab['c'] 3 >>> 'c' in vocab True >>> vocab['d'] 2 >>> 'd' in vocab True Tokens with frequency counts less than the cutoff value will be considered not part of the vocabulary even though their entries in the count dictionary are preserved. >>> vocab['b'] 1 >>> 'b' in vocab False >>> vocab['aliens'] 0 >>> 'aliens' in vocab False Keeping the count entries for seen words allows us to change the cutoff value without having to recalculate the counts. >>> vocab2 = Vocabulary(vocab.counts, unk_cutoff=1) >>> "b" in vocab2 True The cutoff value influences not only membership checking but also the result of getting the size of the vocabulary using the built-in `len`. Note that while the number of keys in the vocabulary's counter stays the same, the items in the vocabulary differ depending on the cutoff. We use `sorted` to demonstrate because it keeps the order consistent. >>> sorted(vocab2.counts) ['-', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab2) ['-', '', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab.counts) ['-', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab) ['', 'a', 'c', 'd'] In addition to items it gets populated with, the vocabulary stores a special token that stands in for so-called "unknown" items. By default it's "". >>> "" in vocab True We can look up words in a vocabulary using its `lookup` method. "Unseen" words (with counts less than cutoff) are looked up as the unknown label. If given one word (a string) as an input, this method will return a string. >>> vocab.lookup("a") 'a' >>> vocab.lookup("aliens") '' If given a sequence, it will return an tuple of the looked up words. >>> vocab.lookup(["p", 'a', 'r', 'd', 'b', 'c']) ('', 'a', '', 'd', '', 'c') It's possible to update the counts after the vocabulary has been created. The interface follows that of `collections.Counter`. >>> vocab['b'] 1 >>> vocab.update(["b", "b", "c"]) >>> vocab['b'] 3 iucCsƒt|tƒr||_n.tƒ|_t|tƒrI|jj|ƒn||_|dkrvtdj|ƒƒ‚n||_dS(uËCreate a new Vocabulary. :param counts: Optional iterable or `collections.Counter` instance to pre-seed the Vocabulary. In case it is iterable, counts are calculated. :param int unk_cutoff: Words that occur less frequently than this value are not considered part of the vocabulary. :param unk_label: Label for marking words not part of vocabulary. iu,Cutoff value cannot be less than 1. Got: {0}N( t isinstanceRtcountsRtupdateRt ValueErrorRt_cutoff(tselfRt unk_cutoffR((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__init__“s     cCs|jS(uiCutoff value. Items with count below this value are not considered part of vocabulary. (R(R((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pytcutoff«scOs|jj||ŽdS(uWUpdate vocabulary counts. Wraps `collections.Counter.update` method. N(RR(Rt counter_argstcounter_kwargs((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyR´scCs t||ƒS(uLook up one or more words in the vocabulary. If passed one word as a string will return that word or `self.unk_label`. Otherwise will assume it was passed a sequence of words, will try to look each of them up and return an iterator over the looked up words. :param words: Word(s) to look up. :type words: Iterable(str) or str :rtype: generator(str) or str :raises: TypeError for types other than strings or iterables >>> from nltk.lm import Vocabulary >>> vocab = Vocabulary(["a", "b", "c", "a", "b"], unk_cutoff=2) >>> vocab.lookup("a") 'a' >>> vocab.lookup("aliens") '' >>> vocab.lookup(["a", "b", "c", ["x", "b"]]) ('a', 'b', '', ('', 'b')) (R (RR ((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pytlookup¼scCs!||jkr|jS|j|S(N(RRR(Rtitem((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt __getitem__ÔscCs|||jkS(uPOnly consider items with counts GE to cutoff as being in the vocabulary.(R(RR ((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt __contains__×scs5t‡fd†ˆjDƒˆjr.ˆjgngƒS(uKBuilding on membership check define how to iterate over vocabulary.c3s!|]}|ˆkr|VqdS(N((R R (R(s1lib/python2.7/site-packages/nltk/lm/vocabulary.pys às(RRR(R((Rs1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__iter__ÜscCstd„|DƒƒS(u1Computing size of vocabulary reflects the cutoff.css|] }dVqdS(iN((R R((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pys æs(tsum(R((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__len__äscCs4|j|jko3|j|jko3|j|jkS(N(RRR(Rtother((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__eq__èsiicCs$|j|ƒ}|tkr|S| S(N(R'tNotImplemented(RR&tequal((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__ne__ñscCs(dj|jj|j|jt|ƒƒS(Nu3<{0} with cutoff={1} unk_label='{2}' and {3} items>(Rt __class__t__name__RRtlen(R((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyt__str__õsN(R,t __module__t__doc__tNoneRtpropertyRRRR!R"R#R%R'tsyst version_infoR*R.(((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyR9sX        (R0t __future__RR3t collectionsRRt itertoolsRtnltkRt functoolsRt ImportErrorR tregisterRt basestringt NameErrortstrRtpython_2_unicode_compatibletobjectR(((s1lib/python2.7/site-packages/nltk/lm/vocabulary.pyts$