B >?ð[²ã@sÞdZddlmZddlZddlmZmZddlmZddl m Z yddl m Z Wn e k rpddl m Z YnXe dd „ƒZe e¡d d „ƒZyeWnek r°eZYnXe e¡d d „ƒZe jGdd„deƒƒZdS)zLanguage Model Vocabularyé)Úunicode_literalsN)ÚCounterÚIterable)Úchain)Úcompat)ÚsingledispatchcCstd t|ƒ¡ƒ‚dS)Nz2Unsupported type for looking up in vocabulary: {0})Ú TypeErrorÚformatÚtype)ÚwordsÚvocab©r ú1lib/python3.7/site-packages/nltk/lm/vocabulary.pyÚ_dispatched_lookupsrcst‡fdd„|DƒƒS)zcLook up a sequence of words in the vocabulary. Returns an iterator over looked up words. c3s|]}t|ˆƒVqdS)N)r)Ú.0Úw)r r rú (sz_..)Útuple)r r r )r rÚ_!srcCs||kr |S|jS)z$Looks up one word in the vocabulary.)Ú unk_label)Zwordr r r rÚ_string_lookup3src@s|eZdZdZddd„Zedd„ƒZd d „Zd d „Zd d„Z dd„Z dd„Z dd„Z dd„Z ejddkrpdd„Zdd„ZdS)Ú Vocabularya´ Stores language model vocabulary. Satisfies two common language modeling requirements for a vocabulary: - When checking membership and calculating its size, filters items by comparing their counts to a cutoff value. - Adds a special "unknown" token which unseen words are mapped to. >>> words = ['a', 'c', '-', 'd', 'c', 'a', 'b', 'r', 'a', 'c', 'd'] >>> from nltk.lm import Vocabulary >>> vocab = Vocabulary(words, unk_cutoff=2) Tokens with counts greater than or equal to the cutoff value will be considered part of the vocabulary. >>> vocab['c'] 3 >>> 'c' in vocab True >>> vocab['d'] 2 >>> 'd' in vocab True Tokens with frequency counts less than the cutoff value will be considered not part of the vocabulary even though their entries in the count dictionary are preserved. >>> vocab['b'] 1 >>> 'b' in vocab False >>> vocab['aliens'] 0 >>> 'aliens' in vocab False Keeping the count entries for seen words allows us to change the cutoff value without having to recalculate the counts. >>> vocab2 = Vocabulary(vocab.counts, unk_cutoff=1) >>> "b" in vocab2 True The cutoff value influences not only membership checking but also the result of getting the size of the vocabulary using the built-in `len`. Note that while the number of keys in the vocabulary's counter stays the same, the items in the vocabulary differ depending on the cutoff. We use `sorted` to demonstrate because it keeps the order consistent. >>> sorted(vocab2.counts) ['-', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab2) ['-', '', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab.counts) ['-', 'a', 'b', 'c', 'd', 'r'] >>> sorted(vocab) ['', 'a', 'c', 'd'] In addition to items it gets populated with, the vocabulary stores a special token that stands in for so-called "unknown" items. By default it's "". >>> "" in vocab True We can look up words in a vocabulary using its `lookup` method. "Unseen" words (with counts less than cutoff) are looked up as the unknown label. If given one word (a string) as an input, this method will return a string. >>> vocab.lookup("a") 'a' >>> vocab.lookup("aliens") '' If given a sequence, it will return an tuple of the looked up words. >>> vocab.lookup(["p", 'a', 'r', 'd', 'b', 'c']) ('', 'a', '', 'd', '', 'c') It's possible to update the counts after the vocabulary has been created. The interface follows that of `collections.Counter`. >>> vocab['b'] 1 >>> vocab.update(["b", "b", "c"]) >>> vocab['b'] 3 NéúcCsVt|tƒr||_ntƒ|_t|tƒr0|j |¡||_|dkrLtd |¡ƒ‚||_dS)aËCreate a new Vocabulary. :param counts: Optional iterable or `collections.Counter` instance to pre-seed the Vocabulary. In case it is iterable, counts are calculated. :param int unk_cutoff: Words that occur less frequently than this value are not considered part of the vocabulary. :param unk_label: Label for marking words not part of vocabulary. rz,Cutoff value cannot be less than 1. Got: {0}N) Ú isinstancerÚcountsrÚupdaterÚ ValueErrorr Ú_cutoff)ÚselfrZ unk_cutoffrr r rÚ__init__“s    zVocabulary.__init__cCs|jS)ziCutoff value. Items with count below this value are not considered part of vocabulary. )r)rr r rÚcutoff«szVocabulary.cutoffcOs|jj||ŽdS)zWUpdate vocabulary counts. Wraps `collections.Counter.update` method. N)rr)rZ counter_argsZcounter_kwargsr r rr´szVocabulary.updatecCs t||ƒS)aLook up one or more words in the vocabulary. If passed one word as a string will return that word or `self.unk_label`. Otherwise will assume it was passed a sequence of words, will try to look each of them up and return an iterator over the looked up words. :param words: Word(s) to look up. :type words: Iterable(str) or str :rtype: generator(str) or str :raises: TypeError for types other than strings or iterables >>> from nltk.lm import Vocabulary >>> vocab = Vocabulary(["a", "b", "c", "a", "b"], unk_cutoff=2) >>> vocab.lookup("a") 'a' >>> vocab.lookup("aliens") '' >>> vocab.lookup(["a", "b", "c", ["x", "b"]]) ('a', 'b', '', ('', 'b')) )r)rr r r rÚlookup¼szVocabulary.lookupcCs||jkr|jS|j|S)N)rrr)rÚitemr r rÚ __getitem__ÔszVocabulary.__getitem__cCs|||jkS)zPOnly consider items with counts GE to cutoff as being in the vocabulary.)r!)rr#r r rÚ __contains__×szVocabulary.__contains__cs(t‡fdd„ˆjDƒˆjr"ˆjgngƒS)zKBuilding on membership check define how to iterate over vocabulary.c3s|]}|ˆkr|VqdS)Nr )rr#)rr rràsz&Vocabulary.__iter__..)rrr)rr )rrÚ__iter__ÜszVocabulary.__iter__cCstdd„|DƒƒS)z1Computing size of vocabulary reflects the cutoff.css|] }dVqdS)rNr )rrr r rræsz%Vocabulary.__len__..)Úsum)rr r rÚ__len__äszVocabulary.__len__cCs$|j|jko"|j|jko"|j|jkS)N)rr!r)rÚotherr r rÚ__eq__ès  zVocabulary.__eq__récCs| |¡}|tkr|S| S)N)r*ÚNotImplemented)rr)Zequalr r rÚ__ne__ñs zVocabulary.__ne__cCsd |jj|j|jt|ƒ¡S)Nz3<{0} with cutoff={1} unk_label='{2}' and {3} items>)r Ú __class__Ú__name__r!rÚlen)rr r rÚ__str__õszVocabulary.__str__)Nrr)r/Ú __module__Ú __qualname__Ú__doc__r Úpropertyr!rr"r$r%r&r(r*ÚsysÚ version_infor-r1r r r rr9sX  r)r4Z __future__rr6Ú collectionsrrÚ itertoolsrZnltkrÚ functoolsrÚ ImportErrorrÚregisterrZ basestringÚ NameErrorÚstrrZpython_2_unicode_compatibleÚobjectrr r r rÚs$