B >?[a @sFdZddlmZmZmZmZddlmZddlm Z m Z m Z ddl m Z ddlZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZe ddddddddgZGddde Z!eGddde Z"Gddde Z#eGddde Z$Gddde$Z%d d!Z&e'd"kr4e&dddddgZ(dS)#a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. )print_functiondivisionunicode_literalsabsolute_import)log) defaultdictCounter namedtuple)reduceN) text_type)FreqDist)ConditionalFreqDist) tokenwrapLazyConcatenation) f_measureBigramAssocMeasures)BigramCollocationFinder)python_2_unicode_compatibleConcordanceLineleftZqueryrightoffset left_print right_printlinec@sTeZdZdZeddZddddfddZd d Zd d ZdddZ dddZ dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cCsH|dkr||dnd}|t|dkr<||dnd}||fS)z;One left token and one right token, normalized to lowercaserz*START*z*END*)lowerlen)tokensirrr!(lib/python3.7/site-packages/nltk/text.py_default_context.s$zContextIndex._default_contextNcCs|S)Nr!)xr!r!r"5szContextIndex.csv|__|r|_nj_r6fddDtfddtD_tfddtD_dS)Ncsg|]}|r|qSr!r!).0t)filterr!r" =sz)ContextIndex.__init__..c3s(|] \}}||fVqdS)N)_key _context_func)r&r w)selfrr!r" ?sz(ContextIndex.__init__..c3s(|] \}}||fVqdS)N)r+r*)r&r r,)r-rr!r"r.Bs)r*_tokensr+r#CFD enumerate_word_to_contexts_context_to_words)r-rZ context_funcr(keyr!)r(r-rr"__init__5szContextIndex.__init__cCs|jS)zw :rtype: list(str) :return: The document that this context index was created from. )r/)r-r!r!r"rEszContextIndex.tokenscCsJ||}t|j|}i}x(|jD]\}}t|t|||<q(W|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r*setr2itemsr)r-wordZ word_contextsscoresr,Z w_contextsr!r!r"word_similarity_dictMs  z!ContextIndex.word_similarity_dictcCs~tt}x\|j||D]H}xB|j|D]4}||kr*|||j|||j||7<q*WqWt||jddd|S)NT)r4reverse)rintr2r*r3sortedget)r-r8nr9cr,r!r!r" similar_words\s(zContextIndex.similar_wordsFcsfddDfddDfddttD}ttj|rf|rftddn&sptStfddD}|Sd S) a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. csg|]}|qSr!)r*)r&r,)r-r!r"r)qsz0ContextIndex.common_contexts..csg|]}tj|qSr!)r6r2)r&r,)r-r!r"r)rscsg|]}|s|qSr!r!)r&r )contextswordsr!r"r)ssz%The following word(s) were not found: c3s*|]"}j|D]}|kr|VqqdS)N)r2)r&r,rA)commonr-r!r"r.|sz/ContextIndex.common_contexts..N)rangerr r6 intersection ValueErrorjoinr )r-rDZfail_on_unknownemptyfdr!)rFrCr-rDr"common_contextsfs  zContextIndex.common_contexts)r;)F) __name__ __module__ __qualname____doc__ staticmethodr#r5rr:rBrMr!r!r!r"r&s  rc@sLeZdZdZddfddZddZdd Zd d Zdd dZdddZ dS)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. cCs|S)Nr!)r$r!r!r"r%szConcordanceIndex.cCsJ||_||_tt|_x.t|D]"\}}||}|j||q WdS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r/r*rlist_offsetsr1append)r-rr4indexr8r!r!r"r5s   zConcordanceIndex.__init__cCs|jS)z{ :rtype: list(str) :return: The document that this concordance index was created from. )r/)r-r!r!r"rszConcordanceIndex.tokenscCs||}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r*rU)r-r8r!r!r"offsetss zConcordanceIndex.offsetscCsdt|jt|jfS)Nz+)rr/rU)r-r!r!r"__repr__szConcordanceIndex.__repr__Pc Cs|t|dd}|d}g}||}|rx|D]}|j|}|jtd|||} |j|d||} d| | d} d| d|} d| || g} t| || || | | }||q4W|S)zB Find all concordance lines given the query word. rrrEN)rrXr/maxrJrrV)r-r8widthZ half_widthcontextconcordance_listrXr Z query_wordZ left_contextZ right_contextrrZ line_printconcordance_liner!r!r"find_concordances,   z!ConcordanceIndex.find_concordancecCsj|j||d}|stdnJt|t|}td|t|x&t|d|D]\}}t|jqPWdS)a Print concordance lines given the query word. :param word: The target word :type word: str :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )r^z no matcheszDisplaying {} of {} matches:N)rbprintminrformatr1r)r-r8r^linesr`r rar!r!r"print_concordances  z"ConcordanceIndex.print_concordanceN)rZ)rZrc) rNrOrPrQr5rrXrYrbrhr!r!r!r"rSs  "rSc@s eZdZdZddZddZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cCsddd|D|_dS)Ncss|]}d|dVqdS)<>Nr!)r&r,r!r!r"r.sz)TokenSearcher.__init__..)rJ_raw)r-rr!r!r"r5szTokenSearcher.__init__cCstdd|}tdd|}tdd|}tdd|}t||j}x(|D] }|dsL|drLtd qLWd d |D}|S) a" Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srjrkz(?:<(?:rlz)>)z (?]z$Bad regexp for TokenSearcher.findallcSsg|]}|dddqS)rz><)split)r&hr!r!r"r)(sz)TokenSearcher.findall..)resubfindallrm startswithendswithrI)r-regexphitsrpr!r!r"rss  zTokenSearcher.findallN)rNrOrPrQr5rsr!r!r!r"ris ric@seZdZdZdZd/ddZddZdd Zd0d d Zd1ddZ d2ddZ ddZ ddZ ddZ d3ddZd4ddZddZd d!Zd"d#Zd$d%Zd&d'Zed(Zd)d*Zd+d,Zd-d.ZdS)5Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcCs|jrt|}||_|r ||_ndd|ddkrb|ddd}ddd|d|D|_n"ddd|dd Dd |_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]Nr;rEcss|]}t|VqdS)N)r )r&tokr!r!r"r.Vsz Text.__init__..rcss|]}t|VqdS)N)r )r&rzr!r!r"r.Xsz...) _COPY_TOKENSrTrnamerWrJ)r-rr}endr!r!r"r5Gs z Text.__init__cCs |j|S)N)r)r-r r!r!r" __getitem__^szText.__getitem__cCs t|jS)N)rr)r-r!r!r"__len__asz Text.__len__OrccCs.d|jkrt|jddd|_|j|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word :type word: str :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexcSs|S)N)r)sr!r!r"r%xsz"Text.concordance..)r4)__dict__rSrrrh)r-r8r^rgr!r!r" concordancehs zText.concordancecCs4d|jkrt|jddd|_|j||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word :type word: str :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rcSs|S)N)r)rr!r!r"r%sz'Text.concordance_list..)r4N)rrSrrrb)r-r8r^rgr!r!r"r`}s zText.concordance_listr;r[csd|jkr|j|kr|j|ks||_||_ddlm}|dt|j|}| d| fddt }| |j ||_dd |jD}tt|d d d S) aA Print collocations derived from the text, ignoring stopwords. :seealso: find_collocations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int _collocationsr) stopwordsZenglishr[cst|dkp|kS)N)rr)r,) ignored_wordsr!r"r%sz#Text.collocations..cSsg|]\}}|d|qS)rEr!)r&w1w2r!r!r"r)sz%Text.collocations..z; )Z separatorN)rZ_numZ _window_size nltk.corpusrrDrZ from_wordsrZapply_freq_filterZapply_word_filterrZnbestZlikelihood_ratiorrdr)r-numZ window_sizerfinderZbigram_measuresZcolloc_stringsr!)rr" collocationss      zText.collocationscCs |j|S)zJ Count the number of times this word appears in the text. )rcount)r-r8r!r!r"rsz Text.countcCs |j|S)zQ Find the index of the first occurrence of the word in the text. )rrW)r-r8r!r!r"rWsz Text.indexcCstdS)N)NotImplementedError)r-methodr!r!r" readabilityszText.readabilitycsd|jkr$t|jddddd|_|jjkrttfddD}dd | |D}t t |nt d d S) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexcSs|S)N)isalpha)r$r!r!r"r%szText.similar..cSs|S)N)r)rr!r!r"r%s)r(r4c3s0|](}|D]}|kr|ks|VqqdS)Nr!)r&r,rA)rCwcir8r!r"r.s zText.similar..cSsg|] \}}|qSr!r!)r&r,_r!r!r"r)sz Text.similar..z No matchesN) rrrrrr2Z conditionsr6r most_commonrdr)r-r8rrLrDr!)rCrr8r"similars   z Text.similarc Csd|jkrt|jddd|_yJ|j|d}|s.)r4TzNo common contexts were foundcSsg|] \}}|qSr!r!)r&r,rr!r!r"r)sz(Text.common_contexts..css|]\}}|d|VqdS)rNr!)r&rrr!r!r"r.sz'Text.common_contexts..N) rrrrrMrdrrrI)r-rDrrLZranked_contextser!r!r"rMs  zText.common_contextscCsddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN)Z nltk.drawr)r-rDrr!r!r"rs zText.dispersion_plotcCsddl}|dtdS)zF Issues a reminder to users following the book online rNz-The generate() method is no longer available.)warningswarnDeprecationWarning)r-rDrr!r!r"generate sz Text.generatecGs|j|dS)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() N)vocabplot)r-argsr!r!r"rsz Text.plotcCsd|jkrt||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rr r)r-r!r!r"rs  z Text.vocabcCs@d|jkrt||_|j|}dd|D}tt|ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searchercSsg|]}d|qS)rE)rJ)r&rpr!r!r"r)@sz Text.findall..z; N)rrirrsrdr)r-rvrwr!r!r"rs$s    z Text.findallz \w+|[\.\!\?]cCs|d}x$|dkr,|j||s,|d8}q W|dkr>||nd}|d}x(|t|krr|j||sr|d7}qLW|t|kr||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrz*START*z*END*) _CONTEXT_REmatchr)r-rr jrrr!r!r"_contextIs  z Text._contextcCs d|jS)Nz )r})r-r!r!r"__str__asz Text.__str__cCs d|jS)Nz )r})r-r!r!r"rYdsz Text.__repr__)N)rrc)rrc)r;r[)r;)r;)rNrOrPrQr|r5rrrr`rrrWrrrMrrrrrsrqcompilerrrrYr!r!r!r"rx,s,     "    # rxc@s0eZdZdZddZddZddZdd Zd S) TextCollectionaVA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> print('hack'); from nltk.book import text1, text2, text3 hack... >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cs@tdr fddD|_t|ti|_dS)NrDcsg|]}|qSr!)rD)r&f)sourcer!r"r)|sz+TextCollection.__init__..)hasattrZfileids_textsrxr5r _idf_cache)r-rr!)rr"r5zs  zTextCollection.__init__cCs||t|S)z$ The frequency of the term in text. )rr)r-termtextr!r!r"tfszTextCollection.tfcsj|j}|dkrftfdd|jD}t|jdkrBtd|rXtt|j|nd}||j<|S)z The number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned. Ncsg|]}|krdqS)Tr!)r&r)rr!r"r)sz&TextCollection.idf..rz+IDF undefined for empty document collectiong)rr?rrrIr)r-ridfZmatchesr!)rr"rs  zTextCollection.idfcCs|||||S)N)rr)r-rrr!r!r"tf_idfszTextCollection.tf_idfN)rNrOrPrQr5rrrr!r!r!r"ris rcCsddlm}t|jdd}t|ttd|dttd|dttd|ttd|dd d d gttd | d ttdtd|dtd|ddtd| ddS)Nr)brownZnews)Z categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:ZreportZsaidZ announcedzVocabulary plot:2z Indexing:ztext[3]:rz text[3:5]:ztext.vocab()['news']:) rrrxrDrdrrrrrr)rrr!r!r"demos.    r__main__))rQZ __future__rrrrZmathr collectionsrrr functoolsr rqZsixr Znltk.probabilityr r r0Z nltk.utilrrZ nltk.metricsrrZnltk.collocationsrZ nltk.compatrrobjectrrSrirxrrrN__all__r!r!r!r"s@       [ q9 >/