ó ù`]c @`sþdZddlmZmZmZmZddlmZddlm Z m Z m Z ddl m Z ddlZddlZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddl m!Z!ddl"m#Z#ddl$m%Z%e ddddddddgƒZ&de'fd„ƒYZ(e#de'fd„ƒYƒZ)de'fd„ƒYZ*e#de'fd „ƒYƒZ+d!e+fd"„ƒYZ,d#„Z-e.d$kråe-ƒnd%d&d'd(d)gZ/dS(*u  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. i(tprint_functiontdivisiontunicode_literalstabsolute_import(tlog(t defaultdicttCountert namedtuple(treduceN(t text_type(tMLE(tpadded_everygram_pipeline(tFreqDist(tConditionalFreqDist(t tokenwraptLazyConcatenation(t f_measuretBigramAssocMeasures(tBigramCollocationFinder(tpython_2_unicode_compatible(t sent_tokenizeuConcordanceLineuleftuqueryurightuoffsetu left_printu right_printulinet ContextIndexcB`s\eZdZed„ƒZd d d„d„Zd„Zd„Zdd„Z e d„Z RS( u A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cC`s`|dkr ||djƒnd}|t|ƒdkrP||djƒnd}||fS(u;One left token and one right token, normalized to lowercaseiiu*START*u*END*(tlowertlen(ttokenstitlefttright((s(lib/python2.7/site-packages/nltk/text.pyt_default_context2s&0cC`s|S(N((tx((s(lib/python2.7/site-packages/nltk/text.pyt9tc`s²|ˆ_ˆˆ_|r$|ˆ_n ˆjˆ_|r^gˆD]}||ƒr=|^q=‰nt‡‡fd†tˆƒDƒƒˆ_t‡‡fd†tˆƒDƒƒˆ_dS(Nc3`s6|],\}}ˆj|ƒˆjˆ|ƒfVqdS(N(t_keyt _context_func(t.0Rtw(tselfR(s(lib/python2.7/site-packages/nltk/text.pys Csc3`s6|],\}}ˆjˆ|ƒˆj|ƒfVqdS(N(R!R (R"RR#(R$R(s(lib/python2.7/site-packages/nltk/text.pys Fs(R t_tokensR!RtCFDt enumeratet_word_to_contextst_context_to_words(R$Rt context_functfiltertkeytt((R$Rs(lib/python2.7/site-packages/nltk/text.pyt__init__9s    (%cC`s|jS(uw :rtype: list(str) :return: The document that this context index was created from. (R%(R$((s(lib/python2.7/site-packages/nltk/text.pyRIscC`se|j|ƒ}t|j|ƒ}i}x6|jjƒD]%\}}t|t|ƒƒ||€sN( R R/R(trangeRRt intersectiont ValueErrortjoinR (R$twordstfail_on_unknownR#tcontextsRtemptytfd((R>R$s(lib/python2.7/site-packages/nltk/text.pytcommon_contextsjs "&3 N( t__name__t __module__t__doc__t staticmethodRtNoneR.RR5R=tFalseRH(((s(lib/python2.7/site-packages/nltk/text.pyR*s   tConcordanceIndexcB`sSeZdZd„d„Zd„Zd„Zd„Zdd„Zddd „ZRS( us An index that can be used to look up the offset locations at which a given word occurs in a document. cC`s|S(N((R((s(lib/python2.7/site-packages/nltk/text.pyRŒRcC`se||_||_ttƒ|_x=t|ƒD]/\}}|j|ƒ}|j|j|ƒq.WdS(ué Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N(R%R Rtlistt_offsetsR'tappend(R$RR,tindexR1((s(lib/python2.7/site-packages/nltk/text.pyR.Œs  cC`s|jS(u{ :rtype: list(str) :return: The document that this concordance index was created from. (R%(R$((s(lib/python2.7/site-packages/nltk/text.pyR¦scC`s|j|ƒ}|j|S(uä :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. (R RQ(R$R1((s(lib/python2.7/site-packages/nltk/text.pytoffsets®scC`s dt|jƒt|jƒfS(Nu+(RR%RQ(R$((s(lib/python2.7/site-packages/nltk/text.pyt__repr__¸s iPc C`s|t|ƒdd}|d}g}|j|ƒ}|rýx½|D]²}|j|}|jtd||ƒ|!} |j|d||!} dj| ƒ| } dj| ƒ| } dj| || gƒ} t| || || | | ƒ}|j|ƒqDWn|S(uB Find all concordance lines given the query word. iiiiu (RRTR%tmaxRBtConcordanceLineRR(R$R1twidtht half_widthtcontexttconcordance_listRTRt query_wordt left_contextt right_contextt left_printt right_printt line_printtconcordance_line((s(lib/python2.7/site-packages/nltk/text.pytfind_concordance¾s,    icC`s‹|j|d|ƒ}|s(tdƒn_t|t|ƒƒ}tdj|t|ƒƒƒx+t|| ƒD]\}}t|jƒqjWdS(u‹ Print concordance lines given the query word. :param word: The target word :type word: str :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool RXu no matchesuDisplaying {} of {} matches:N(RctprinttminRtformatR'tline(R$R1RXtlinesR[RRb((s(lib/python2.7/site-packages/nltk/text.pytprint_concordanceàs  ( RIRJRKR.RRTRURcRi(((s(lib/python2.7/site-packages/nltk/text.pyRO…s   "t TokenSearchercB`s eZdZd„Zd„ZRS(uâ A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cC`s djd„|Dƒƒ|_dS(Nucs`s|]}d|dVqdS(uN((R"R#((s(lib/python2.7/site-packages/nltk/text.pys s(RBt_raw(R$R((s(lib/python2.7/site-packages/nltk/text.pyR.scC`sÕtjdd|ƒ}tjdd|ƒ}tjdd|ƒ}tjdd|ƒ}tj||jƒ}x<|D]4}|jdƒ rp|jdƒrptd ƒ‚qpqpWg|D]}|d d !jd ƒ^q¯}|S( u" Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str u\suuu)>)u (?]u$Bad regexp for TokenSearcher.findalliiÿÿÿÿu><(tretsubtfindallRkt startswithtendswithRAtsplit(R$tregexpthitsth((s(lib/python2.7/site-packages/nltk/text.pyRns )(RIRJRKR.Rn(((s(lib/python2.7/site-packages/nltk/text.pyRj÷s  tTextcB`s eZdZeZdd„Zd„Zd„Zddd„Z ddd„Z dd d „Z dd d „Z d „Z d „Zd„Zdd„Zdd„Zd„Zdd„Zdddd„Zd„Zd„Zd„ZejdƒZd„Zd„Zd„ZRS(uÛ A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) cC`s¤|jrt|ƒ}n||_|r3||_nmd|d kr||d jdƒ}djd„|d|!Dƒƒ|_n$djd„|d Dƒƒd|_d S( uv Create a Text object. :param tokens: The source text. :type tokens: sequence of str u]iu cs`s|]}t|ƒVqdS(N(R (R"ttok((s(lib/python2.7/site-packages/nltk/text.pys Zsics`s|]}t|ƒVqdS(N(R (R"Rv((s(lib/python2.7/site-packages/nltk/text.pys \siu...N(t _COPY_TOKENSRPRtnameRSRB(R$RRxtend((s(lib/python2.7/site-packages/nltk/text.pyR.Ks   &cC`s |j|S(N(R(R$R((s(lib/python2.7/site-packages/nltk/text.pyt __getitem__bscC`s t|jƒS(N(RR(R$((s(lib/python2.7/site-packages/nltk/text.pyt__len__esiOicC`sCd|jkr-t|jdd„ƒ|_n|jj|||ƒS(u¦ Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word :type word: str :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` u_concordance_indexR,cS`s |jƒS(N(R(ts((s(lib/python2.7/site-packages/nltk/text.pyR|R(t__dict__RORt_concordance_indexRi(R$R1RXRh((s(lib/python2.7/site-packages/nltk/text.pyt concordancelscC`sDd|jkr-t|jdd„ƒ|_n|jj||ƒ| S(u¨ Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word :type word: str :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` u_concordance_indexR,cS`s |jƒS(N(R(R|((s(lib/python2.7/site-packages/nltk/text.pyR‘R(R}RORR~Rc(R$R1RXRh((s(lib/python2.7/site-packages/nltk/text.pyR[siic`sâd|jko*|j|ko*|j|ksº||_||_ddlm}|jdƒ‰tj|j|ƒ}|j dƒ|j ‡fd†ƒt ƒ}|j |j |ƒ|_ng|jD]\}}|d|^qÄS(u Return collocations derived from the text, ignoring stopwords. :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int u _collocationsi(t stopwordsuenglishic`s"t|ƒdkp!|jƒˆkS(Ni(RR(R#(t ignored_words(s(lib/python2.7/site-packages/nltk/text.pyR¬Ru (R}t_numt _window_sizet nltk.corpusR€RCRt from_wordsRtapply_freq_filtertapply_word_filterRtnbesttlikelihood_ratiot _collocations(R$tnumt window_sizeR€tfindertbigram_measurestw1tw2((Rs(lib/python2.7/site-packages/nltk/text.pytcollocation_list•s     cC`sMg|j||ƒD]\}}|d|^q}tt|ddƒƒdS(u Print collocations derived from the text, ignoring stopwords. :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int u t separatoru; N(R‘RdR(R$R‹RŒRRtcollocation_strings((s(lib/python2.7/site-packages/nltk/text.pyt collocations±s 3cC`s|jj|ƒS(uJ Count the number of times this word appears in the text. (Rtcount(R$R1((s(lib/python2.7/site-packages/nltk/text.pyR•ÀscC`s|jj|ƒS(uQ Find the index of the first occurrence of the word in the text. (RRS(R$R1((s(lib/python2.7/site-packages/nltk/text.pyRSÆscC`s t‚dS(N(tNotImplementedError(R$tmethod((s(lib/python2.7/site-packages/nltk/text.pyt readabilityÌsc`sád|jkr6t|jdd„dd„ƒ|_nˆjƒ‰|jj‰ˆˆjƒkrÓtˆˆƒ‰t‡‡‡fd†ˆjƒDƒƒ}g|j |ƒD]\}}|^q¨}t t |ƒƒn t dƒdS( u~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() u_word_context_indexR+cS`s |jƒS(N(tisalpha(R((s(lib/python2.7/site-packages/nltk/text.pyRÞRR,cS`s |jƒS(N(R(R|((s(lib/python2.7/site-packages/nltk/text.pyRÞRc3`s?|]5}ˆ|D]$}|ˆkr|ˆk r|VqqdS(N((R"R#R<(REtwciR1(s(lib/python2.7/site-packages/nltk/text.pys èsu No matchesN( R}RRt_word_context_indexRR(t conditionsR/Rt most_commonRdR(R$R1R‹RGR#t_RC((RERšR1s(lib/python2.7/site-packages/nltk/text.pytsimilarÐs $  (cC`s¿d|jkr-t|jdd„ƒ|_nyn|jj|tƒ}|sXtdƒnBg|j|ƒD]\}}|^qh}ttd„|DƒƒƒWnt k rº}t|ƒnXdS(uY Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() u_word_context_indexR,cS`s |jƒS(N(R(R|((s(lib/python2.7/site-packages/nltk/text.pyRRuNo common contexts were foundcs`s#|]\}}|d|VqdS(u_N((R"RR((s(lib/python2.7/site-packages/nltk/text.pys sN( R}RRR›RHR:RdRRRA(R$RCR‹RGR#Ržtranked_contextste((s(lib/python2.7/site-packages/nltk/text.pyRHòs  (cC`s!ddlm}|||ƒdS(uü Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() i(tdispersion_plotN(t nltk.drawR¢(R$RCR¢((s(lib/python2.7/site-packages/nltk/text.pyR¢s icC`s8t||ƒ\}}td|ƒ}|j||ƒ|S(Ntorder(R R tfit(R$ttokenized_sentsR;t train_datat padded_sentstmodel((s(lib/python2.7/site-packages/nltk/text.pyt_train_default_ngram_lmsidi*c C`s]gtdj|jƒƒD]}|jdƒ^q|_t|dƒswtddtjƒ|j |jddƒ|_ ng}|dks•t dƒ‚x„t |ƒ|krxat |j j|d |d |ƒƒD];\}}|d kríqÏn|d krýPn|j|ƒqÏW|d 7}q˜W|r5dj|ƒdnd}|t|| ƒ} t| ƒ| S(u  Print random text, generated using a trigram language model. See also `help(nltk.lm)`. :param length: The length of text to generate (default=100) :type length: int :param text_seed: Generation can be conditioned on preceding context. :type text_seed: list(str) :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. (default=42) :type random_seed: int u u trigram_modeluBuilding ngram index...tfileR;iiu!The `length` must be more than 0.t text_seedt random_seeduuiu(RRBRRqt_tokenized_sentsthasattrRdtsyststderrRªt_trigram_modeltAssertionErrorRR'tgenerateRRR( R$tlengthR¬R­tsenttgenerated_tokenstidxttokentprefixt output_str((s(lib/python2.7/site-packages/nltk/text.pyR´!s*7 "   cG`s|jƒj|ŒdS(uc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() N(tvocabtplot(R$targs((s(lib/python2.7/site-packages/nltk/text.pyR½PscC`s(d|jkr!t|ƒ|_n|jS(u. :seealso: nltk.prob.FreqDist u_vocab(R}R t_vocab(R$((s(lib/python2.7/site-packages/nltk/text.pyR¼WscC`sld|jkr!t|ƒ|_n|jj|ƒ}g|D]}dj|ƒ^q:}tt|dƒƒdS(uò Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str u_token_searcheru u; N(R}Rjt_token_searcherRnRBRdR(R$RrRsRt((s(lib/python2.7/site-packages/nltk/text.pyRn`s "u \w+|[\.\!\?]cC`sÊ|d}x1|dkr=|jj||ƒ r=|d8}q W|dkrT||nd}|d}x7|t|ƒkr|jj||ƒ r|d7}qgW|t|ƒkrº||nd}||fS(uÙ One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. iiu*START*u*END*(t _CONTEXT_REtmatchR(R$RRtjRR((s(lib/python2.7/site-packages/nltk/text.pyt_context…s & ,"cC`s d|jS(Nu (Rx(R$((s(lib/python2.7/site-packages/nltk/text.pyt__str__scC`s d|jS(Nu (Rx(R$((s(lib/python2.7/site-packages/nltk/text.pyRU sN(RIRJRKR:RwRMR.RzR{RR[R‘R”R•RSR˜RŸRHR¢RªR´R½R¼RnRltcompileRÁRÄRÅRU(((s(lib/python2.7/site-packages/nltk/text.pyRu0s0       "  /  #  tTextCollectioncB`s2eZdZd„Zd„Zd„Zd„ZRS(uVA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> print('hack'); from nltk.book import text1, text2, text3 hack... >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cC`sft|dƒr:g|jƒD]}|j|ƒ^q}n||_tj|t|ƒƒi|_dS(Nuwords(R¯tfileidsRCt_textsRuR.Rt _idf_cache(R$tsourcetf((s(lib/python2.7/site-packages/nltk/text.pyR.¶s + cC`s|j|ƒt|ƒS(u$ The frequency of the term in text. (R•R(R$ttermttext((s(lib/python2.7/site-packages/nltk/text.pyttf¾scC`s©|jj|ƒ}|dkr¥tg|jD]}||kr+t^q+ƒ}t|jƒdkrptdƒ‚n|rtt|jƒ|ƒnd}||j|sF"  [q9ÿu/