ó ù`]c@sdZddlZddlmZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZddlmZdd lmZmZdd lmZmZmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)dd„Z*e"ƒZ+ej,dej-ƒZ.ej,dej-ƒZ/ej,dej-ƒZ0ej,dej-ƒZ1e+j2j3de.dfƒe+j2j4e/dfƒe+j5j3de0dfƒe+j6j3de1dfƒde7d„Z8dS(s¼ NLTK Tokenizer Package Tokenizers divide strings into lists of substrings. For example, tokenizers can be used to find the words and punctuation in a string: >>> from nltk.tokenize import word_tokenize >>> s = '''Good muffins cost $3.88\nin New York. Please buy me ... two of them.\n\nThanks.''' >>> word_tokenize(s) ['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] This particular tokenizer requires the Punkt sentence tokenization models to be installed. NLTK also provides a simpler, regular-expression based tokenizer, which splits text on whitespace and punctuation: >>> from nltk.tokenize import wordpunct_tokenize >>> wordpunct_tokenize(s) ['Good', 'muffins', 'cost', '$', '3', '.', '88', 'in', 'New', 'York', '.', 'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.'] We can also operate at the level of sentences, using the sentence tokenizer directly as follows: >>> from nltk.tokenize import sent_tokenize, word_tokenize >>> sent_tokenize(s) ['Good muffins cost $3.88\nin New York.', 'Please buy me\ntwo of them.', 'Thanks.'] >>> [word_tokenize(t) for t in sent_tokenize(s)] [['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York', '.'], ['Please', 'buy', 'me', 'two', 'of', 'them', '.'], ['Thanks', '.']] Caution: when tokenizing a Unicode string, make sure you are not using an encoded version of the string (it may be necessary to decode it first, e.g. with ``s.decode("utf8")``. NLTK tokenizers can produce token-spans, represented as tuples of integers having the same semantics as string slices, to support efficient comparison of tokenizers. (These methods are implemented as generators.) >>> from nltk.tokenize import WhitespaceTokenizer >>> list(WhitespaceTokenizer().span_tokenize(s)) [(0, 4), (5, 12), (13, 17), (18, 23), (24, 26), (27, 30), (31, 36), (38, 44), (45, 48), (49, 51), (52, 55), (56, 58), (59, 64), (66, 73)] There are numerous ways to tokenize text. If you need more control over tokenization, see the other methods provided in this package. For further information, please see Chapter 3 of the NLTK book. iÿÿÿÿN(tload(tTweetTokenizertcasual_tokenize(t MWETokenizer(tPunktSentenceTokenizer(tRegexpTokenizertWhitespaceTokenizertBlanklineTokenizertWordPunctTokenizertwordpunct_tokenizetregexp_tokenizetblankline_tokenize(t ReppTokenizer(tSExprTokenizertsexpr_tokenize(tSpaceTokenizert TabTokenizert LineTokenizert line_tokenize(tTextTilingTokenizer(tToktokTokenizer(tTreebankWordTokenizer(tstring_span_tokenizetregexp_span_tokenize(tStanfordSegmenter(tSyllableTokenizertenglishcCs"tdj|ƒƒ}|j|ƒS(s Return a sentence-tokenized copy of *text*, using NLTK's recommended sentence tokenizer (currently :class:`.PunktSentenceTokenizer` for the specified language). :param text: text to split into sentences :param language: the model name in the Punkt corpus stokenizers/punkt/{0}.pickle(Rtformatttokenize(ttexttlanguaget tokenizer((s5lib/python2.7/site-packages/nltk/tokenize/__init__.pyt sent_tokenize_s u([«“‘„]|[`]+)s"(?i)(\')(?!re|ve|ll|m|t|s|d)(\w)\bu ([»â€â€™])u&([^\.])(\.)([\]\)}>"\'»â€â€™ ]*)\s*$is \1 s\1 \2s \1 \2 \3 cCsK|r|gn t||ƒ}g|D]"}tj|ƒD] }|^q8q%S(sõ Return a tokenized copy of *text*, using NLTK's recommended word tokenizer (currently an improved :class:`.TreebankWordTokenizer` along with :class:`.PunktSentenceTokenizer` for the specified language). :param text: text to split into words :type text: str :param language: the model name in the Punkt corpus :type language: str :param preserve_line: An option to keep the preserve the sentence and not sentence tokenize it. :type preserve_line: bool (R t_treebank_word_tokenizerR(RRt preserve_linet sentencestsentttoken((s5lib/python2.7/site-packages/nltk/tokenize/__init__.pyt word_tokenizes(9t__doc__tret nltk.dataRtnltk.tokenize.casualRRtnltk.tokenize.mweRtnltk.tokenize.punktRtnltk.tokenize.regexpRRRRR R R tnltk.tokenize.reppR tnltk.tokenize.sexprR Rtnltk.tokenize.simpleRRRRtnltk.tokenize.texttilingRtnltk.tokenize.toktokRtnltk.tokenize.treebankRtnltk.tokenize.utilRRt nltk.tokenize.stanford_segmenterRt!nltk.tokenize.sonority_sequencingRR R!tcompiletUtimproved_open_quote_regext improved_open_single_quote_regextimproved_close_quote_regextimproved_punct_regextSTARTING_QUOTEStinserttappendt ENDING_QUOTESt PUNCTUATIONtFalseR&(((s5lib/python2.7/site-packages/nltk/tokenize/__init__.pyt=s4 4 "