B P?ð[5ã@s~dZddlmZddlmZmZddlmZmZGdd„deƒZ Gdd„deƒZ Gd d „d eƒZ Gd d „d eƒZ ddd„Z dS)aq Simple Tokenizers These tokenizers divide strings into substrings using the string ``split()`` method. When tokenizing using a particular delimiter string, use the string ``split()`` method directly, as this is more efficient. The simple tokenizers are *not* available as separate functions; instead, you should just use the string ``split()`` method directly: >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> s.split() ['Good', 'muffins', 'cost', '$3.88', 'in', 'New', 'York.', 'Please', 'buy', 'me', 'two', 'of', 'them.', 'Thanks.'] >>> s.split(' ') ['Good', 'muffins', 'cost', '$3.88\nin', 'New', 'York.', '', 'Please', 'buy', 'me\ntwo', 'of', 'them.\n\nThanks.'] >>> s.split('\n') ['Good muffins cost $3.88', 'in New York. Please buy me', 'two of them.', '', 'Thanks.'] The simple tokenizers are mainly useful because they follow the standard ``TokenizerI`` interface, and so can be used with any code that expects a tokenizer. For example, these tokenizers can be used to specify the tokenization conventions when building a `CorpusReader`. é)Úunicode_literals)Ú TokenizerIÚStringTokenizer)Ústring_span_tokenizeÚregexp_span_tokenizec@seZdZdZdZdS)ÚSpaceTokenizera­Tokenize a string using the space character as a delimiter, which is the same as ``s.split(' ')``. >>> from nltk.tokenize import SpaceTokenizer >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> SpaceTokenizer().tokenize(s) ['Good', 'muffins', 'cost', '$3.88\nin', 'New', 'York.', '', 'Please', 'buy', 'me\ntwo', 'of', 'them.\n\nThanks.'] ú N)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú_string©rrú3lib/python3.7/site-packages/nltk/tokenize/simple.pyr*s rc@seZdZdZdZdS)Ú TabTokenizerzäTokenize a string use the tab character as a delimiter, the same as ``s.split('\t')``. >>> from nltk.tokenize import TabTokenizer >>> TabTokenizer().tokenize('a\tb c\n\t d') ['a', 'b c\n', ' d'] ú N)r r r r r rrrrr8src@s eZdZdZdd„Zdd„ZdS)Ú CharTokenizerz„Tokenize a string into individual characters. If this functionality is ever required directly, use ``for char in string``. cCst|ƒS)N)Úlist)ÚselfÚsrrrÚtokenizeIszCharTokenizer.tokenizeccs2x,ttdt|ƒdƒƒD]\}}||fVqWdS)Né)Ú enumerateÚrangeÚlen)rrÚiÚjrrrÚ span_tokenizeLs zCharTokenizer.span_tokenizeN)r r r r rrrrrrrDsrc@s*eZdZdZd dd„Zdd„Zdd„Zd S) Ú LineTokenizeraVTokenize a string into its lines, optionally discarding blank lines. This is similar to ``s.split('\n')``. >>> from nltk.tokenize import LineTokenizer >>> s = "Good muffins cost $3.88\nin New York. Please buy me\ntwo of them.\n\nThanks." >>> LineTokenizer(blanklines='keep').tokenize(s) ['Good muffins cost $3.88', 'in New York. Please buy me', 'two of them.', '', 'Thanks.'] >>> # same as [l for l in s.split('\n') if l.strip()]: >>> LineTokenizer(blanklines='discard').tokenize(s) ['Good muffins cost $3.88', 'in New York. Please buy me', 'two of them.', 'Thanks.'] :param blanklines: Indicates how blank lines should be handled. Valid values are: - ``discard``: strip blank lines out of the token list before returning it. A line is considered blank if it contains only whitespace characters. - ``keep``: leave all blank lines in the token list. - ``discard-eof``: if the string ends with a newline, then do not generate a corresponding token ``''`` after that newline. ÚdiscardcCs(d}||krtdd |¡ƒ‚||_dS)N)rÚkeepz discard-eofzBlank lines must be one of: %sr)Ú ValueErrorÚjoinÚ _blanklines)rÚ blanklinesZvalid_blanklinesrrrÚ__init__hs zLineTokenizer.__init__cCsH| ¡}|jdkr"dd„|Dƒ}n"|jdkrD|rD|d ¡sD| ¡|S)NrcSsg|]}| ¡r|‘qSr)Úrstrip)Ú.0Úlrrrú usz*LineTokenizer.tokenize..z discard-eoféÿÿÿÿ)Ú splitlinesr#ÚstripÚpop)rrÚlinesrrrrqs  zLineTokenizer.tokenizeccsD|jdkr&x4t|dƒD] }|VqWnxt|dƒD] }|Vq2WdS)Nr z\nz \n(\s+\n)*)r#rr)rrÚspanrrrr|s   zLineTokenizer.span_tokenizeN)r)r r r r r%rrrrrrrQs  rrcCst|ƒ |¡S)N)rr)Útextr$rrrÚ line_tokenize‹sr1N)r)r Z __future__rZnltk.tokenize.apirrZnltk.tokenize.utilrrrrrrr1rrrrÚ$s   :