B P?ð[$ã@sddZddlmZmZddlmZddlmZddlm Z eeƒGdd„de ƒƒZ Gdd „d e ƒZ d S) z Tokenizer Interface é)ÚABCMetaÚabstractmethod)Ú add_metaclass)Ú overridden)Ústring_span_tokenizec@s4eZdZdZedd„ƒZdd„Zdd„Zdd „Zd S) Ú TokenizerIz† A processing interface for tokenizing a string. Subclasses must define ``tokenize()`` or ``tokenize_sents()`` (or both). cCst|jƒr| |g¡dSdS)zN Return a tokenized copy of *s*. :rtype: list of str rN)rÚtokenize_sents)ÚselfÚs©r ú0lib/python3.7/site-packages/nltk/tokenize/api.pyÚtokenizes zTokenizerI.tokenizecCs tƒ‚dS)z· Identify the tokens using integer offsets ``(start_i, end_i)``, where ``s[start_i:end_i]`` is the corresponding token. :rtype: iter(tuple(int, int)) N)ÚNotImplementedError)r r r r r Ú span_tokenize%szTokenizerI.span_tokenizecs‡fdd„|DƒS)z« Apply ``self.tokenize()`` to each element of ``strings``. I.e.: return [self.tokenize(s) for s in strings] :rtype: list(list(str)) csg|]}ˆ |¡‘qSr )r )Ú.0r )r r r ú 6sz-TokenizerI.tokenize_sents..r )r Ústringsr )r r r.szTokenizerI.tokenize_sentsccs"x|D]}t| |¡ƒVqWdS)zÁ Apply ``self.span_tokenize()`` to each element of ``strings``. I.e.: return [self.span_tokenize(s) for s in strings] :rtype: iter(list(tuple(int, int))) N)Úlistr)r rr r r r Úspan_tokenize_sents8s zTokenizerI.span_tokenize_sentsN) Ú__name__Ú __module__Ú __qualname__Ú__doc__rr rrrr r r r rs    rc@s eZdZdZdd„Zdd„ZdS)ÚStringTokenizerzxA tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses). cCs | |j¡S)N)ÚsplitÚ_string)r r r r r r IszStringTokenizer.tokenizeccs xt||jƒD] }|VqWdS)N)rr)r r Úspanr r r rLszStringTokenizer.span_tokenizeN)rrrrr rr r r r rDsrN) rÚabcrrZsixrZnltk.internalsrZnltk.tokenize.utilrÚobjectrrr r r r Ú s   /