ó ù`]c@sˆdZddlmZmZddlmZddlmZddlm Z eeƒde fd„ƒYƒZ de fd „ƒYZ d S( s Tokenizer Interface iÿÿÿÿ(tABCMetatabstractmethod(t add_metaclass(t overridden(tstring_span_tokenizet TokenizerIcBs8eZdZed„ƒZd„Zd„Zd„ZRS(s† A processing interface for tokenizing a string. Subclasses must define ``tokenize()`` or ``tokenize_sents()`` (or both). cCs't|jƒr#|j|gƒdSdS(sN Return a tokenized copy of *s*. :rtype: list of str iN(Rttokenize_sents(tselfts((s0lib/python2.7/site-packages/nltk/tokenize/api.pyttokenizescCs tƒ‚dS(s· Identify the tokens using integer offsets ``(start_i, end_i)``, where ``s[start_i:end_i]`` is the corresponding token. :rtype: iter(tuple(int, int)) N(tNotImplementedError(RR((s0lib/python2.7/site-packages/nltk/tokenize/api.pyt span_tokenize%scCs g|D]}|j|ƒ^qS(s« Apply ``self.tokenize()`` to each element of ``strings``. I.e.: return [self.tokenize(s) for s in strings] :rtype: list(list(str)) (R (RtstringsR((s0lib/python2.7/site-packages/nltk/tokenize/api.pyR.sccs)x"|D]}t|j|ƒƒVqWdS(sÁ Apply ``self.span_tokenize()`` to each element of ``strings``. I.e.: return [self.span_tokenize(s) for s in strings] :rtype: iter(list(tuple(int, int))) N(tlistR (RR R((s0lib/python2.7/site-packages/nltk/tokenize/api.pytspan_tokenize_sents8s (t__name__t __module__t__doc__RR R RR(((s0lib/python2.7/site-packages/nltk/tokenize/api.pyRs  tStringTokenizercBs eZdZd„Zd„ZRS(sxA tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses). cCs|j|jƒS(N(tsplitt_string(RR((s0lib/python2.7/site-packages/nltk/tokenize/api.pyR Isccs&xt||jƒD] }|VqWdS(N(RR(RRtspan((s0lib/python2.7/site-packages/nltk/tokenize/api.pyR Ls(RRRR R (((s0lib/python2.7/site-packages/nltk/tokenize/api.pyRDs N( RtabcRRtsixRtnltk.internalsRtnltk.tokenize.utilRtobjectRR(((s0lib/python2.7/site-packages/nltk/tokenize/api.pyt s /