`]c@sLdZddlZddlmZddlmZdefdYZdS(s The tok-tok tokenizer is a simple, general tokenizer, where the input has one sentence per line; thus only final period is tokenized. Tok-tok has been tested on, and gives reasonably good results for English, Persian, Russian, Czech, French, German, Vietnamese, Tajik, and a few others. The input should be in UTF-8 encoding. Reference: Jon Dehdari. 2014. A Neurophysiologically-Inspired Statistical Language Model (Doctoral dissertation). Columbus, OH, USA: The Ohio State University. iN(t text_type(t TokenizerItToktokTokenizercBseZdZejddfZejddfZejddfZejddfZejddfZ ejd d fZ ejd d fZ ejd dfZ ejddfZ ejddfZejddfZejddfZejddfZejddfZejddfZejddfZedZedZedZejdjedfZejdjedfZejdjedfZejdd fZejd!d"fZejd#d$fZejd%d$fZ ejd&d'fZ!ejd(d)fZ"ejd*dfZ#eeeeee e e e eeee ee eeeeeeeee#gZ$e%d+Z&RS(,s This is a Python port of the tok-tok.pl from https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl >>> toktok = ToktokTokenizer() >>> text = u'Is 9.5 or 525,600 my favorite number?' >>> print (toktok.tokenize(text, return_str=True)) Is 9.5 or 525,600 my favorite number ? >>> text = u'The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things' >>> print (toktok.tokenize(text, return_str=True)) The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things >>> text = u'This, is a sentence with weird symbols\u2026 appearing everywhere' >>> expected = u' This , is a sentence with weird symbols \u2026 appearing everywhere ' >>> assert toktok.tokenize(text, return_str=True) == expected >>> toktok.tokenize(text) == [u'', u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'', u'symbols', u'\u2026', u'appearing', u'everywhere', u''] True u t u1([،;؛¿!"\])}»›”؟¡%٪°±©®।॥…])s \1 u([({\[“‘„‚«‹「『])u ([–—])s& s& s s s\|s | s(?s