B P?[, @sdZddlmZddlZddlmZmZddlmZdZ dZ e de d d d d d df Z e dd e ejejBejBZe dZe e ejejBejBZe dZd$ddZd%ddZGdddZddZdd Zd&d"d#ZdS)'a Twitter-aware tokenizer, designed to be flexible and easy to adapt to new domains and tasks. The basic logic is this: 1. The tuple regex_strings defines a list of regular expression strings. 2. The regex_strings strings are put, in order, into a compiled regular expression object called word_re. 3. The tokenization is done by word_re.findall(s), where s is the user-supplied string, inside the tokenize() method of the class Tokenizer. 4. When instantiating Tokenizer objects, there is a single option: preserve_case. By default, it is set to True. If it is set to False, then the tokenizer will downcase everything except for emoticons. )unicode_literalsN)int2byteunichr) html_entitiesac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | <3 # heart )u # Capture 1: entire matched URL (?: https?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);strictcCs&|dkr d}t|tr"|||S|S)Nzutf-8) isinstancebytesdecode)textencodingerrorsr3lib/python3.7/site-packages/nltk/tokenize/casual.py_str_to_unicodes   rrTutf-8cs fdd}t|t||S)u Remove entities from text by converting them to their corresponding unicode character. :param text: a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). :param list keep: list of entity names which should not be replaced. This supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). :param bool remove_illegal: If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are kept "as is". :returns: A unicode string with the entities removed. See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py >>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b'Price: £100') 'Price: \xa3100' >>> print(_replace_html_entities(b'Price: £100')) Price: £100 >>> cs|d}|drzyJ|dr,t|d}n t|d}d|krJdkr\nnt|dSWqtk rvd}YqXn|kr|d Stj|}|dk ryt|Stk rYnXrd S|d S) N cp1252r) groupintrr ValueErrorrZname2codepointgetr)matchZ entity_bodyZnumber)keepremove_illegalrr_convert_entitys&        z/_replace_html_entities.._convert_entity)ENT_REsubr)r r r!r r"r)r r!r_replace_html_entitiessr%c@s"eZdZdZd ddZddZdS) TweetTokenizera Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] Examples using `strip_handles` and `reduce_len parameters`: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = '@remy: This is waaaaayyyy too much for you!!!!!!' >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] TFcCs||_||_||_dS)N) preserve_case reduce_len strip_handles)selfr'r(r)rrr__init__szTweetTokenizer.__init__cCsVt|}|jrt|}|jr$t|}td|}t|}|j sRt t dd|}|S)z :param text: str :rtype: list(str) :return: a tokenized list of strings; concatenating this list returns the original string if `preserve_case=False` z\1\1\1cSst|r|S|S)N) EMOTICON_REsearchlower)xrrr/sz)TweetTokenizer.tokenize..) r%r)remove_handlesr(reduce_lengtheningHANG_REr$WORD_REfindallr'listmap)r*r Z safe_textZwordsrrrtokenizes  zTweetTokenizer.tokenizeN)TFF)__name__ __module__ __qualname____doc__r+r8rrrrr&s r&cCstd}|d|S)ze Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}z\1\1\1)recompiler$)r patternrrrr29s r2cCstd}|d|S)z4 Remove Twitter username handles from text. zv(?r$)r r?rrrr1Bsr1FcCst|||d|S)z: Convenience function for wrapping the tokenizer. )r'r(r))r&r8)r r'r(r)rrrcasual_tokenizeRs rA)Nr)rTr)TFF)r<Z __future__rr=ZsixrrZ six.movesrZ EMOTICONSZURLSZREGEXPSr>joinVERBOSEIUNICODEr4r3r,r#rr%r&r2r1rArrrr!s2  . $    ?7