B >?[e@s ddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZddlmZddlmZmZddlmZddlmZmZmZddlmZmZmZmZmZmZmZdd l m!Z!m"Z"dd l#Tdd l$m%Z%dQd dZ&ddZ'dRddZ(dSddZ)dTddZ*ddZ+ddZ,ddZ-Gdd d eZ.dUd#d$Z/d%d&Z0e1d'fd(d)Z2d*d+Z3d,d-Z4d.d/Z5dVd1d2Z6d3d4Z7d5d6Z8d7d8Z9d9d:Z:dWd;d<Z;dXd=d>ZdYdDdEZ?dFdGZ@id'fdHdIZAdZdKdLZBd[dMdNZCdOdPZDdS)\)print_functionN)islicechain combinations)pprint) defaultdictdeque) version_info) class_types string_types text_type) build_openerinstall_opener getproxies ProxyHandlerProxyBasicAuthHandlerProxyDigestAuthHandlerHTTPPasswordMgrWithDefaultRealm) slice_boundsraise_unorderable_types)*)python_2_unicode_compatibleselfc Cst|t|ts|j}td|jxtt| D]\}}| drNq:t |ddr\q:t j ddkrrtj}ntj}||dd\}}}}|r|ddkr|dkst|t|kr|d d}d ||f}t||||} ttjd || fd d t|ddq:WdS)Nz%%s supports the following operations:_Z__deprecated__Frrz%s.%sz%s%sz -  )Zinitial_indentZsubsequent_indent)str isinstancer __class__print__name__sortedpydocZ allmethodsitems startswithgetattrsysr inspectZgetfullargspec getargspeclenZ formatargspectextwrapZfill) objZselfnamenamemethodr+argsZvarargsZvarkwdefaultsZargspecr3(lib/python3.7/site-packages/nltk/util.pyusage-s0       r5cCsddl}|jjjdkS)a Return True if this function is run within idle. Tkinter programs that are run in idle should never call ``Tk.mainloop``; so this function should be used to gate all calls to ``Tk.mainloop``. :warning: This function works by checking ``sys.stdin``. If the user has modified ``sys.stdin``, then it may return incorrect results. :rtype: bool rN)ZPyShellZRPCProxy)r)stdinr!r#)r)r3r3r4in_idleUs r7cCsttt|||dS)z Pretty print a sequence of data items :param data: the data stream to print :type data: sequence or iter :param start: the start position :type start: int :param end: the end position :type end: int N)rlistr)datastartendr3r3r4prjs r<FcCstdtj||ddS)z Pretty print a string, breaking lines on whitespace :param s: the string to print, consisting of words and spaces :type s: str :param width: the display width :type width: int  )widthN)r"joinr-wrap)sr?r3r3r4 print_stringxs rCrcCsdtj|||dS)a# Pretty print a list of text tokens, breaking lines on whitespace :param tokens: the tokens to print :type tokens: list :param separator: the string to use to separate tokens :type separator: str :param width: the display width (default=70) :type width: int r>)r?)r@r-rA)tokensZ separatorr?r3r3r4 tokenwraps rEcCstddkotddkS)Nrrr)r r3r3r3r4py25srGcCstddkotddkS)NrrFr)r r3r3r3r4py26srIcCstddkotddkS)NrrFr)r r3r3r3r4py27srKc@seZdZddZdS)IndexcCs0t|tx|D]\}}|||qWdS)N)r__init__r8append)rZpairskeyvaluer3r3r4rMs zIndex.__init__N)r# __module__ __qualname__rMr3r3r3r4rLsrL{}cCs*tt|tj|d||dS)a3 Return a string with markers surrounding the matched substrings. Search str for substrings matching ``regexp`` and wrap the matches with braces. This is convenient for learning about regular expressions. :param regexp: The regular expression. :type regexp: str :param string: The string being matched. :type string: str :param left: The left delimiter (printed before the matched substring) :type left: str :param right: The right delimiter (printed after the matched substring) :type right: str :rtype: str z\g<0>N)r"recompileMsubrstrip)Zregexpstringleftrightr3r3r4re_showsr]c CsDt|dr|St|tr8t|d }|SQRXntddS)Nreadrz2Must be called with a filename or file-like object)hasattrr^r r open ValueError)fZinfiler3r3r4 filestrings    rdc#slt|dfg}xX|rf|\}|V|kry |fdd||DWqtk rbYqXqWdS)aTraverse the nodes of a tree in breadth-first order. (No need to check for cycles.) The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. rc3s|]}|dfVqdS)rNr3).0c)depthr3r4 sz breadth_first..N)rpopleftextend TypeError)ZtreeZchildrenZmaxdepthZqueueZnoder3)rhr4 breadth_firsts  rmc Csd}dg}y|ttjWntk r4YnXy|tdWnttfk rdYnXy|tdWnttfk rYnX|dx@|D]8}|sqyt||}|}Wnt t fk rYqXPqW|st dd dd|Dn||fSdS) at Given a byte string, attempt to decode it. Tries the standard 'UTF8' and 'latin-1' encodings, Plus several gathered from locale information. The calling program *must* first call:: locale.setlocale(locale.LC_ALL, '') If successful it returns ``(decoded_unicode, successful_encoding)``. If unsuccessful it raises a ``UnicodeError``. Nzutf-8rzlatin-1z?Unable to decode input data. Tried the following encodings: %s.z, cSsg|]}|rt|qSr3)repr)rfencr3r3r4 )sz"guess_encoding..) rNlocale nl_langinfoCODESETAttributeErrorZ getlocale IndexErrorZgetdefaultlocaler UnicodeError LookupErrorr@)r9Zsuccessful_encodingZ encodingsroZdecodedr3r3r4guess_encodings:    rxcstfdd|DS)Ncs"g|]}|kr|s|qSr3)add)rfx)seenr3r4rp7szunique_list..)set)Zxsr3)r{r4 unique_list4sr}cCsVtt}xH|D]@}t||drBx,||D]}|||q*Wq||||<qW|S)N__iter__)rr8r`rN)dZ inverted_dictrOZtermr3r3r4 invert_dict?s rFcs|rddnddtfddD}tfddD}xhD]`}||}||}xJ|r|}||||||O}||||O}||8}q^WqHW|S)a Calculate the transitive closure of a directed graph, optionally the reflexive transitive closure. The algorithm is a slight modification of the "Marking Algorithm" of Ioannidis & Ramakrishnan (1998) "Efficient Transitive Closure Algorithms". :param graph: the initial graph, represented as a dictionary of sets :type graph: dict(set) :param reflexive: if set, also make the closure reflexive :type reflexive: bool :rtype: dict(set) cSs t|gS)N)r|)kr3r3r4_sz$transitive_closure..cSstS)N)r|)rr3r3r4rasc3s|]}||fVqdS)N)copy)rfr)graphr3r4ricsz%transitive_closure..c3s|]}||fVqdS)Nr3)rfr)base_setr3r4ries)dictpopry setdefaultget)rZ reflexiveZ agenda_graphZ closure_graphiZagendaZclosurejr3)rrr4transitive_closurePs   rcCs<i}x2|D]*}x$||D]}||t|qWq W|S)z Inverts a directed graph. :param graph: the graph, represented as a dictionary of sets :type graph: dict(set) :return: the inverted graph :rtype: dict(set) )rr|ry)rinvertedrOrPr3r3r4 invert_graphrs  rcCs tddS)Nz>To remove HTML markup, use BeautifulSoup's get_text() function)NotImplementedError)Zhtmlr3r3r4 clean_htmlsrcCs tddS)Nz>To remove HTML markup, use BeautifulSoup's get_text() function)r)Zurlr3r3r4 clean_urlsrcGs`g}xV|D]N}t|ttfs"|g}x4|D],}t|ttfrJ|t|q(||q(Wq W|S)z Flatten a list. >>> from nltk.util import flatten >>> flatten(1, 2, ['b', 'a' , ['c', 'd']], 3) [1, 2, 'b', 'a', 'c', 'd', 3] :param args: items and lists to be combined into a single list :rtype: list )r r8tuplerkflattenrN)r1rzlitemr3r3r4rs   rcCs<t|}|r t|f|d|}|r8t||f|d}|S)a Returns a padded sequence of items before ngram extraction. >>> list(pad_sequence([1,2,3,4,5], 2, pad_left=True, pad_right=True, left_pad_symbol='', right_pad_symbol='')) ['', 1, 2, 3, 4, 5, ''] >>> list(pad_sequence([1,2,3,4,5], 2, pad_left=True, left_pad_symbol='')) ['', 1, 2, 3, 4, 5] >>> list(pad_sequence([1,2,3,4,5], 2, pad_right=True, right_pad_symbol='')) [1, 2, 3, 4, 5, ''] :param sequence: the source data to be padded :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param pad_left: whether the ngrams should be left-padded :type pad_left: bool :param pad_right: whether the ngrams should be right-padded :type pad_right: bool :param left_pad_symbol: the symbol to use for left padding (default is None) :type left_pad_symbol: any :param right_pad_symbol: the symbol to use for right padding (default is None) :type right_pad_symbol: any :rtype: sequence or iter r)iterr)sequencenpad_left pad_rightleft_pad_symbolright_pad_symbolr3r3r4 pad_sequences rc cst||||||}g}x@|dkrVy t|}Wntk r@dSX|||d8}qWx&|D]}||t|V|d=q^WdS)a Return the ngrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import ngrams >>> list(ngrams([1,2,3,4,5], 3)) [(1, 2, 3), (2, 3, 4), (3, 4, 5)] Wrap with list for a list version of this function. Set pad_left or pad_right to true in order to get additional ngrams: >>> list(ngrams([1,2,3,4,5], 2, pad_right=True)) [(1, 2), (2, 3), (3, 4), (4, 5), (5, None)] >>> list(ngrams([1,2,3,4,5], 2, pad_right=True, right_pad_symbol='')) [(1, 2), (2, 3), (3, 4), (4, 5), (5, '')] >>> list(ngrams([1,2,3,4,5], 2, pad_left=True, left_pad_symbol='')) [('', 1), (1, 2), (2, 3), (3, 4), (4, 5)] >>> list(ngrams([1,2,3,4,5], 2, pad_left=True, pad_right=True, left_pad_symbol='', right_pad_symbol='')) [('', 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, '')] :param sequence: the source data to be converted into ngrams :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param pad_left: whether the ngrams should be left-padded :type pad_left: bool :param pad_right: whether the ngrams should be right-padded :type pad_right: bool :param left_pad_symbol: the symbol to use for left padding (default is None) :type left_pad_symbol: any :param right_pad_symbol: the symbol to use for right padding (default is None) :type right_pad_symbol: any :rtype: sequence or iter rNr)rnext StopIterationrNr) rrrrrrhistoryZ next_itemrr3r3r4ngramss+       rcks"xt|df|D] }|VqWdS)a Return the bigrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import bigrams >>> list(bigrams([1,2,3,4,5])) [(1, 2), (2, 3), (3, 4), (4, 5)] Use bigrams for a list version of this function. :param sequence: the source data to be converted into bigrams :type sequence: sequence or iter :rtype: iter(tuple) rFN)r)rkwargsrr3r3r4bigramssrcks"xt|df|D] }|VqWdS)a Return the trigrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import trigrams >>> list(trigrams([1,2,3,4,5])) [(1, 2, 3), (2, 3, 4), (3, 4, 5)] Use trigrams for a list version of this function. :param sequence: the source data to be converted into trigrams :type sequence: sequence or iter :rtype: iter(tuple) rN)r)rrrr3r3r4trigrams3srrcksJ|dkrt|}x4t||dD]"}xt||f|D] }|Vq4Wq WdS)a Returns all possible ngrams generated from a sequence of items, as an iterator. >>> sent = 'a b c'.split() >>> list(everygrams(sent)) [('a',), ('b',), ('c',), ('a', 'b'), ('b', 'c'), ('a', 'b', 'c')] >>> list(everygrams(sent, max_len=2)) [('a',), ('b',), ('c',), ('a', 'b'), ('b', 'c')] :param sequence: the source data to be converted into trigrams :type sequence: sequence or iter :param min_len: minimum length of the ngrams, aka. n-gram order/degree of ngram :type min_len: int :param max_len: maximum length of the ngrams (set to length of sequence by default) :type max_len: int :rtype: iter(tuple) rerN)r,ranger)rZmin_lenZmax_lenrrZngr3r3r4 everygramsGs rc ksd|ksd|krt||f|}t}xdt|||d|dD]L}|dd}|dd}x.t||dD]}|d|krxqf||VqfWq:WdS)a Returns all possible skipgrams generated from a sequence of items, as an iterator. Skipgrams are ngrams that allows tokens to be skipped. Refer to http://homepages.inf.ed.ac.uk/ballison/pdf/lrec_skipgrams.pdf >>> sent = "Insurgents killed in ongoing fighting".split() >>> list(skipgrams(sent, 2, 2)) [('Insurgents', 'killed'), ('Insurgents', 'in'), ('Insurgents', 'ongoing'), ('killed', 'in'), ('killed', 'ongoing'), ('killed', 'fighting'), ('in', 'ongoing'), ('in', 'fighting'), ('ongoing', 'fighting')] >>> list(skipgrams(sent, 3, 2)) [('Insurgents', 'killed', 'in'), ('Insurgents', 'killed', 'ongoing'), ('Insurgents', 'killed', 'fighting'), ('Insurgents', 'in', 'ongoing'), ('Insurgents', 'in', 'fighting'), ('Insurgents', 'ongoing', 'fighting'), ('killed', 'in', 'ongoing'), ('killed', 'in', 'fighting'), ('killed', 'ongoing', 'fighting'), ('in', 'ongoing', 'fighting')] :param sequence: the source data to be converted into trigrams :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param k: the skip distance :type k: int :rtype: iter(tuple) rrT)rrNrre)robjectrr) rrrrZSENTINELZngramheadtailZ skip_tailr3r3r4 skipgramsas   rc Cs|d}t|}d}d}t|dr6t|jjd}n"|dd|d}|dxT||kr||f}||d} || r|| \} } nzd} x^|t d| d| dkr| |} | } | dkrP|| d} | |dkrdSqW||kr | | f|| <| |kr6|| dks,t d| d}nZ| d||krL| S| |krv|| dkslt d| d}n| |kr| t| d}|d7}||f} || kr\dSq\WdS) a Return the line from the file with first word key. Searches through a sorted file using the binary search algorithm. :type file: file :param file: the file to be searched through. :type key: str :param key: the identifier we are searching for. rrr/rrFNz infinite loop) r,r`osstatr/st_sizeseektellrmaxZ discard_linereadlineAssertionError) filerOcacheZ cacheDepthZkeylenr:Z currentDepthr;Z lastStateZmiddleoffsetlineZ thisStater3r3r4binary_search_filesV                rrcCsddlm}|dkr@ytd}Wntk r>tdYnXt||d}t|}|dk rt}|jd|||d| t || t |t |dS)a Set the HTTP proxy for Python to download through. If ``proxy`` is None then tries to set proxy from environment or system settings. :param proxy: The HTTP proxy server to use. For example: 'http://proxy.example.com:3128/' :param user: The username to authenticate with. Use None to disable authentication. :param password: The password to authenticate with. r)compatNhttpz'Could not detect default proxy settings)Zhttpsr)ZrealmZuriuserZpasswd) ZnltkrrKeyErrorrbrr rZ add_passwordZ add_handlerrrr)proxyrZpasswordrZ proxy_handleropenerZpassword_managerr3r3r4 set_proxys rcCsd|d}t|rb|jr$|js.|d|_x|D]}t||dq4W|jrZ|js|||_n|r||jrv|js|||_dS)a Recursive function to indent an ElementTree._ElementInterface used for pretty printing. Run indent on elem and then output in the normal way. :param elem: element to be indented. will be modified. :type elem: ElementTree._ElementInterface :param level: level of indentation for this element :type level: nonnegative integer :rtype: ElementTree._ElementInterface :return: Contents of elem indented to reflect its structure r>z rN)r,textstripelementtree_indentr)elemlevelrr3r3r4rs   rcCsjd|kr|krbnnJd\}}x8tdt|||dD]}||9}||9}|d8}q:W||SdSdS)a9 This function is a fast way to calculate binomial coefficients, commonly known as nCk, i.e. the number of combinations of n things taken k at a time. (https://en.wikipedia.org/wiki/Binomial_coefficient). This is the *scipy.special.comb()* with long integer computation but this approximation is faster, see https://github.com/nltk/nltk/issues/1181 >>> choose(4, 2) 6 >>> choose(6, 2) 15 :param n: The number of things. :type n: int :param r: The number of times a thing is taken. :type r: int r)rrrN)rmin)rrZntokZktoktr3r3r4choose s r)r)rN)r=)rr=)rSrT)F)FFNN)FFNN)rre)Nr)r)EZ __future__rr)r*rqrUtypesr-r%Zbisectr itertoolsrrrr collectionsrrr Zsixr r r Zsix.moves.urllib.requestr rrrrrrZnltk.internalsrrZnltk.collectionsZ nltk.compatrr5r7r<rCrErGrIrKrLr]rdrrmrxr}rrrrrrrrrrrrrrrrr3r3r3r4sl   $   (    =  "   ( 9 +J *