ó xił\c @sŅdZddlmZddlZddlZddlZddlmZmZddl m Z ddl m Z m Z mZmZddlmZmZmZmZmZmZmZmZmZddlmZd d d d d dddddddg Zd;d<d=d>d?gZed„ƒZd e fd!„ƒYZ!ee!ƒd e"fd"„ƒYƒZ#d e#fd#„ƒYZ$de%fd$„ƒYZ&d%e"fd&„ƒYZ'e'ƒZ(d'e)fd(„ƒYZ*d)e"fd*„ƒYZ+d+„Z,d,e"fd-„ƒYZ-e-ƒZ.d.„Z/dd@d/„ƒYZ0defd0„ƒYZ1d1e!fd2„ƒYZ2ee2ƒd e#fd3„ƒYƒZ3d e"fd4„ƒYZ4d e3fd5„ƒYZ5d6„Z6d7e2fd8„ƒYZ7ee7ƒd9e3fd:„ƒYƒZ8dS(As“ pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2019 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. i’’’’(tprint_functionN(t apply_filterstFilter(tget_filter_by_name(tErrortTexttOthert _TokenType( t get_bool_optt get_int_optt get_list_opttmake_analysatort text_typet add_metaclasst iteritemstFuturet guess_decode(t regex_opttLexert RegexLexertExtendedRegexLexertDelegatingLexert LexerContexttincludetinherittbygroupstusingtthistdefaulttwordssļ»æsutf-8s’žsutf-32sž’sutf-32bes’žsutf-16sž’sutf-16becCsdS(Ng((tx((s-lib/python2.7/site-packages/pygments/lexer.pyt$tt LexerMetacBseZdZd„ZRS(s‚ This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. cCs9d|kr#t|dƒ|ds(R4t __class__R)(R9((s-lib/python2.7/site-packages/pygments/lexer.pyt__repr__ns   cKs5t|tƒs!t||}n|jj|ƒdS(s8 Add a new stream filter to this lexer. N(t isinstanceRRR3tappend(R9R:R4((s-lib/python2.7/site-packages/pygments/lexer.pyR8uscCsdS(s~ Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. N((ttext((s-lib/python2.7/site-packages/pygments/lexer.pyR"}R c sAtˆtƒsAˆjdkr3tˆƒ\‰}qcˆjdkr yddl}Wntk rqtdƒ‚nXd}xCtD];\}}ˆj|ƒrˆt |ƒj |dƒ}PqqW|dkr|j ˆd ƒ}ˆj |j dƒpõd dƒ}n|‰qcˆj ˆjƒ‰ˆjd ƒrcˆt d ƒ‰qcn"ˆjd ƒrcˆt d ƒ‰nˆj d d ƒ‰ˆj d d ƒ‰ˆjrŸˆjƒ‰nˆjrŗˆjd ƒ‰nˆjdkrŽˆjˆjƒ‰nˆjrˆjd ƒ rˆd 7‰n‡‡fd†} | ƒ} |s=t| ˆjˆƒ} n| S(s= Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. R1tchardeti’’’’NskTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/treplaceiR0sutf-8uļ»æs s s ic3s2x+ˆjˆƒD]\}}}||fVqWdS(N(tget_tokens_unprocessed(t_tttv(R9R@(s-lib/python2.7/site-packages/pygments/lexer.pytstreamer¾s(R>R R0RRAt ImportErrortNonet _encoding_mapt startswithtlentdecodetdetectR7RBR-tstripR,R/t expandtabsR.tendswithRR3( R9R@t unfilteredRDRAtdecodedtbomR0tencRGtstream((R9R@s-lib/python2.7/site-packages/pygments/lexer.pyt get_tokens‹sL         cCs t‚dS(sś Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text. In subclasses, implement this method as a generator to maximize effectiveness. N(tNotImplementedError(R9R@((s-lib/python2.7/site-packages/pygments/lexer.pyRCĘsN(R)R*R+RIR&taliasest filenamestalias_filenamest mimetypestpriorityR;R=R8R"R6RWRC(((s-lib/python2.7/site-packages/pygments/lexer.pyR3s    ;cBs#eZdZed„Zd„ZRS(s  This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. cKs;|||_|||_||_tj||dS(N(t root_lexertlanguage_lexertneedleRR;(R9t _root_lexert_language_lexert_needleR4((s-lib/python2.7/site-packages/pygments/lexer.pyR;Ūs cCsĶd}g}g}x}|jj|ƒD]i\}}}||jkrx|rk|jt|ƒ|fƒg}n||7}q%|j|||fƒq%W|r“|jt|ƒ|fƒnt||jj|ƒƒS(NR (R_RCR`R?RLt do_insertionsR^(R9R@tbufferedt insertionst lng_buffertiRERF((s-lib/python2.7/site-packages/pygments/lexer.pyRCįs"  (R)R*R+RR;RC(((s-lib/python2.7/site-packages/pygments/lexer.pyRŃs cBseZdZRS(sI Indicates that a state should include rules from another state. (R)R*R+(((s-lib/python2.7/site-packages/pygments/lexer.pyRųst_inheritcBseZdZd„ZRS(sC Indicates the a state should inherit from its superclass. cCsdS(NR((R9((s-lib/python2.7/site-packages/pygments/lexer.pyR=s(R)R*R+R=(((s-lib/python2.7/site-packages/pygments/lexer.pyRi’stcombinedcBs eZdZd„Zd„ZRS(s: Indicates a state combined from multiple states. cGstj||ƒS(N(ttupleR$(tclstargs((s-lib/python2.7/site-packages/pygments/lexer.pyR$scGsdS(N((R9Rm((s-lib/python2.7/site-packages/pygments/lexer.pyR;s(R)R*R+R$R;(((s-lib/python2.7/site-packages/pygments/lexer.pyRj s t _PseudoMatchcBsMeZdZd„Zdd„Zdd„Zdd„Zd„Zd„Z RS(s: A pseudo match object constructed from a string. cCs||_||_dS(N(t_textt_start(R9tstartR@((s-lib/python2.7/site-packages/pygments/lexer.pyR;s cCs|jS(N(Rp(R9targ((s-lib/python2.7/site-packages/pygments/lexer.pyRqscCs|jt|jƒS(N(RpRLRo(R9Rr((s-lib/python2.7/site-packages/pygments/lexer.pytend"scCs|rtdƒ‚n|jS(Ns No such group(t IndexErrorRo(R9Rr((s-lib/python2.7/site-packages/pygments/lexer.pytgroup%scCs |jfS(N(Ro(R9((s-lib/python2.7/site-packages/pygments/lexer.pytgroups*scCsiS(N((R9((s-lib/python2.7/site-packages/pygments/lexer.pyt groupdict-sN( R)R*R+R;RIRqRsRuRvRw(((s-lib/python2.7/site-packages/pygments/lexer.pyRns     csd‡fd†}|S(sL Callback that yields multiple actions for each group in the match. c3sxśtˆƒD]ģ\}}|dkr+q q t|ƒtkrw|j|dƒ}|rł|j|dƒ||fVqłq |j|dƒ}|dk r |rµ|j|dƒ|_nxA||t|j|dƒ|ƒ|ƒD]}|rŽ|VqŽqŽWq q W|r|jƒ|_ndS(Ni( t enumerateRIR#RRuRqtposRnRs(tlexertmatchtctxRhtactiontdatatitem(Rm(s-lib/python2.7/site-packages/pygments/lexer.pytcallback5s" !  &N(RI(RmR€((Rms-lib/python2.7/site-packages/pygments/lexer.pyR1st_ThiscBseZdZRS(sX Special singleton used for indicating the caller class. Used by ``using``. (R)R*R+(((s-lib/python2.7/site-packages/pygments/lexer.pyRKsc s–i‰dˆkrVˆjdƒ}t|ttfƒrC|ˆdtlistRkRRI(RŠR‰R‡R€((RŠRˆR‰s-lib/python2.7/site-packages/pygments/lexer.pyRSs     cBseZdZd„ZRS(sŃ Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 cCs ||_dS(N(R‚(R9R‚((s-lib/python2.7/site-packages/pygments/lexer.pyR;s(R)R*R+R;(((s-lib/python2.7/site-packages/pygments/lexer.pyR…scBs&eZdZddd„Zd„ZRS(s• Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 R cCs||_||_||_dS(N(Rtprefixtsuffix(R9RRRŽ((s-lib/python2.7/site-packages/pygments/lexer.pyR;˜s  cCst|jd|jd|jƒS(NRRŽ(RRRRŽ(R9((s-lib/python2.7/site-packages/pygments/lexer.pyR7s(R)R*R+R;R7(((s-lib/python2.7/site-packages/pygments/lexer.pyR‘stRegexLexerMetacBsPeZdZd„Zd„Zd„Zd„Zdd„Zd„Z d„Z RS( sw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. cCs1t|tƒr|jƒ}ntj||ƒjS(sBPreprocess the regular expression component of a token definition.(R>RR7tretcompileR{(RltregextrflagsR‚((s-lib/python2.7/site-packages/pygments/lexer.pyt_process_regex§scCs5t|ƒtks1t|ƒs1td|fƒ‚|S(s5Preprocess the token component of a token definition.s2token type must be simple type or callable, not %r(R#RtcallabletAssertionError(Rlttoken((s-lib/python2.7/site-packages/pygments/lexer.pyt_process_token­s!cCskt|tƒrz|dkrdS||kr2|fS|dkrB|S|d dkrat|dƒ Stsgtd|ƒ‚nķt|tƒrd|j}|jd7_g}xF|D]>}||ksŌtd |ƒ‚|j|j|||ƒƒq²W|||<|fSt|t ƒrQx6|D].}||ks|d kstd |ƒ‚qW|Stsgtd |ƒ‚d S(s=Preprocess the state transition action of a token definition.s#popi’’’’s#pushis#pop:sunknown new state %rs_tmp_%discircular state ref %rsunknown new state sunknown new state def %rN(s#pops#push( R>tstrtintR6R–Rjt_tmpnametextendt_process_stateRk(Rlt new_statet unprocessedt processedt tmp_statetitokenstistate((s-lib/python2.7/site-packages/pygments/lexer.pyt_process_new_state³s6        c Cst|ƒtks"td|ƒ‚|ddksBtd|ƒ‚||krV||Sg}||<|j}x›||D]}t|tƒrŃ||ks©td|ƒ‚|j|j||t|ƒƒƒqxnt|tƒręqxnt|t ƒr5|j |j ||ƒ}|j t jdƒjd |fƒqxnt|ƒtksWtd|ƒ‚y|j|d||ƒ}Wn3tk r©} td|d||| fƒ‚nX|j|d ƒ} t|ƒd krŲd }n|j |d ||ƒ}|j || |fƒqxW|S( s%Preprocess a single state definition.swrong state name %rit#sinvalid state name %rscircular state reference %rR swrong rule def %rs+uncompilable regex %r in state %r of %r: %siiN(R#R™R–tflagsR>RRœRRiRR¤R‚R?RR‘R{RIRkR”t Exceptiont ValueErrorR˜RL( RlRŸR R‚ttokensR“ttdefRžtrexterrR—((s-lib/python2.7/site-packages/pygments/lexer.pyRÖs>"   ""   cCsRi}|j|<|p!|j|}x't|ƒD]}|j|||ƒq1W|S(s-Preprocess a dictionary of token definitions.(t _all_tokensR©RŒR(RlR&t tokendefsR R‚((s-lib/python2.7/site-packages/pygments/lexer.pytprocess_tokendefs c Csi}i}x |jD]}|jjdiƒ}xćt|ƒD]Õ\}}|j|ƒ}|dkr©|||nX|||n|j|dƒ}|dkrĶq>n||||d+y|jtƒ} Wntk rq>X|| ||WqW|S(s Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. R©iN( t__mro__t__dict__R7RRItindexRRØR‹( RlR©t inheritabletcttoksR‚titemstcuritemst inherit_ndxt new_inh_ndx((s-lib/python2.7/site-packages/pygments/lexer.pyt get_tokendefs s0       cOsmd|jkrZi|_d|_t|dƒr<|jr<qZ|jd|jƒƒ|_ntj |||ŽS(s:Instantiate cls after preprocessing its token definitions.t_tokensittoken_variantsR ( R±R­R›thasattrR¼RÆRŗR»R#t__call__(RlRmtkwds((s-lib/python2.7/site-packages/pygments/lexer.pyR¾;s  N( R)R*R+R”R˜R¤RRIRÆRŗR¾(((s-lib/python2.7/site-packages/pygments/lexer.pyR”s   # ,  1cBs)eZdZejZiZdd„ZRS(s± Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. R„c csLd}|j}t|ƒ}||d}xx|D]—\}}} |||ƒ} | r3|d k r®t|ƒtkrŒ||| jƒfVq®x||| ƒD] } | VqœWn| jƒ}| d k rĘt| tƒrDxŻ| D]a} | dkrt |ƒdkr=|j ƒq=qÜ| dkr0|j |dƒqÜ|j | ƒqÜWnqt| t ƒrt | ƒt |ƒkru|d3qµ|| 3n6| dkrŸ|j |dƒntsµtd| ƒ‚||d}nPq3q3Wya||dkrdg}|d}|td fV|d7}w,n|t||fV|d7}Wq,tk rCPq,Xq,Wd S( s} Split ``text`` into (tokentype, text) pairs. ``stack`` is the inital stack (default: ``['root']``) ii’’’’s#popis#pushswrong state def: %rs R„u N(R»RŒRIR#RRuRsR>RkRLR‹R?RštabsR6R–RRRt( R9R@RƒRyR®t statestackt statetokenstrexmatchR}RžtmRR‚((s-lib/python2.7/site-packages/pygments/lexer.pyRChsV                (R„(R)R*R+Rt MULTILINER¦R©RC(((s-lib/python2.7/site-packages/pygments/lexer.pyRIs cBs&eZdZddd„Zd„ZRS(s9 A helper object that holds lexer position data. cCs=||_||_|p!t|ƒ|_|p3dg|_dS(NR„(R@RyRLRsRƒ(R9R@RyRƒRs((s-lib/python2.7/site-packages/pygments/lexer.pyR;¬s  cCsd|j|j|jfS(NsLexerContext(%r, %r, %r)(R@RyRƒ(R9((s-lib/python2.7/site-packages/pygments/lexer.pyR=²sN(R)R*R+RIR;R=(((s-lib/python2.7/site-packages/pygments/lexer.pyR§scBseZdZddd„ZRS(sE A RegexLexer that uses a context object to store its state. c csź|j}|s+t|dƒ}|d}n |}||jd}|j}x˜x‘|D]ä\}}}|||j|jƒ} | rU|d k rt|ƒtkrÉ|j|| j ƒfV| jƒ|_qx||| |ƒD] } | VqÜW|s||jd}qn|d k r5t |t ƒr”xū|D]p} | dkrdt |jƒdkrš|jj ƒqšq*| dkrŠ|jj|jdƒq*|jj| ƒq*Wn€t |tƒråt|ƒt |jƒkrŲ|jd3q!|j|3n<|dkr |jj|jdƒnts!td|ƒ‚||jd}nPqUqUWy|j|jkrVPn||jdkr„dg|_|d}|jtd fV|jd7_wNn|jt||jfV|jd7_WqNtk rįPqNXqNWd S( s Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. iR„i’’’’s#popis#pushswrong state def: %rs u N(R»RRƒR@RyRsRIR#RRuR>RkRLR‹R?RšRĄR‚R6R–RRRt( R9R@tcontextR®R|RĀRĆR}RžRÄRR‚((s-lib/python2.7/site-packages/pygments/lexer.pyRC¼sd               N(R)R*R+RIRC(((s-lib/python2.7/site-packages/pygments/lexer.pyR·sccs’t|ƒ}yt|ƒ\}}Wn(tk rLx|D] }|Vq9WdSXd}t}x|D]\}}} |dkr„|}nd} xĀ|rN|t| ƒ|krN| | ||!} ||| fV|t| ƒ7}x5|D]-\} } }|| |fV|t|ƒ7}qßW||} yt|ƒ\}}Wqtk rJt}PqXqW||| | fV|t| ƒ| 7}q`Wx|rś|p‹d}x5|D]-\}}} ||| fV|t| ƒ7}q•Wyt|ƒ\}}Wq|tk röt}Pq|Xq|WdS(sg Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. Ni(titertnextt StopIterationRIR5RLR6(RfR©R²R¢RtrealpostinsleftRhRERFtoldittmpvaltit_indextit_tokentit_valuetp((s-lib/python2.7/site-packages/pygments/lexer.pyRdsL            tProfilingRegexLexerMetacBseZdZd„ZRS(s>Metaclass for ProfilingRegexLexer, collects regex timing info.csmt|tƒr3t|jd|jd|jƒ‰n|‰tjˆ|ƒ‰tj‡‡‡‡fd†}|S(NRRŽcszˆjdjˆˆfddgƒ}tjƒ}ˆj|||ƒ}tjƒ}|dcd7<|dc||7<|S(Ni’’’’igi(t _prof_datat setdefaultttimeR{(R@Rytendpostinfott0trestt1(RltcompiledR«R‚(s-lib/python2.7/site-packages/pygments/lexer.pyt match_funcJs%  ( R>RRRRŽRR‘tsystmaxsize(RlR’R“R‚RÜ((RlRŪR«R‚s-lib/python2.7/site-packages/pygments/lexer.pyR”Bs(R)R*R+R”(((s-lib/python2.7/site-packages/pygments/lexer.pyRŅ?stProfilingRegexLexercBs&eZdZgZdZdd„ZRS(sFDrop-in replacement for RegexLexer that does profiling of its regexes.iR„c#sˆjjjiƒx"tjˆ||ƒD] }|Vq)Wˆjjjƒ}td„|jƒDƒd‡fd†dtƒ}t d„|Dƒƒ}t ƒt dˆjj t |ƒ|fƒt ddƒt d dƒt d dƒx|D]}t d |ƒqģWt ddƒdS(Ncssa|]W\\}}\}}|t|ƒjdƒjddƒd |d|d||fVqdS(su's\\s\iAičN(treprRORB(t.0R‡trtnRE((s-lib/python2.7/site-packages/pygments/lexer.pys bstkeycs |ˆjS(N(t_prof_sort_index(R(R9(s-lib/python2.7/site-packages/pygments/lexer.pyReR treversecss|]}|dVqdS(iN((RįR((s-lib/python2.7/site-packages/pygments/lexer.pys gss2Profiling result for %s lexing %d chars in %.3f mst=ins$%-20s %-64s ncalls tottime percallR‚R’t-s%-20s %-65s %5d %8.4f %8.4f(R‚R’( R<RÓR?RRCR‹tsortedR¶R5tsumtprintR)RL(R9R@RƒttoktrawdataR~t sum_totalR(((R9s-lib/python2.7/site-packages/pygments/lexer.pyRC\s$    (R„(R)R*R+RÓRåRC(((s-lib/python2.7/site-packages/pygments/lexer.pyRßUs(sļ»æsutf-8(s’žsutf-32(sž’sutf-32be(s’žsutf-16(sž’sutf-16be((9R+t __future__RRRŻRÕtpygments.filterRRtpygments.filtersRtpygments.tokenRRRRt pygments.utilRR R R R R RRRtpygments.regexoptRt__all__RJt staticmethodt_default_analyseR#R!tobjectRRR™RRiRRkRjRnRRRRRRRRRRRdRŅRß(((s-lib/python2.7/site-packages/pygments/lexer.pyt sT   "@    '     2 Ø ]I ?