B 18™\0kã@sdZdZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZdd lmZd!d d „ZGd d„deƒZdd„Zdd„Zdd„Zdd„Zdd„ZGdd„deƒZGdd„deƒZGdd„deƒZ Gdd „d eƒZ!dS)"zÛRefactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. z#Guido van Rossum éN)Úchainé)ÚdriverÚtokenizeÚtoken)Ú find_root)ÚpytreeÚpygram)Ú btm_matcherTcCstt|ggdgƒ}tj |j¡}g}xLtt |¡ƒD]:}| d¡r2| d¡r2|rZ|dd…}|  |dd…¡q2W|S)zEReturn a sorted list of all available fix names in the given package.Ú*Úfix_z.pyéNéýÿÿÿ) Ú __import__ÚosÚpathÚdirnameÚ__file__ÚsortedÚlistdirÚ startswithÚendswithÚappend)Ú fixer_pkgZ remove_prefixÚpkgZ fixer_dirZ fix_namesÚname©rúT/oak/stanford/groups/akundaje/marinovg/programs/Python-3.7.3/Lib/lib2to3/refactor.pyÚget_all_fix_namess rc@s eZdZdS)Ú _EveryNodeN)Ú__name__Ú __module__Ú __qualname__rrrrr+srcCsšt|tjtjfƒr(|jdkr t‚|jhSt|tjƒrH|jrDt|jƒSt‚t|tj ƒrŠt ƒ}x*|jD] }x|D]}|  t|ƒ¡qlWqbW|St d|ƒ‚dS)zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) Ú isinstancerÚ NodePatternÚ LeafPatternÚtyperÚNegatedPatternÚcontentÚ_get_head_typesÚWildcardPatternÚsetÚupdateÚ Exception)ÚpatÚrÚpÚxrrrr)/s      r)c CsÊt t¡}g}x„|D]|}|jrjyt|jƒ}Wntk rJ| |¡YqXxB|D]}|| |¡qRWq|jdk r†||j |¡q| |¡qWx,tt j j   ¡t j j ƒD]}|| |¡q¬Wt|ƒS)z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N)Ú collectionsÚ defaultdictÚlistÚpatternr)rrZ _accept_typerr Úpython_grammarZ symbol2numberÚvaluesÚtokensÚextendÚdict)Z fixer_listZ head_nodesZeveryÚfixerZheadsZ node_typerrrÚ_get_headnode_dictKs"    r<cs‡fdd„tˆdƒDƒS)zN Return the fully qualified names for fixers in the package pkg_name. csg|]}ˆd|‘qS)Ú.r)Ú.0Úfix_name)Úpkg_namerrú hsz+get_fixers_from_package..F)r)r@r)r@rÚget_fixers_from_packageds rBcCs|S)Nr)ÚobjrrrÚ _identityksrDcsVd}t t |¡j¡‰‡fdd„}ttjtjtj hƒ}t ƒ}yúxô|ƒ\}}||krVq@q@|tj krl|rfPd}q@|tj kr.|dkr.|ƒ\}}|tj ksž|dkr P|ƒ\}}|tj ks¼|dkr¾P|ƒ\}}|tj krä|dkrä|ƒ\}}xJ|tj kr*| |¡|ƒ\}}|tj ks|d krP|ƒ\}}qæWq@Pq@WWntk rLYnXt|ƒS) NFcstˆƒ}|d|dfS)Nrr)Únext)Útok)ÚgenrrÚadvancersz(_detect_future_features..advanceTÚfromZ __future__Úimportú(ú,)rÚgenerate_tokensÚioÚStringIOÚreadlineÚ frozensetrÚNEWLINEÚNLÚCOMMENTr+ÚSTRINGÚNAMEÚOPÚaddÚ StopIteration)ÚsourceZhave_docstringrHÚignoreÚfeaturesÚtpÚvaluer)rGrÚ_detect_future_featuresosD          r_c@seZdZdZdS)Ú FixerErrorzA fixer could not be loaded.N)r r!r"Ú__doc__rrrrr`—sr`c@säeZdZdddœZdZdZd4dd„Zdd „Zd d „Zd d „Z dd„Z dd„Z d5dd„Z d6dd„Z dd„Zd7dd„Zdd„Zd8dd„Zdd„Zd d!„Zd9d"d#„Zd:d$d%„Zd&Zd'Zd(d)„Zd*d+„Zd,d-„Zd.d/„Zd0d1„Zd2d3„ZdS);ÚRefactoringToolF)Úprint_functionÚwrite_unchanged_filesZFixr NcCs2||_|p g|_|j ¡|_|dk r0|j |¡|jdrDtj|_ntj |_|j  d¡|_ g|_ t  d¡|_g|_d|_tj|jtj|jd|_| ¡\|_|_g|_t ¡|_g|_g|_xXt|j|jƒD]F}|j rä|j !|¡qÌ||jkrü|j "|¡qÌ||jkrÌ|j "|¡qÌWt#|jƒ|_$t#|jƒ|_%dS)zÑInitializer. Args: fixer_names: a list of fixers to import options: a dict with configuration. explicit: a list of fixers to run even if they are explicit. NrcrdrbF)ÚconvertÚlogger)&ÚfixersÚexplicitÚ_default_optionsÚcopyÚoptionsr,r Ú!python_grammar_no_print_statementÚgrammarr6ÚgetrdÚerrorsÚloggingÚ getLoggerrfÚ fixer_logÚwroterÚDriverrreÚ get_fixersÚ pre_orderÚ post_orderÚfilesÚbmZ BottomMatcherÚBMZ bmi_pre_orderZbmi_post_orderrZ BM_compatibleZ add_fixerrr<Úbmi_pre_order_headsÚbmi_post_order_heads)ÚselfZ fixer_namesrkrhr;rrrÚ__init__£s<           zRefactoringTool.__init__c Cs^g}g}x(|jD]}t|iidgƒ}| dd¡d}| |j¡rV|t|jƒd…}| d¡}|jd dd „|Dƒ¡}yt ||ƒ}Wn&t k r®t d ||fƒd‚YnX||j |j ƒ} | jræ|jd k ræ||jkræ| d |¡q| d |¡| jdkr | | ¡q| jdkr"| | ¡qt d| jƒ‚qWt d¡} |j| d|j| d||fS)aInspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r r=réÿÿÿÿNÚ_ÚcSsg|] }| ¡‘qSr)Útitle)r>r0rrrrAçsz.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sÚpreÚpostzIllegal fixer order: %rZ run_order)Úkey)rgrÚrsplitrÚ FILE_PREFIXÚlenÚsplitÚ CLASS_PREFIXÚjoinÚgetattrÚAttributeErrorr`rkrrrhÚ log_messageÚ log_debugÚorderrÚoperatorÚ attrgetterÚsort) r}Zpre_order_fixersZpost_order_fixersZ fix_mod_pathÚmodr?ÚpartsÚ class_nameZ fix_classr;Zkey_funcrrrru×s8            zRefactoringTool.get_fixerscOs‚dS)zCalled when an error occurs.Nr)r}ÚmsgÚargsÚkwdsrrrÚ log_errorÿszRefactoringTool.log_errorcGs|r ||}|j |¡dS)zHook to log a message.N)rfÚinfo)r}r—r˜rrrrŽszRefactoringTool.log_messagecGs|r ||}|j |¡dS)N)rfÚdebug)r}r—r˜rrrr szRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a refactored file.Nr)r}Úold_textÚnew_textÚfilenameÚequalrrrÚ print_outputszRefactoringTool.print_outputcCs<x6|D].}tj |¡r&| |||¡q| |||¡qWdS)z)Refactor a list of files and directories.N)rrÚisdirÚ refactor_dirÚ refactor_file)r}ÚitemsÚwriteÚ doctests_onlyZ dir_or_filerrrÚrefactors  zRefactoringTool.refactorc Cs¢tjd}x’t |¡D]„\}}}| d|¡| ¡| ¡xF|D]>}| d¡sBtj |¡d|krBtj ||¡} |  | ||¡qBWdd„|Dƒ|dd…<qWdS)zÄDescends down a directory and refactor every Python file found. Python files are assumed to have a .py extension. Files and subdirectories starting with '.' are skipped. ÚpyzDescending into %sr=rcSsg|]}| d¡s|‘qS)r=)r)r>ÚdnrrrrA.sz0RefactoringTool.refactor_dir..N) rÚextsepÚwalkrr“rrÚsplitextr‹r¤) r}Zdir_namer¦r§Zpy_extÚdirpathÚdirnamesÚ filenamesrÚfullnamerrrr£s    zRefactoringTool.refactor_dirc CsŠyt|dƒ}Wn0tk r>}z| d||¡dSd}~XYnXzt |j¡d}Wd| ¡Xtj|d|dd}| ¡|fSQRXdS) zG Do our best to decode a Python source file correctly. ÚrbzCan't open %s: %s)NNNrr/r)ÚencodingÚnewline) ÚopenÚOSErrorršrÚdetect_encodingrPÚcloserNÚread)r}rŸÚfÚerrr³rrrÚ_read_python_source0s z#RefactoringTool._read_python_sourcecCsº| |¡\}}|dkrdS|d7}|rn| d|¡| ||¡}|jsL||kr`| |||||¡q¶| d|¡nH| ||¡}|jsŠ|rª|jrª|jt|ƒdd…|||dn | d|¡dS)zRefactors a file.NÚ zRefactoring doctests in %szNo doctest changes in %sr)r¦r³zNo changes in %s)r¼rÚrefactor_docstringrdÚprocessed_fileÚrefactor_stringÚ was_changedÚstr)r}rŸr¦r§Úinputr³ÚoutputÚtreerrrr¤@s    zRefactoringTool.refactor_filec Cs–t|ƒ}d|krtj|j_zLy|j |¡}Wn6tk rb}z| d||jj |¡dSd}~XYnXWd|j|j_X||_ |  d|¡|  ||¡|S)aFRefactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. rczCan't parse %s: %s: %sNzRefactoring %s) r_r rlrrmÚ parse_stringr-ršÚ __class__r Úfuture_featuresrÚ refactor_tree)r}Údatarr\rÅr»rrrrÀWs     zRefactoringTool.refactor_stringcCsŒtj ¡}|rN| d¡| |d¡}|js2||krB| |d|¡qˆ| d¡n:| |d¡}|jsj|r~|jr~| t |ƒd|¡n | d¡dS)NzRefactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin) ÚsysÚstdinr¹rr¾rdr¿rÀrÁrÂ)r}r§rÃrÄrÅrrrÚrefactor_stdinrs     zRefactoringTool.refactor_stdinc Csüx"t|j|jƒD]}| ||¡qW| |j| ¡¡| |j| ¡¡|j |  ¡¡}xtt |  ¡ƒrÎx^|jj D]P}||krv||rv||j tjjdd|jr¼||j tjjdxt||ƒD]ø}|||krê|| |¡y t|ƒWntk rwÌYnX|jr&||jkr&qÌ| |¡}|rÌ| ||¡}|dk rÌ| |¡x,| ¡D] }|jsng|_|j |¡q\W|j |  ¡¡}x2|D]*} | |kr¬g|| <||  || ¡q–WqÌWqvWq\Wx$t|j|jƒD]}| ||¡qàW|jS)aÏRefactors a parse tree (modifying the tree in place). For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches. Args: tree: a pytree.Node instance representing the root of the tree to be refactored. name: a human-readable name for this tree. Returns: True if the tree was modified, False otherwise. T)r…Úreverse)r…N)rrvrwZ start_treeÚ traverse_byr{r|rzÚrunÚleavesÚanyr7rgr“rÚBaseÚdepthZkeep_line_orderÚ get_linenor4ÚremoverÚ ValueErrorÚfixers_appliedÚmatchÚ transformÚreplacerr9Z finish_treerÁ) r}rÅrr;Z match_setÚnodeÚresultsÚnewZ new_matchesZfxrrrrrÉ‚sJ       $zRefactoringTool.refactor_treecCs^|sdSxP|D]H}xB||jD]4}| |¡}|r| ||¡}|dk r| |¡|}qWqWdS)aTraverse an AST, applying a set of fixers to each node. This is a helper method for refactor_tree(). Args: fixers: a list of fixer instances. traversal: a generator that yields AST nodes. Returns: None N)r&rÙrÚrÛ)r}rgZ traversalrÜr;rÝrÞrrrrÏÑs     zRefactoringTool.traverse_bycCs†|j |¡|dkr.| |¡d}|dkr.dS||k}| ||||¡|r`| d|¡|js`dS|rv| ||||¡n | d|¡dS)zR Called when a file has been refactored and there may be changes. NrzNo changes to %szNot writing changes to %s)rxrr¼r¡rrdÚ write_file)r}ržrŸrr¦r³r rrrr¿ès  zRefactoringTool.processed_filec Cs®ytj|d|dd}Wn0tk rF}z| d||¡dSd}~XYnX|Fy| |¡Wn0tk rŒ}z| d||¡Wdd}~XYnXWdQRX| d|¡d|_dS) zÑWrites a string to a file. It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set. Úwr)r³r´zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rNrµr¶ršr¦rrs)r}ržrŸrr³Úfpr»rrrrßýs* zRefactoringTool.write_filez>>> z... c Csg}d}d}d}d}xØ|jddD]È}|d7}| ¡ |j¡r€|dk r\| | ||||¡¡|}|g}| |j¡} |d| …}q"|dk rº| ||j¡s®|||j ¡dkrº|  |¡q"|dk rØ| | ||||¡¡d}d}|  |¡q"W|dk r| | ||||¡¡d  |¡S)aËRefactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) NrT)Úkeependsrr½r) Ú splitlinesÚlstriprÚPS1r9Úrefactor_doctestÚfindÚPS2Úrstriprr‹) r}rÃrŸÚresultÚblockZ block_linenoÚindentÚlinenoÚlineÚirrrr¾s:          z"RefactoringTool.refactor_docstringc s*yˆ ||ˆ¡}Wnftk rx}zHˆj tj¡rRx|D]}ˆ d| d¡¡q8Wˆ d|||j j |¡|Sd}~XYnXˆ  ||¡r&t |ƒj dd}|d|d…||dd…} }| dg|dksÔt| ƒ‚|d d¡sò|dd7<ˆˆj| d ¡g}|r&|‡‡fd d „|Dƒ7}|S) zÞRefactors one doctest. A doctest is given as a block of lines, the first of which starts with ">>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %sr½z+Can't parse docstring in %s line %s: %s: %sNT)rârrrcsg|]}ˆˆj|‘qSr)rè)r>rî)rìr}rrrAZsz4RefactoringTool.refactor_doctest..)Ú parse_blockr-rfÚ isEnabledForrpÚDEBUGrréršrÇr rÉrÂrãÚAssertionErrorrråÚpop) r}rërírìrŸrÅr»rîrÞZclippedr)rìr}rræ@s& "z RefactoringTool.refactor_doctestcCsÒ|jr d}nd}|js$| d|¡n&| d|¡x|jD]}| |¡q8W|jrt| d¡x|jD]}| |¡qbW|jrÎt|jƒdkr”| d¡n| dt|jƒ¡x&|jD]\}}}|j|f|ž|Žq®WdS) NÚwerez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rsrxrŽrrrorˆ)r}rõÚfileÚmessager—r˜r™rrrÚ summarize]s$     zRefactoringTool.summarizecCs"|j | |||¡¡}tƒ|_|S)z³Parses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )rZ parse_tokensÚ wrap_toksrQrÈ)r}rërírìrÅrrrrðtszRefactoringTool.parse_blockc csht | ||¡j¡}xN|D]F\}}\}}\} } } ||d7}| |d7} ||||f| | f| fVqWdS)z;Wraps a tokenize stream to systematically modify start/end.rN)rrMÚ gen_linesÚ__next__) r}rërírìr8r&r^Zline0Zcol0Úline1Zcol1Z line_textrrrrù~s   zRefactoringTool.wrap_toksccs€||j}||j}|}xV|D]N}| |¡r@|t|ƒd…Vn(|| ¡dkrXdVntd||fƒ‚|}qWx dVqrWdS)z–Generates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nr½zline=%r, prefix=%rr)rårèrrˆréró)r}rërìÚprefix1Zprefix2ÚprefixrîrrrrúŒs    zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N)r r!r"rirŠr‡r~ruršrŽrr¡r¨r£r¼r¤rÀrÍrÉrÏr¿rßrårèr¾rærørðrùrúrrrrrb›s: 4(   O  + rbc@s eZdZdS)ÚMultiprocessingUnsupportedN)r r!r"rrrrrÿ srÿcsBeZdZ‡fdd„Zd ‡fdd„ Z‡fdd„Z‡fd d „Z‡ZS) ÚMultiprocessRefactoringToolcs"tt|ƒj||Žd|_d|_dS)N)Úsuperrr~ÚqueueÚ output_lock)r}r˜Úkwargs)rÇrrr~¦sz$MultiprocessRefactoringTool.__init__Frc s|dkrttˆƒ |||¡Sy ddl‰Wntk r@t‚YnXˆjdk rTtdƒ‚ˆ ¡ˆ_ˆ  ¡ˆ_ ‡‡fdd„t |ƒDƒ}z.x|D] }|  ¡qˆWttˆƒ |||¡Wdˆj  ¡xt |ƒD]}ˆj d¡qÄWx|D]}| ¡rÞ|  ¡qÞWdˆ_XdS)Nrrz already doing multiple processescsg|]}ˆjˆjd‘qS))Útarget)ÚProcessÚ_child)r>rï)Úmultiprocessingr}rrrA¸sz8MultiprocessRefactoringTool.refactor..)rrr¨rÚ ImportErrorrÿrÚ RuntimeErrorÚ JoinableQueueÚLockrÚrangeÚstartr‹ÚputÚis_alive)r}r¥r¦r§Z num_processesÚ processesr0rï)rÇ)rr}rr¨«s2               z$MultiprocessRefactoringTool.refactorcsR|j ¡}xB|dk rL|\}}ztt|ƒj||ŽWd|j ¡X|j ¡}q WdS)N)rrnrrr¤Ú task_done)r}Ztaskr˜r)rÇrrrÈs     z"MultiprocessRefactoringTool._childcs2|jdk r|j ||f¡ntt|ƒj||ŽSdS)N)rrrrr¤)r}r˜r)rÇrrr¤Ós  z)MultiprocessRefactoringTool.refactor_file)FFr)r r!r"r~r¨rr¤Ú __classcell__rr)rÇrr¤s   r)T)"raÚ __author__rNrrËrpr‘r2Ú itertoolsrZpgen2rrrZ fixer_utilrrrr r ryrr-rr)r<rBrDr_r`ÚobjectrbrÿrrrrrÚ s6    (