`]c @ sdZddlmZmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlmZddlmZyzeZddlmZmZmZmZmZmZmZmZmZddlm Z ddl!m"Z"dd l#m$Z$Wne%k rSe&Ze'ZnXdd l(m)Z)m*Z*dd l+m,Z,dd l-m.Z.dd l/m0Z0m1Z1ddl2Z2ddl3m4Z4e4de5fdYZ6e4de5fdYZ7de5fdYZ8de8fdYZ9de8fdYZ:de8fdYZ;de8fdYZ<de8fdYZ=de8fd YZ>d!e8fd"YZ?d#e8fd$YZ@d%e8fd&YZAd'e8fd(YZBd)e8fd*YZCd+e8fd,YZDd-e8fd.YZEd/e5fd0YZFd1e5fd2YZGd3e5fd4YZHd5ZId6ZJed7ZKed8ZLd9ZMd:d;ZNd<ZOd=ZPd>ZQd?ZReFZSeSjTZTd@ZUdAZVdBZWeXdCkrddDlYmZZZeZZ[e[j\dEdFdGdHdIdJdKdLe[j\dMdNdGdOdPdQdRe&dIdSe[j\dTdUdGdVdPdQdRe&dIdWe[j\dXdYdGdZdPdQdRe&dId[e[j\d\d]dGd^dRej]j^d_dId`e[j_\Z`ZaeFdae`jbZceartx]eaD]UZdecjTdbeddce`jedde`jfdee`jgdfe`jhZieie&kofe`jhrmPnqWn.ecjTdce`jedde`jfdee`jgdfe`jhndS(guH The NLTK corpus and module downloader. This module defines several interfaces which can be used to download corpora, models, and other data packages that can be used with NLTK. Downloading Packages ==================== If called with no arguments, ``download()`` will display an interactive interface which can be used to download and install new packages. If Tkinter is available, then a graphical interface will be shown, otherwise a simple text interface will be provided. Individual packages can be downloaded by calling the ``download()`` function with a single argument, giving the package identifier for the package that should be downloaded: >>> download('treebank') # doctest: +SKIP [nltk_data] Downloading package 'treebank'... [nltk_data] Unzipping corpora/treebank.zip. NLTK also provides a number of "package collections", consisting of a group of related packages. To download all packages in a colleciton, simply call ``download()`` with the collection's identifier: >>> download('all-corpora') # doctest: +SKIP [nltk_data] Downloading package 'abc'... [nltk_data] Unzipping corpora/abc.zip. [nltk_data] Downloading package 'alpino'... [nltk_data] Unzipping corpora/alpino.zip. ... [nltk_data] Downloading package 'words'... [nltk_data] Unzipping corpora/words.zip. Download Directory ================== By default, packages are installed in either a system-wide directory (if Python has sufficient access to write to it); or in the current user's home directory. However, the ``download_dir`` argument may be used to specify a different installation target, if desired. See ``Downloader.default_download_dir()`` for more a detailed description of how the default download directory is chosen. NLTK Download Server ==================== Before downloading any packages, the corpus and module downloader contacts the NLTK download server, to retrieve an index file describing the available packages. By default, this index file is loaded from ``https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml``. If necessary, it is possible to create a new ``Downloader`` object, specifying a different URL for the package index file. Usage:: python nltk/downloader.py [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS or:: python -m nltk.downloader [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS i(tprint_functiontdivisiontunicode_literalsN(tmd5(t ElementTree( tTktFrametLabeltEntrytButtontCanvastMenutIntVartTclError(t showerror(tTable(tShowText(t string_typest text_type(tinput(turlopen(t HTTPErrortURLError(tpython_2_unicode_compatibletPackagec B sYeZdZdddddddddded ZedZdZdZ RS(u4 A directory entry for a downloadable package. These entries are extracted from the XML index file that is downloaded by ``Downloader``. Each package consists of a single file; but if that file is a zip file, then it can be automatically decompressed when the package is installed. uuUnknowncK s||_|p||_||_||_t||_t||_||_||_| |_ | |_ | |_ | |_ t jj|jddd}t jj||||_tt| |_|jj|dS(Nu/ii(tidtnametsubdirturltinttsizet unzipped_sizetchecksumt svn_revisiont copyrighttcontacttlicensetauthortostpathtsplitexttsplittjointfilenametbooltunzipt__dict__tupdate(tselfRRRRRRR R!R"R#R$R%R-tkwtext((s.lib/python2.7/site-packages/nltk/downloader.pyt__init__s          #cC s\t|tr!tj|}nx+|jD] }t|j||j|(R(R0((s.lib/python2.7/site-packages/nltk/downloader.pyt__repr__%sN( t__name__t __module__t__doc__tNonetTrueR3t staticmethodR9R;R<(((s.lib/python2.7/site-packages/nltk/downloader.pyRs 7 t CollectioncB s;eZdZddZedZdZdZRS(u A directory entry for a collection of downloadable packages. These entries are extracted from the XML index file that is downloaded by ``Downloader``. cK s>||_|p||_||_d|_|jj|dS(N(RRtchildrenR@tpackagesR.R/(R0RRDRR1((s.lib/python2.7/site-packages/nltk/downloader.pyR31s    cC st|tr!tj|}nx+|jD] }t|j||j|(R(R0((s.lib/python2.7/site-packages/nltk/downloader.pyR<OsN( R=R>R?R@R3RBR9R;R<(((s.lib/python2.7/site-packages/nltk/downloader.pyRC)s   tDownloaderMessagecB seZdZRS(uVA status message object, used by ``incr_download`` to communicate its progress.(R=R>R?(((s.lib/python2.7/site-packages/nltk/downloader.pyRIXstStartCollectionMessagecB seZdZdZRS(u<Data server has started working on a collection of packages.cC s ||_dS(N(t collection(R0RK((s.lib/python2.7/site-packages/nltk/downloader.pyR3`s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRJ]stFinishCollectionMessagecB seZdZdZRS(u=Data server has finished working on a collection of packages.cC s ||_dS(N(RK(R0RK((s.lib/python2.7/site-packages/nltk/downloader.pyR3gs(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRLdstStartPackageMessagecB seZdZdZRS(u-Data server has started working on a package.cC s ||_dS(N(tpackage(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3ns(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRMkstFinishPackageMessagecB seZdZdZRS(u.Data server has finished working on a package.cC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3us(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyROrstStartDownloadMessagecB seZdZdZRS(u.Data server has started downloading a package.cC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3|s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRPystFinishDownloadMessagecB seZdZdZRS(u/Data server has finished downloading a package.cC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRQstStartUnzipMessagecB seZdZdZRS(u,Data server has started unzipping a package.cC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRRstFinishUnzipMessagecB seZdZdZRS(u-Data server has finished unzipping a package.cC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRSstUpToDateMessagecB seZdZdZRS(u/The package download file is already up-to-datecC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRTst StaleMessagecB seZdZdZRS(u3The package download file is out-of-date or corruptcC s ||_dS(N(RN(R0RN((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRUst ErrorMessagecB seZdZdZRS(u Data server encountered an errorcC s7||_t|tr*t||_n ||_dS(N(RNR4t Exceptiontstrtmessage(R0RNRY((s.lib/python2.7/site-packages/nltk/downloader.pyR3s (R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRVstProgressMessagecB seZdZdZRS(u4Indicates how much progress the data server has madecC s ||_dS(N(tprogress(R0R[((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyRZstSelectDownloadDirMessagecB seZdZdZRS(u:Indicates what download directory the data server is usingcC s ||_dS(N(t download_dir(R0R]((s.lib/python2.7/site-packages/nltk/downloader.pyR3s(R=R>R?R3(((s.lib/python2.7/site-packages/nltk/downloader.pyR\st Downloaderc B seZdZd$ZdZdZdZdZdZd%d%dZ d%e e e e e dZ d Zd Zd Zd Zd Zd%e dZdZdZdZd%d%e e de e ejdZd%dZd%dZd%dZd%dZdZe ddZd%dZ dZ!dZ"dZ#dZ$dZ%e&e$e%Z'd Z(d!Z)d"Z*e&e)e*Z+d#Z,RS(&uy A class used to access the NLTK data server, which can be used to download corpora and other data packages. i<uChttps://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xmlu installedu not installedu out of dateupartialcC sv|p |j|_i|_i|_||_d|_d|_i|_d|_ |jdkrr|j |_ndS(N( t DEFAULT_URLt_urlt _collectionst _packagest _download_dirR@t_indext_index_timestampt _status_cachet_errorstdefault_download_dir(R0tserver_index_urlR]((s.lib/python2.7/site-packages/nltk/downloader.pyR3s       cC sd}|dkr,|j}td|n|rtddt|jtd|jtddt|j|d7}nt}} g} |r| jdn|r| jdnx| D]} td | j|d 7}xett || d t D]E} |j | |} | |j kr?|r?q n| |j krWt}n| |jkrot} nid |j 6d |j 6d|j6d|j6| }tjd d| jp| jddddd}td|| jjdd|f|t|jd7}|r |dkr td}|jd krHdSd}q q WtqWd}|ry|d7}n| r|d7}nttj|ddddddS(!Niu!Using default data directory (%s)u=iu Data server index for <%s>iupackagesu collectionsu%s:iR8u*u-uPu iiKtsubsequent_indentu [%s] %s %siu.u uHit Enter to continue: uxuqu([*] marks installed packagesu+; [-] marks out-of-date or corrupt packagesu+; [P] marks partially installed collectionsu)twidthiL(uxuq(R@RctprinttlenR`tFalsetappendt capitalizetsortedtgetattrRXtstatust INSTALLEDtSTALERAtPARTIALt NOT_INSTALLEDttextwraptfillRRtljustR)Rtlower(R0R]t show_packagestshow_collectionstheadert more_prompttskip_installedtlineststaletpartialt categoriestcategorytinfoRstprefixRt user_inputtmsg((s.lib/python2.7/site-packages/nltk/downloader.pytlistsb       %      '&     cC s|j|jjS(N(t _update_indexRbtvalues(R0((s.lib/python2.7/site-packages/nltk/downloader.pyREKs cC s?|jg|jjD]!\}}|jdkr|^qS(Nucorpora(RRbtitemsR(R0Rtpkg((s.lib/python2.7/site-packages/nltk/downloader.pytcorporaOs cC s?|jg|jjD]!\}}|jdkr|^qS(Nucorpora(RRbRR(R0RR((s.lib/python2.7/site-packages/nltk/downloader.pytmodelsSs cC s|j|jjS(N(RRaR(R0((s.lib/python2.7/site-packages/nltk/downloader.pyt collectionsWs cC s$t|tr|j|S|SdS(N(R4RR(R0t info_or_id((s.lib/python2.7/site-packages/nltk/downloader.pyt _info_or_id_s cc s"|dkr#|j}t|Vnt|ttfrax"|j|||D] }|VqNWdSy|j|}Wn2tt fk r}t dd||fVdSXt|t rt |Vx%|j |j||D] }|VqWt|Vn%x"|j|||D] }|VqWdS(NuError loading %s: %s(R@RcR\R4Rttuplet_download_listRtIOErrort ValueErrorRVRCRJt incr_downloadRDRLt_download_package(R0RR]tforceRRte((s.lib/python2.7/site-packages/nltk/downloader.pyRns&     cC s$t|trdSt|jSdS(Ni(R4RRmRE(R0titem((s.lib/python2.7/site-packages/nltk/downloader.pyt _num_packagessc # s4xdtt|D]P}yj||||sig?id(trangeRmRRRRVtsumt enumerateR4RRERRZR[( R0RR]RtiRt num_packagesR[RtdeltaR((R0s.lib/python2.7/site-packages/nltk/downloader.pyRs"   c c st|VtdV|j||}| rc||jkrct|VtdVt|VdS|jj|jdt j j ||j }t j j|r||jkrt|Vnt j|nt j j|st j|nt j jt j j ||js8t jt j j ||jnt|VtdVyt|j}t|d}td|jd}xmtjD]_} |jd} |j| | sPn| ddkrttd dd | |VqqWWdQX|jWn5t k r@} t!|d |j|j| fVdSXt"|Vtd V|j j#d rt j j ||j} |j$st j jt j j | |jrt%|Vx+t&|| d t'D]} || _(| VqWt)|Vqnt|VdS(NiidiuwbiiiiiPiKu$Error downloading %r from <%s>: %su.ziptverbosei@i@(*RMRZRsRtRTRORftpopRR@R&R'R*R+texistsRuRUtremovetmkdirRRPRRtopentmaxRt itertoolstcounttreadtwritetmintcloseRRVRQtendswithR-RRt _unzip_iterRnRNRS(R0RR]RRstfilepathtinfiletoutfilet num_blockstblocktsRtzipdirR((s.lib/python2.7/site-packages/nltk/downloader.pyRs`     $"   -  -   u [nltk_data] c  sltjtd||dkrG|dk r9||_n|jtSdfd} x |j|||D]} t| t r.| | j |rt | j n|rt St|_ |s.dtj} | dkr|j| jj|||||s(t Sq+| dkr+t Sq.n|srt| trn| d | jjd 7qdt| trd |j r| d | jjqa| d | jjqdt| tr| d| jj|fqdt| tr| d| jjdqdt| trF| d| jjdqdt| trd| j}qdqrqrWtS(Ntfileuc  s/tj|d|d|ddS(Ntinitial_indentRju iu (RxRy(Rtprefix2(Rtprint_to(s.lib/python2.7/site-packages/nltk/downloader.pytshow s  u(Error installing package. Retry? [n/y/e]uyuYueuEuDownloading collection %ru | iu$Downloaded collection %r with errorsuDone downloading collection %suDownloading package %s to %s...u!Package %s is already up-to-date!u u Unzipping %s.(uyuY(ueuE(t functoolsRRlR@Rct_interactive_downloadRARR4RVRYRRnRgRtstriptdownloadRNRRJRKRLRMRTRRR+R\R]( R0RR]tquietRRt halt_on_errortraise_on_errortprint_error_toRRtchoice((RRs.lib/python2.7/site-packages/nltk/downloader.pyRsl                   cC s|j|||jkS(N(RsRu(R0RR]((s.lib/python2.7/site-packages/nltk/downloader.pytis_staleRscC s|j|||jkS(N(RsRt(R0RR]((s.lib/python2.7/site-packages/nltk/downloader.pyt is_installedUscC s3|dkr|jjn|jj|ddS(N(R@RftclearR(R0R((s.lib/python2.7/site-packages/nltk/downloader.pytclear_status_cacheXs cC sI|dkr|j}n|j|}t|trg|jD]}|j|j^q@}|j|krt|jS|j |kr|j S|j |kr|j |kr|j S|j |kr|j S|j Snvt j j||j}||jkr|j||S|j|jkr7|j|||j|js (R&R'RRwRtOSErrorRRRRut md5_hexdigestR RRttisdirRtwalkR(R0RRtfilestattunzipdirR((s.lib/python2.7/site-packages/nltk/downloader.pyRs,   cC sY|jxH|jD]:}|j||jkr|j|d|d|qqWdS(uA Re-download any packages whose status is STALE. RRN(RRERsRuR(R0RRR((s.lib/python2.7/site-packages/nltk/downloader.pyR/s c C sU|jdkp4|dk p4tj|j|jks;dS|pG|j|_tjjt j t |jj |_tj|_g|jj dD]}tj|^q}td|D|_g|jj dD]}tj|^q}td|D|_x|jjD]}xt|jD]u\}}||jkro|j||j|sucollections/collectioncs s|]}|j|fVqdS(N(R(Rtc((s.lib/python2.7/site-packages/nltk/downloader.pys su.removing collection member with no package: {}(RdR@ttimeRet INDEX_TIMEOUTR`tnltkt internalstElementWrapperRR5RtgetrootRFRR9tdictRbRCRaRRRDRltformatR4textendRRERfR( R0RRRERRRKRtchild_idtqueueRH((s.lib/python2.7/site-packages/nltk/downloader.pyRsB  $..   cC s|j|jS(u Return the XML index describing the packages available from the data server. If necessary, this index will be downloaded from the data server. (RRd(R0((s.lib/python2.7/site-packages/nltk/downloader.pytindexs cC sR|j||jkr$|j|S||jkr>|j|Std|dS(uNReturn the ``Package`` or ``Collection`` record for the given item.uPackage %r not found in indexN(RRbRaR(R0R((s.lib/python2.7/site-packages/nltk/downloader.pyRs    cC s|jx3|jjdD]}|jd|kr|SqWx3|jjdD]}|jd|krS|SqSWtd|dS(u-Return the XML info record for the given itemupackages/packageuiducollections/collectionuPackage %r not found in indexN(RRdRFRGR(R0RRNRK((s.lib/python2.7/site-packages/nltk/downloader.pytxmlinfos cC s|jS(u)The URL for the data server's index file.(R`(R0((s.lib/python2.7/site-packages/nltk/downloader.pyt_get_urlscC s4|j}y|j|Wn||_nXdS(u Set a new URL for the data server. If we're unable to contact the given url, then the original url is kept. N(R`R(R0Rt original_url((s.lib/python2.7/site-packages/nltk/downloader.pyt_set_urls   cC sdtjkrdSx<tjjD].}tjj|r tjj|r |Sq Wtj dkrdtjkrtjd}n-tjj d}|dkrt dntjj |dS(u Return the directory to which packages will be downloaded by default. This value can be overridden using the constructor, or on a case-by-case basis using the ``download_dir`` argument when calling ``download()``. On Windows, the default download directory is ``PYTHONHOME/lib/nltk``, where *PYTHONHOME* is the directory containing Python, e.g. ``C:\Python25``. On all other platforms, the default directory is the first of the following which exists or which can be created with write permission: ``/usr/share/nltk_data``, ``/usr/local/share/nltk_data``, ``/usr/lib/nltk_data``, ``/usr/local/lib/nltk_data``, ``~/nltk_data``. uAPPENGINE_RUNTIMENuwin32uAPPDATAu~/u+Could not find a default download directoryu nltk_data( R&tenvironRtdataR'RRt is_writabletsystplatformt expanduserRR*(R0tnltkdirthomedir((s.lib/python2.7/site-packages/nltk/downloader.pyRh#s$ cC s|jS(u The default directory to which packages will be downloaded. This defaults to the value returned by ``default_download_dir()``. To override this default on a case-by-case basis, use the ``download_dir`` argument when calling ``download()``. (Rc(R0((s.lib/python2.7/site-packages/nltk/downloader.pyt_get_download_dirJscC s||_|jjdS(N(RcRfR(R0R]((s.lib/python2.7/site-packages/nltk/downloader.pyt_set_download_dirSs cC sUtrAyt|jWqQtk r=t|jqQXnt|jdS(N(tTKINTERt DownloaderGUItmainloopR tDownloaderShelltrun(R0((s.lib/python2.7/site-packages/nltk/downloader.pyR^s  iN(-R=R>R?RR_RtRwRuRvR@R3RARnRRERRRRRRRRRtstderrRRRRRsRR/RRRRRRtpropertyRRhRRR]R(((s.lib/python2.7/site-packages/nltk/downloader.pyR^sb ' 5        GX    " $ >    ' RcB sPeZdZdZdZdZdZdZdZdZ RS(cC s ||_dS(N(t_ds(R0t dataserver((s.lib/python2.7/site-packages/nltk/downloader.pyR3kscG sctdddtd|Dt|dd}td|j|tdddS(Nu-iKiDcs s|]}t|VqdS(N(Rm(Rto((s.lib/python2.7/site-packages/nltk/downloader.pys psiu u (RlRRmR*(R0toptionstspc((s.lib/python2.7/site-packages/nltk/downloader.pyt_simple_interactive_menuns,cC stdxtr|jddddddtdj}|sTtq n|jjd }|jd }y|d krt|jj|jj d t d tn|dkr|j nl|dkr|j nS|dkrdS|dkr|j |n'|dkr,|jntd|WnDtk r^}td|n$tk r}td|jnXtq WdS(NuNLTK Downloaderu d) Downloadul) Listu u) Updateu c) Configuh) Helpuq) Quitu Downloader> iiulR~RuhucuquxuduuuCommand %r unrecognizeduError reading from server: %suError connecting to server: %s(uqux(RlRARRRR{R)RRR]Rnt_simple_interactive_helpt_simple_interactive_configt_simple_interactive_downloadt_simple_interactive_updateRRtreason(R0RtcommandtargsR((s.lib/python2.7/site-packages/nltk/downloader.pyRwsF   %        cC s?|rZx2|D]F}y|jj|ddWq ttfk rR}t|q Xq Wnxtr:ttdtd}|jdkr|jj|jj dt dtdtq]q]|jd krdS|r]xT|j D]F}y|jj|ddWqttfk r.}t|qXqWPq]q]WdS( NRu u*Download which package (l=list; x=cancel)?u Identifier> ulR~RRuxuqu(uxuqu( RRRRRlRARR{RR]RnR)(R0R targRRR((s.lib/python2.7/site-packages/nltk/downloader.pyRs4      c C sxtrg}t}}xctt|jddtD]@}|jj||jjkr;|j|j |j fq;q;Wt |rt dx\|D]T\}}t j dd|ddddd}t d |jd d |fqWt td }|jd kruxT|D]L\}}y|jj|ddWq!ttfk rl} t | q!Xq!WPq|jdkrdSqt ddSqWdS(NupackagesR8u/Will update following packages (o=ok; x=cancel)u-iiKRju u [ ] %s %siu.u Identifier> uoRu uxuquuNothing to update.(uxuqu(RARnRqRrRRXRsRuRoRRRlRxRyRzRR{RRR( R0tstale_packagesRRRtpidtpnameRRR((s.lib/python2.7/site-packages/nltk/downloader.pyRs6  (  $  cC s3ttdtdtdtddS(Nu Commands:uH d) Download a package or collection u) Update out of date packagesu1 l) List packages & collections h) Helpu1 c) View & Modify Configuration q) Quit(Rl(R0((s.lib/python2.7/site-packages/nltk/downloader.pyRs   cC sttdtd|jjtdt|jjtdt|jjttdtd|jjdS(Nu Data Server:u - URL: <%s>u$ - %d Package Collections Availableu$ - %d Individual Packages AvailableuLocal Machine:u - Data directory: %s(RlRRRmRRER](R0((s.lib/python2.7/site-packages/nltk/downloader.pyt _show_configs  cC sr|jxatrmt|jddddtdjj}|dkra|jq |dkrtdj}|dkrtdqjtjj |r||j _ qjtd|q |dkrZtdj}|dkrtdqj|j dsd|}ny||j _ Wqjtk rV}td||fqjXq |dkr Pq q WdS(Nus) Show Configuu) Set Server URLud) Set Data Diru m) Main MenuuConfig> usudu New Directory> uuxuquXuQu Cancelled!u)Directory %r not found! Create it first.uuu New URL> uhttp://uhttps://uError reading <%r>: %sum(uuxuquXuQ(uuxuquXuQ(uhttp://uhttps://(RRARlRRRR{R&R'RRR]t startswithRRW(R0Rt new_dl_dirtnew_urlR((s.lib/python2.7/site-packages/nltk/downloader.pyRs6            ( R=R>R3RRRRRRR(((s.lib/python2.7/site-packages/nltk/downloader.pyRjs  )  Rc B seZdZddddddddd d d d g Zid d6dd6d d6d d6Zidd6dd6dd6dd6dd6dd6ZdZdddddgZx eD]Zeekst qWx eD]Zeekst qWx eD]Zeekst qWdNZ idOe j 6dPe j 6dQe j6dRe j6ZdSZdTZdUZdVZd#Zed$Zd%Zd&Zd'Zd(Zd)Zd*ZdWd+Zd,Z d-Z!d.Z"d/Z#d0Z$d1Z%d2Z&d3Z'dWZ(d4Z)d5Z*d6Z+dZ,d7Z-d8Z.d9Z/d:Z0d;Z1d<Z2d=Z3d>Z4d?Z5d@Z6dAZ7e8j9dBZ:dCZ;dDZ<dZ=dEZ>dFZ?dGZ@dHZAdIZBdJeCjDfdKYZEdLZFdMZGRS(XuU Graphical interface for downloading packages from the NLTK data server. uu IdentifieruNameuSizeuStatusu Unzipped Sizeu CopyrightuContactuLicenseuAuthoruSubdiruChecksumiiiii-i i iu#000u#cccu#afau#080u#ffau#880u#faau#800u#fffu#888u#45cu#aaau#67au#f00uhelvetica -16 boldcC s||_||_tj|_g|_g|_t|_i|_ g|_ d|_ |j dt }|_|jd|jd|jd|jd|jd|j|jd|jt|_i|_|j|jy|jWnBtk r!}td |n#tk rC}td |jnX|j|j|j j!d|j jd |j"dS( NiuNLTK Downloader Started!u+50+50uNLTK Downloadert backgroundiu u uError reading from serveruError connecting to serveru (#Rt _use_threadst threadingtLockt_download_lockt_download_msg_queuet_download_abort_queueRnt _downloadingt_afteridt _log_messagest _log_indentt_logRttoptgeometryttitlet configuret_BACKDROP_COLORtbindtdestroyt _destroyedt _column_varst _init_widgetst _init_menut _fill_tableRRRRt _show_infot_select_columnst_tabletselectt_destroy(R0Rt use_threadsRR((s.lib/python2.7/site-packages/nltk/downloader.pyR3ls<                 cC s.|jjdtjd|j|fdS(Nu%s %s%su | (RRoRtctimeR(R0R((s.lib/python2.7/site-packages/nltk/downloader.pyRs c  stjdddddddd}|jd d d td d |jddd|jdddt|ddjddddt|}|jddddddt|}|jddddddt|}|jddddddt|ddjddddt|}|jddddddt|ddjddddtjdddddjd}|jddd ddjd|jddd|dd?q%Wjj d'j%t&|d#d@dAjd.d_'j'jdd%t&|d#dBdAj(d.d_)j)jddCt |d#dDd-jddjd_*t+|d.dEdd+dj,dddFdd_-j.j-jddCj*jdd%dS(GNtreliefuraisedtborderitpadxitpadyitsidutoptexpandRyubothtweightitheighttcolumntrowtstickyunewsiiiiRtsideubottomuxuborderuhighlightthicknessu CollectionsuCorporauModelsu All Packagesttexttfontulefti u tcolumn_weightsthighlightthicknesstlistbox_heightitreprfunct foregroundRkuuuuuu uurlu Server Index:u download_diruDownload Directory:ueucourierugroovetdisabledforegroundublackc s j|S(N(t _info_edit(RR8(R0(s.lib/python2.7/site-packages/nltk/downloader.pyttuewuDownloadR uRefreshurightuiusunken(/RRtpackRAtgrid_rowconfiguretgrid_columnconfiguretgridR#t _tab_namest_tabsRRt _TAB_FONTR$t _select_tabR{tCOLUMNStCOLUMN_WEIGHTSRGRt_table_reprfuncR-t columnconfigt _MARK_COLORt COLUMN_WIDTHStDEFAULT_COLUMN_WIDTHtfocustbind_to_listboxest _downloadt _table_markt _prev_tabt _next_tabt _mark_allRRt_infoRt _info_saveR t_download_buttont_refresht_refresh_buttont_progresslabelR t_PROGRESS_COLORt _progressbart_init_progressbar(R0tf1ttabframet tableframet buttonframet infoframet progressframeRttabtlabelR:R@RkRR8tcallbacktentry((R0s.lib/python2.7/site-packages/nltk/downloader.pyR(s'"   " "%   "+   "( !!    c  s%tj}t|dd}|jdddddjdd|j|jdd dd dfd |jdd dddfd |j|jdddddj|j|jdddddjdd|jddddd|t|dd}xjj dD]{}t j}|j ks[t |j |<|j kr|jdn|jd|ddd|djq1W|jddddd|t|dd}x>jj dD],}|jdd|d|fdqW|jx>jj dD],}|jdd|d|fdq:W|jddddd|t|dd}|jdddddj|jdddddjdd|jdd ddd|jjd!jjjd|dS("NttearoffiRouDownloadt underlineR t acceleratoruReturnuChange Server Indexic s jdS(Nuurl(RF((R0(s.lib/python2.7/site-packages/nltk/downloader.pyRGRHuChange Download Directoryc s jdS(Nu download_dir(RF((R0(s.lib/python2.7/site-packages/nltk/downloader.pyRG!RHuShow LogiuExitiuCtrl-xuFiletmenuitvariableuViewu Sort by %sc sjj|dS(Nu ascending(R-tsort_by(R(R0(s.lib/python2.7/site-packages/nltk/downloader.pyRGARHuReverse sort by %sc sjj|dS(Nu descending(R-Rw(R(R0(s.lib/python2.7/site-packages/nltk/downloader.pyRGHRHuSortuAboutu InstructionsuF1uHelpu(R Rt add_commandRZt add_separatort _show_logR%t add_cascadeR-t column_namesR R'tAssertionErrortINITIAL_COLUMNStsettadd_checkbuttonR,taboutthelpR$tconfig(R0tmenubartfilemenutviewmenuR:tvartsortmenuthelpmenu((R0s.lib/python2.7/site-packages/nltk/downloader.pyR)s^                cC sSxL|jjD];\}}|jr;|jj|q|jj|qWdS(N(R'RRGR-t show_columnt hide_column(R0R:R((s.lib/python2.7/site-packages/nltk/downloader.pyR,Vs cC st|jjy|jWnBtk r=}td|n#tk r_}td|jnX|jjddS(NuError reading from serveruError connecting to serveri( RRR*RRRRR-R.(R0R((s.lib/python2.7/site-packages/nltk/downloader.pyRb]s cC s?|j|j|\}}d|d((s.lib/python2.7/site-packages/nltk/downloader.pyRzjscC sg}xt|jD]\}}|dkr>|jdq|dkr]|j|jq|dkr|j|jj|q|jjdd}|jt||dqW|S(u Given a package, return a list of values describing that package, one for each column in ``self.COLUMNS``. iuu IdentifieruStatusu u_un/a( RRQRoRRRsR{treplaceRr(R0RR;t column_indext column_nametattr((s.lib/python2.7/site-packages/nltk/downloader.pyRns   cG s'|jr dS|jjt|_dS(N(R&RR%RA(R0R((s.lib/python2.7/site-packages/nltk/downloader.pyR%s  cG sl|jdk r<x*|jjD]}|jj|qWn|jr[|jr[|jn|jj dS(N( RR@RRt after_cancelRRt_abort_downloadR'R(R0RR((s.lib/python2.7/site-packages/nltk/downloader.pyR/s  cO s|jj||dS(N(RR(R0R tkwargs((s.lib/python2.7/site-packages/nltk/downloader.pyRsun This tool can be used to download a variety of corpora and models that can be used with NLTK. Each corpus or model is distributed in a single zip file, known as a "package file." You can download packages individually, or you can download pre-defined collections of packages. When you download a package, it will be saved to the "download directory." A default download directory is chosen when you run the downloader; but you may also select a different download directory. On Windows, the default download directory is "package." The NLTK downloader can be used to download a variety of corpora, models, and other data packages. Keyboard shortcuts:: [return] Download [up] Select previous package [down] Select next package [left] Select previous tab [right] Select next tab cG s\y,t|jd|jjddddWn)t|jd|jjddnXdS(NuHelp: NLTK DowloaderRkiKR?ufixeduHelp: NLTK Downloader(RRtHELPR(R0R((s.lib/python2.7/site-packages/nltk/downloader.pyRs cG shdd}d}y-ddlm}|d|d|jWn$tk rct|j||nXdS(NuNLTK Downloader uWritten by Edward LoperuAbout: NLTK Downloaderi(tMessageRYR!(tsix.moves.tkinter_messageboxRRt ImportErrorRR(R0RtABOUTtTITLER((s.lib/python2.7/site-packages/nltk/downloader.pyRs  cC s |j}t|dt|d}}xtdt|dd|jD]c}|j||jdd||j|d|dd|jdd d t|d d d qOW|jd|jddd|jd|j ddddd|j ddS(NuwidthuheightiiiiRkRyu #%02x0000iPiii ugradienttstateuhiddenuredbox( RfRRt_gradient_widtht create_linetabst addtag_allt itemconfigtaddtag_withtagtcreate_rectangleRe(R0RRkR9R((s.lib/python2.7/site-packages/nltk/downloader.pyRgs !+  % cC s|j}|dkrD|jddddd|jdddnVt|dt|d}}|t|dd }|jddd||d dS( NuredboxiugradientRuhiddenuwidthuheightidi(RfR@tcoordsRR(R0tpercentRRkR9tx((s.lib/python2.7/site-packages/nltk/downloader.pyRs  !cC s|j}|js(|jdddn|jddd|jd\}}}}|dkr|jd|jdddn|jdd d|jjd |j}||j d R3R(((s.lib/python2.7/site-packages/nltk/downloader.pyR;s idc s1fd}jjs"dSxjD]}|dksJ|dkrjt_djd<j2j2jj|dkr|dj dn(j j dj d}|j dR?RQRRRVRWR~RR}R#R^RtRvRuRwRRURRReRORAR3RR(R)R,RbRFR@R`RSRRR+R\R]RPRt_rowsR*RRZRRRRRR^R[RzRR%R/RRxtdedentRRRRRgRRRRRRRRR(((s.lib/python2.7/site-packages/nltk/downloader.pyRs "         1 k E      )   ,              , cC s>t|tr4t|d}t|SWdQXnt|S(uz Calculate and return the MD5 checksum for a given file. ``file`` may either be a filename or an open stream. urbN(R4RRt_md5_hexdigest(RR((s.lib/python2.7/site-packages/nltk/downloader.pyRscC sFt}x0tr;|jd}|s+Pn|j|q W|jS(Niii@(RRARR/t hexdigest(tfpt md5_digestR((s.lib/python2.7/site-packages/nltk/downloader.pyRs  cC s?x8t|||D]$}t|trt|qqWdS(u\ Extract the contents of the zip file ``filename`` into the directory ``root``. N(RR4RVRW(R+trootRRY((s.lib/python2.7/site-packages/nltk/downloader.pyR-scc s|r:tjjdtjj|dtjjnytj|}WnFtj k rt}t |dVdSt k r}t ||VdSX|j ||rt ndS(Nu Unzipping %siuError with downloaded zip file(RtstdoutRR&R'R)tflushtzipfiletZipFileterrorRVRWt extractallRl(R+RRtzfR((s.lib/python2.7/site-packages/nltk/downloader.pyRs$ cC s#g}xttjj|dD]\}}}tj|j}d||tjj|jdf}td|jD}|j dd||j dd|j |j ddt |j|j d ||j d s|j d |n|j |q"Wtttjj|d } t } xY|| D]M} | j d | kr|td | j d n| j| j d qEWtjd} | j tjdx|D]} | dj | qW| j tjd x| D]}| dj |qWt| | S(u} Create a new data.xml index file, by combining the xml description files for various packages and collections. ``root`` should be the path to a directory containing the package xml and zip files; and the collection xml files. The ``root`` directory is expected to have the following subdirectories:: root/ packages/ .................. subdirectory for packages corpora/ ................. zip & xml files for corpora grammars/ ................ zip & xml files for grammars taggers/ ................. zip & xml files for taggers tokenizers/ .............. zip & xml files for tokenizers etc. collections/ ............... xml files for collections For each package, there should be two files: ``package.zip`` (where *package* is the package name) which contains the package itself as a compressed zip file; and ``package.xml``, which is an xml description of the package. The zipfile ``package.zip`` should expand to a single subdirectory named ``package/``. The base filename ``package`` must match the identifier given in the package's xml file. For each collection, there should be a single file ``collection.zip`` describing the collection, where *collection* is the name of the collection. All identifiers (for both packages and collections) must be unique. upackagesu%s/%s/%sics s|]}|jVqdS(N(t file_size(Rtzf_info((s.lib/python2.7/site-packages/nltk/downloader.pys  su unzipped_sizeu%susizeuchecksumusubdiruurlu collectionsuiduDuplicate UID: %su nltk_datai(t_find_packagesR&R'R*RR+R)RtinfolistRRRRGRoRt_find_collectionsRtaddRtElementt _indent_xml(Rtbase_urlREtpkg_xmlRRtzipstatRRRtuidsRttop_eltRNRK((s.lib/python2.7/site-packages/nltk/downloader.pyt build_indexs6+&!    ucC st|dkr|jpdjd|d|_x|D]}t||dq=Wx6|d D]*}|jpudjd|d|_qcW|djpdjd||d_ndS(u Helper for ``build_index()``: Given an XML ``ElementTree``, modify it (and its descendents) ``text`` and ``tail`` attributes to generate an indented tree, where each nested element is indented by 2 spaces with respect to its parent. iuu u iN(RmR>RRttail(R7RRH((s.lib/python2.7/site-packages/nltk/downloader.pyR+ s$ (c stjjtjj|dd|jdkr]td|jdfntfd|jDrtdfndS(ur Helper for ``build_index()``: Perform some checks to make sure that the given package is consistent. iiuidu&package identifier mismatch (%s vs %s)c3 s/|]%}|ko&|jd VqdS(u/N(R(RR(tuid(s.lib/python2.7/site-packages/nltk/downloader.pys H su;Zipfile %s.zip does not expand to a single subdirectory %s/N(R&R'R(R)RGRRtnamelist(Rt zipfilenameR((Rs.lib/python2.7/site-packages/nltk/downloader.pyt_check_package; s&"cC stjddd|gdtjdtj}|j\}}|jdks[|s[| rtdtjj|dt j |fn|jd S( u Helper for ``build_index()``: Calculate the subversion revision number for a given file (by using ``subprocess`` to run ``svn``). usvnustatusu-vRRiu)Error determining svn_revision for %s: %sii( t subprocesstPopentPIPEt communicatet returncodeRR&R'R)RxRy(R+RRR((s.lib/python2.7/site-packages/nltk/downloader.pyt _svn_revisionP s  ,cc syg}xltj|D][\}}}xI|D]A}|jdr,tjj||}tj|jVq,q,WqWdS(u Helper for ``build_index()``: Yield a list of ElementTree.Element objects, each holding the xml for a single package collection. u.xmlN(R&RRR'R*RR5R(RREtdirnametsubdirsRR+txmlfile((s.lib/python2.7/site-packages/nltk/downloader.pyRc s  c # sddlm}g}xtj|D]\}}}dj|||}xa|D]Y}|jdrTtjj||}|d d} ytj| } Wn)t k r} t d| | fnXyt j |j } Wn)t k r} t d|| fnXtjj|d d| jd krat d | jd fntfd | jDrt d fn| | |fVqTqTWy|jd Wq&t k rq&Xq&WdS(u Helper for ``build_index()``: Yield a list of tuples ``(pkg_xml, zf, subdir)``, where: - ``pkg_xml`` is an ``ElementTree.Element`` holding the xml for a package - ``zf`` is a ``zipfile.ZipFile`` for the package's contents. - ``subdir`` is the subdirectory (relative to ``root``) where the package was found (e.g. 'corpora' or 'grammars'). i(t _path_fromu/u.xmliu.zipuError reading file %r! %siuidu&package identifier mismatch (%s vs %s)c3 s/|]%}|ko&|jd VqdS(u/N(R(RR(R(s.lib/python2.7/site-packages/nltk/downloader.pys su;Zipfile %s.zip does not expand to a single subdirectory %s/u.svnN(tnltk.corpus.reader.utilRR&RR*RR'RRRWRRR5RR)RGRRR( RRRER R RtrelpathR+t xmlfilenameRRRR((Rs.lib/python2.7/site-packages/nltk/downloader.pyRp s@    cC sttjdS(N(Rt _downloaderR(((s.lib/python2.7/site-packages/nltk/downloader.pytdownload_shell scC sttjdS(N(RRR(((s.lib/python2.7/site-packages/nltk/downloader.pyt download_gui scC stjdS(N(RR/(((s.lib/python2.7/site-packages/nltk/downloader.pyR/ su__main__(t OptionParseru-du--dirtdestudirRu!download package to directory DIRtmetavaruDIRu-qu--quietuquiettactionu store_truetdefaultu work quietlyu-fu--forceuforceu"download even if already installedu-eu--exit-on-erroru halt_on_erroruexit if an error occursu-uu--urluserver_index_urluNLTK_DOWNLOAD_URLudownload server index urlRiRR]RRR(jR?t __future__RRRRR&RRRxRRtshutilRRthashlibRt xml.etreeRRARtsix.moves.tkinterRRRRR R R R R RRtnltk.draw.tableRtnltk.draw.utilRRRnRtsixRRt six.movesRtsix.moves.urllib.requestRtsix.moves.urllib.errorRRRt nltk.compatRtobjectRRCRIRJRLRMRORPRQRRRSRTRURVRZR\R^RRRRR-RRRRR RRRRRRR/R=toptparseRtparsert add_optionRRGt parse_argsRR Rit downloadertpkg_idtdirRRRtrv(((s.lib/python2.7/site-packages/nltk/downloader.pytDs\l @     ].    H    ?