[c@`s#dZddlmZmZmZmZddlmZddlm Z m Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlmZmZddlmZdd lmZdd lmZdd lm Z m!Z!yddl"Z"Wne#k rKe$Z%nXe&Z%d d ddddddddddddddddgZ'iZ(dej)fdYZ*e*Z+d efd!YZ,d"Z-d#Z.e j/de$e&dd$Z1d%Z2e j/dde&d&Z3de&dd'Z4dde&d(Z5dd)d*Z6dd)dd+Z7d,Z8dd-Z9d.Z:d/Z;d0Z<e$e&dd1Z=d2Z>d3Z?e$e&dd4Z@gaAe jBd5ZCdd6ZDd7ZEe$d8ZFd9ZGd:ZHd;ZIdS(<u[ This module contains helper functions for accessing, downloading, and caching data files. i(tabsolute_importtdivisiontprint_functiontunicode_literalsi(tsix(turllibtrangeN(tNamedTemporaryFilet gettempdir(twarn(tconfig(tAstropyWarning(tfind_current_modulet resolve_nameuConfuconfuget_readable_fileobjuget_file_contentsuget_pkg_data_fileobjuget_pkg_data_filenameuget_pkg_data_contentsuget_pkg_data_fileobjsuget_pkg_data_filenamesu compute_hashuclear_download_cacheuCacheMissingWarninguget_free_space_in_dirucheck_free_space_in_diru download_fileudownload_files_in_paralleluis_url_in_cacheuget_cached_urlstConfcB`seZdZejddZejddZejddddgZejdd Zejdd Z ejd ddZ eje dZ RS(u< Configuration parameters for `astropy.utils.data`. uhttp://data.astropy.org/u)Primary URL for astropy remote data site.u$http://www.astropy.org/astropy-data/u(Mirror URL for astropy remote data site.g$@u2Time to wait for remote data queries (in seconds).taliasesu5astropy.coordinates.name_resolve.name_resolve_timeoutiiu)Block size for computing MD5 file hashes.u4Number of bytes of remote data to download per step.iu'Number of times to try to get the lock u0while accessing the data cache before giving up.uzIf True, temporary download files created when the cache is inaccessible will be deleted at the end of the python session.ii( t__name__t __module__t__doc__t_configt ConfigItemtdataurltdataurl_mirrortremote_timeouttcompute_hash_block_sizetdownload_block_sizetdownload_cache_lock_attemptstTruet"delete_temporary_downloads_at_exit(((s1lib/python2.7/site-packages/astropy/utils/data.pyR2s0      tCacheMissingWarningcB`seZdZRS(u1 This warning indicates the standard cache directory is not accessible, with the first argument providing the warning message. If args[1] is present, it is a filename indicating the path to a temporary file that was created to store a remote data download in the absence of the cache. (RRR(((s1lib/python2.7/site-packages/astropy/utils/data.pyRTscC`s%tjj|}|jjdkS(uy Test whether a string is a valid URL Parameters ---------- string : str The string to test uhttpuhttpsuftpusftpusshufile(uhttpuhttpsuftpusftpusshufile(Rtparseturlparsetschemetlower(tstringturl((s1lib/python2.7/site-packages/astropy/utils/data.pyt_is_url]s cC`sLtjj|jtjj|pKtjj|jtjj|S(N(tostpathtabspatht startswithtrealpath(R&t parent_path((s1lib/python2.7/site-packages/astropy/utils/data.pyt _is_insidems'cc`stj}tr"|tjf7}ng}g}|d krFtj}nt||rtrjt |}nt |}|rt |d|d|d|}ntj rt |d} ntj|d} |r| r|j| n|j| n|} t| ds$tj| j} n| jd} | jd| d d krdd l} y5dd l} | jd | d d} | jdWnXttfk r| jd| jqM| jk r| jd| jqMX| jd| } nK| d dkr ydd l}Wn8tk r\x|D]}|jq9WtdnXyatddt <}|j!| j|j|j"|j#d d} Wd QX| jdWn(tk r| jd| jqMX| jd|j| | } nA| d dkrMyt$j%dkrRdd l&}|j'| d d} nmddl(m&}ddl)m'}tddt 9}|j!| j|j||j#d d} Wd QX| jdWngtk rx|D]}|jqWtdqMttfk r6}| jd| jqMX| jd| } ntj rq|dkok|d k }n |dk}|rydd l}Wntk rnXt| |j"r>tddt }| j}|j!||j|j|tj rt |j#d} ntj|j#d} |j| ntj rnt| t*rntj| j+} ntj,| } tj-| d|} | jdnz | VWd x|D]}|jqWx|D]}t.j/|j#qWXd S(u Given a filename, pathlib.Path object or a readable file-like object, return a context manager that yields a readable file-like object. This supports passing filenames, URLs, and readable file-like objects, any of which can be compressed in gzip, bzip2 or lzma (xz) if the appropriate compression libraries are provided by the Python installation. Notes ----- This function is a context manager, and should be used for example as:: with get_readable_fileobj('file.dat') as f: contents = f.read() Parameters ---------- name_or_obj : str or file-like object The filename of the file to access (if given as a string), or the file-like object to access. If a file-like object, it must be opened in binary mode. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that on Python 2.x returns `bytes` objects and on Python 3.x returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool, optional Whether to cache the contents of remote URLs. show_progress : bool, optional Whether to display a progress bar if the file is downloaded from a remote server. Default is `True`. remote_timeout : float Timeout for remote requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`, which is 3s by default) Returns ------- file : readable file-like object tcachet show_progressttimeouturburuseekiiisNtfileobjtmodeitBZhu`.bz2 format files are not supported since the Python interpreter does not include the bz2 moduleuwbtdeletes7z(tlzma(tLZMAFileu.xz format files are not supported since the Python interpreter does not include the lzma module. On Python versions < 3.3 consider installing backports.lzmaubinarytencoding(iii(0Rt string_typest HAS_PATHLIBtpathlibtPathtNonetconfRt isinstancetstrR$t download_filetPY2topentiotFileIOtappendthasattrtBytesIOtreadtseektstructtgziptGzipFiletIOErrortEOFErrortcloseterrortbz2t ImportErrort ValueErrorRtFalsetwritetBZ2Filetnametsyst version_infoR3R4t backportstbackports.lzmatfiletfilenotBufferedReadert TextIOWrapperR%tremove(t name_or_objR5R,R-Rt PATH_TYPESt close_fdst delete_fdstis_urlR/t signatureRHRIt fileobj_newROtfdttmpR3R4tetneeds_textio_wrappertdata((s1lib/python2.7/site-packages/astropy/utils/data.pytget_readable_fileobjvsA                                            cO`s&t||}|jSWdQXdS(u Retrieves the contents of a filename or file-like object. See the `get_readable_fileobj` docstring for details on parameters. Returns ------- content The content of the file (as requested by ``encoding``). N(RkRF(targstkwargstf((s1lib/python2.7/site-packages/astropy/utils/data.pytget_file_contentsos c c`s)t|d|}tjj|r3tdntjj|rht|d| }|VWdQXntjtj f}x|D]i}yIt||d|d|'}|j d|j d|VPWdQXWqt j jk rqXqWdjd |D}t j jd j||dS( uJ Retrieves a data file from the standard locations for the package and provides the file as a file-like object that reads bytes. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that on Python 2.x returns `bytes` objects and on Python 3.x returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool If True, the file will be downloaded and saved locally or the already-cached local copy will be accessed. If False, the file-like object will directly access the resource (e.g. if a remote URL is accessed, an object like that from `urllib2.urlopen` on Python 2 or `urllib.request.urlopen` on Python 3 is returned). Returns ------- fileobj : file-like An object with the contents of the data file available via ``read`` function. Can be used as part of a ``with`` statement, automatically closing itself after the ``with`` block. Raises ------ urllib2.URLError, urllib.error.URLError If a remote file cannot be found. IOError If problems occur writing or reading a local file. Examples -------- This will retrieve a data file and its contents for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_fileobj >>> with get_pkg_data_fileobj('data/3d_cd.hdr', ... package='astropy.wcs.tests') as fobj: ... fcontents = fobj.read() ... This next example would download a data file from the astropy data server because the ``allsky/allsky_rosat.fits`` file is not present in the source distribution. It will also save the file locally so the next time it is accessed it won't need to be downloaded.:: >>> from astropy.utils.data import get_pkg_data_fileobj >>> with get_pkg_data_fileobj('allsky/allsky_rosat.fits', ... encoding='binary') as fobj: # doctest: +REMOTE_DATA ... fcontents = fobj.read() ... Downloading http://data.astropy.org/allsky/allsky_rosat.fits [Done] This does the same thing but does *not* cache it locally:: >>> with get_pkg_data_fileobj('allsky/allsky_rosat.fits', ... encoding='binary', cache=False) as fobj: # doctest: +REMOTE_DATA ... fcontents = fobj.read() ... Downloading http://data.astropy.org/allsky/allsky_rosat.fits [Done] See Also -------- get_pkg_data_contents : returns the contents of a file or url as a bytes object get_pkg_data_filename : returns a local name for a file containing the data tpackageuDTried to access a data file that's actually a package data directoryR5NR,iiu cs`s|]}dj|VqdS(u - {0}N(tformat(t.0R#((s1lib/python2.7/site-packages/astropy/utils/data.pys su<Failed to download {0} from the following repositories: {1}(t_find_pkg_data_pathR%R&tisdirRKtisfileRkR;RRRFRGRRNtURLErrortjoinRq( t data_nameRpR5R,tdatafnR/tall_urlsR#turls((s1lib/python2.7/site-packages/astropy/utils/data.pytget_pkg_data_fileobjs(i     c C`s|d krtj}n|jdrt|d}|d krtjtjf}xL|D]D}y$t||dtd|d|SWq\t j j k rq\Xq\Wdj d|D}t j j dj ||q|Sntjj|}t|d |} tjj| r*td ntjj| r@| Stjtjf}xL|D]D}y$t||dtd|d|SWqYt j j k rqYXqYWdj d |D}t j j d j ||d S(uA Retrieves a data file from the standard locations for the package and provides a local filename for the data. This function is similar to `get_pkg_data_fileobj` but returns the file *name* instead of a readable file-like object. This means that this function must always cache remote files locally, unlike `get_pkg_data_fileobj`. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. show_progress : bool, optional Whether to display a progress bar if the file is downloaded from a remote server. Default is `True`. remote_timeout : float Timeout for the requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`, which is 3s by default) Raises ------ urllib2.URLError, urllib.error.URLError If a remote file cannot be found. IOError If problems occur writing or reading a local file. Returns ------- filename : str A file path on the local file system corresponding to the data requested in ``data_name``. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filename >>> fn = get_pkg_data_filename('data/3d_cd.hdr', ... package='astropy.wcs.tests') >>> with open(fn) as f: ... fcontents = f.read() ... This retrieves a data file by hash either locally or from the astropy data server:: >>> from astropy.utils.data import get_pkg_data_filename >>> fn = get_pkg_data_filename('hash/34c33b3eb0d56eb9462003af249eff28') # doctest: +SKIP >>> with open(fn) as f: ... fcontents = f.read() ... See Also -------- get_pkg_data_contents : returns the contents of a file or url as a bytes object get_pkg_data_fileobj : returns a file-like object with the data uhash/iR,R-R.u cs`s|]}dj|VqdS(u - {0}N(Rq(RrR#((s1lib/python2.7/site-packages/astropy/utils/data.pys hsu>Failed to download {0} from the following repositories: {1} RpuDTried to access a data file that's actually a package data directorycs`s|]}dj|VqdS(u - {0}N(Rq(RrR#((s1lib/python2.7/site-packages/astropy/utils/data.pys su<Failed to download {0} from the following repositories: {1}N(R:R;RR(t _find_hash_fnRRR>RRRNRvRwRqR%R&tnormpathRsRtRKRu( RxRpR-RthashfnRzR#R{tfs_pathRy((s1lib/python2.7/site-packages/astropy/utils/data.pytget_pkg_data_filenamesDU       cC`s7t|d|d|d|}|j}WdQX|S(ub Retrieves a data file from the standard locations and returns its contents as a bytes object. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. * A URL to some other file. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that on Python 2.x returns `bytes` objects and on Python 3.x returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool If True, the file will be downloaded and saved locally or the already-cached local copy will be accessed. If False, the file-like object will directly access the resource (e.g. if a remote URL is accessed, an object like that from `urllib2.urlopen` on Python 2 or `urllib.request.urlopen` on Python 3 is returned). Returns ------- contents : bytes The complete contents of the file as a bytes object. Raises ------ urllib2.URLError, urllib.error.URLError If a remote file cannot be found. IOError If problems occur writing or reading a local file. See Also -------- get_pkg_data_fileobj : returns a file-like object with the data get_pkg_data_filename : returns a local name for a file containing the data RpR5R,N(R|RF(RxRpR5R,Rftcontents((s1lib/python2.7/site-packages/astropy/utils/data.pytget_pkg_data_contentssH u*cc`st|d|}tjj|r3tdndtjj|rxOtj|D]/}tj||rUtjj||VqUqUWn tddS(u Returns the path of all of the data files in a given directory that match a given glob pattern. Parameters ---------- datadir : str Name/location of the desired data files. One of the following: * The name of a directory included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data'`` to get the files in ``astropy/pkgname/data``. * Remote URLs are not currently supported. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. pattern : str, optional A UNIX-style filename glob pattern to match files. See the `glob` module in the standard library for more information. By default, matches all files. Returns ------- filenames : iterator of str Paths on the local filesystem in *datadir* matching *pattern*. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filenames >>> for fn in get_pkg_data_filenames('maps', 'astropy.wcs.tests', ... '*.hdr'): ... with open(fn) as f: ... fcontents = f.read() ... RpuDTried to access a data directory that's actually a package data fileuPath not foundN( RsR%R&RuRKRttlistdirtfnmatchRw(tdatadirRptpatternR&tfilename((s1lib/python2.7/site-packages/astropy/utils/data.pytget_pkg_data_filenamess, cc`sGx@t|d|d|D]&}t|d| }|VWdQXqWdS(u Returns readable file objects for all of the data files in a given directory that match a given glob pattern. Parameters ---------- datadir : str Name/location of the desired data files. One of the following: * The name of a directory included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data'`` to get the files in ``astropy/pkgname/data`` * Remote URLs are not currently supported package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. pattern : str, optional A UNIX-style filename glob pattern to match files. See the `glob` module in the standard library for more information. By default, matches all files. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that on Python 2.x returns `bytes` objects and on Python 3.x returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. Returns ------- fileobjs : iterator of file objects File objects for each of the files on the local filesystem in *datadir* matching *pattern*. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filenames >>> for fd in get_pkg_data_fileobjs('maps', 'astropy.wcs.tests', ... '*.hdr'): ... fcontents = fd.read() ... RpRR5N(RRk(RRpRR5tfnRf((s1lib/python2.7/site-packages/astropy/utils/data.pytget_pkg_data_fileobjs s; cC`slt|dQ}tj}|jtj}x)|r[|j||jtj}q3WWdQX|jS(ug Computes the MD5 hash for a file. The hash for a data file is used for looking up data files in a unique fashion. This is of particular use for tests; a test may require a particular version of a particular file, in which case it can be accessed via hash to get the appropriate version. Typically, if you wish to write a test that requires a particular data file, you will want to submit that file to the astropy data servers, and use e.g. ``get_pkg_data_filename('hash/34c33b3eb0d56eb9462003af249eff28')``, but with the hash for your file in place of the hash in the example. Parameters ---------- localfn : str The path to the file for which the hash should be generated. Returns ------- md5hash : str The hex digest of the MD5 hash for the contents of the ``localfn`` file. urbN(R@thashlibtmd5RFR;Rtupdatet hexdigest(tlocalfnRnthtblock((s1lib/python2.7/site-packages/astropy/utils/data.pyt compute_hashLs   cC`s#|d krtddddg}|d kr4|St|d sN|j rd|jkrv|jjdd}q|j}q|j}n t|}|jdd}t|}tj j |j }tj j ||}tj j |j }t ||stdj|n|S( uX Look for data in the source-included data directories and return the path. itfinddiffuastropy.utils.datau contextlibu __package__u.iu:attempted to get a local data file outside of the {} tree.N(R:R RDt __package__Rt rpartitionR t partitionR%R&tdirnamet__file__RwR+t RuntimeErrorRq(RxRptmodulet rootpkgnametrootpkgt module_pathR&troot_dir((s1lib/python2.7/site-packages/astropy/utils/data.pyRsqs&       cC`syt\}}Wn:ttfk rO}d}tt|t|dSXtjj ||}tjj |r{|SdSdS(us Looks for a local file by hash - returns file name if found and a valid file, otherwise returns None. u:Could not access cache directory to search for data file: N( t_get_download_cache_locsRKtOSErrorR RR=R:R%R&RwRu(thashtdldirturlmapfnRhtmsgR((s1lib/python2.7/site-packages/astropy/utils/data.pyR}scC`stjjdrddl}|jd}|jjj|j|dd|j |}|dkrt dj |n|j Stj|}|j|jSdS(u5 Given a path to a directory, returns the amount of free space (in bytes) on that filesystem. Parameters ---------- path : str The path to a directory Returns ------- bytes : int The amount of free space on the partition that the directory is on. uwiniNu0Checking free space on {!r} failed unexpectedly.(RVtplatformR(tctypest c_ulonglongtwindlltkernel32tGetDiskFreeSpaceExWt c_wchar_pR:tpointerRKRqtvalueR%tstatvfstf_bavailtf_frsize(R&Rt free_bytestretvaltstat((s1lib/python2.7/site-packages/astropy/utils/data.pytget_free_space_in_dirs  $  cC`sMddlm}t|}||krItdj|||ndS(ug Determines if a given directory has enough space to hold a file of a given size. Raises an IOError if the file would be too large. Parameters ---------- path : str The path to a directory size : int A proposed filesize (in bytes) Raises ------- IOError : There is not enough room on the filesystem i(thuman_file_sizeu5Not enough free space in '{0}' to download a {1} fileN(t utils.consoleRRRKRq(R&tsizeRtspace((s1lib/python2.7/site-packages/astropy/utils/data.pytcheck_free_space_in_dirs   cC`srddlm}|d kr(tj}nt}|ryt\}}Wqttfk r}d} t |j dkrdn dt |} t t | |jj| t}t}qXntjrt|tjr|jd} n|} | jtjratjtkratjtjjtjd|#} tj| jgttj|j8||| | s"  + %                      3! cC`syt\}}Wnittfk r~}d}t|jdkrLdn dt|}tt||jj |t SXt j rt |t jr|jd}nt|t}||krtSWdQXt S(u Check if a download from ``url_key`` is in the cache. Parameters ---------- url_key : string The URL retrieved Returns ------- in_cache : bool `True` if a download from ``url_key`` is in the cache u/Remote data cache could not be accessed due to iuu: uutf-8N(RRKRRRlR=R RRRRRRR?R<RRRR(RRRRhRRR((s1lib/python2.7/site-packages/astropy/utils/data.pytis_url_in_caches+  cC`s t|S(N(R>(Rl((s1lib/python2.7/site-packages/astropy/utils/data.pyt_do_download_files_in_parallelsc C`sddlm}|dkr(tj}n|r:tj}n tj}t t |}|j t g|D]}||t |f^qhd|dt}g} x(|D] } | j||j| qW| S(u Downloads multiple files in parallel from the given URLs. Blocks until all files have downloaded. The result is a list of local file paths corresponding to the given urls. Parameters ---------- urls : list of str The URLs to retrieve. cache : bool, optional Whether to use the cache show_progress : bool, optional Whether to display a progress bar during the download (default is `True`) timeout : float, optional Timeout for each individual requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`). Returns ------- paths : list of str The local file paths corresponding to the downloaded URLs. i(t ProgressBarRZt multiprocessN(tconsoleRR:R;RRVRRAREtlisttsettmapRRRRRCtindex( R{R,R-R.Rtprogresst combined_urlstxtcombined_pathstpathsR#((s1lib/python2.7/site-packages/astropy/utils/data.pytdownload_files_in_parallels     %  cC`sZtdk rVxGttdkrRtj}tjj|rtj|qqWndS(Ni(RR:RtpopR%R&RuR^(R((s1lib/python2.7/site-packages/astropy/utils/data.pyt _deltempss   c C`syt\}}Wnittfk r~}d}t|jdkrLdn dt|}tt||jj |dSXt zM|dkrt j j|rtj|qnt|t}t j j||}t||stdntjr,t|tjr,|jd}n|}t j j|rx3tj|D]"\} } | |krT|| =qTqTWt j|nB||kr||}||=t j j|rt j|qnWdQXWdt j jt j j|drtnXdS( uJ Clears the data file cache by deleting the local file(s). Parameters ---------- hashorurl : str or None If None, the whole cache is cleared. Otherwise, either specifies a hash for the cached file that is supposed to be deleted, or a URL that should be removed from the cache if present. u3Not clearing data cache - cache inacessable due to iuu: NuPattempted to use clear_download_cache on a path outside the data cache directoryuutf-8ulock(RRKRRRlR=R RRRRR:R%R&RRtrmtreeRRRwR+RRR?R<RRt iteritemstunlinkR( t hashorurlRRRhRRRtfilepaththash_keytktv((s1lib/python2.7/site-packages/astropy/utils/data.pytclear_download_caches> +    !cC`sddlm}dttjj}tjj|d|}tjj|d}tjj |sytj |Wqt k r}tjj |sqqXn0tjj |sd}t |j|ntjj |r d}t |j|n||fS(u Finds the path to the data cache directory and makes them if they don't exist. Returns ------- datadir : str The path to the data cache directory. shelveloc : str The path to the shelve object that stores the cache info. i(t get_cache_dirupyudownloaduurlmapu+Data cache directory {0} is not a directoryu4Data cache shelve object location {0} is a directory(t config.pathsRR=RVRWtmajorR%R&RwRtmakedirsRRtRKRq(Rt py_versionRt shelvelocRhR((s1lib/python2.7/site-packages/astropy/utils/data.pyR-s"  cC`s<ddl}|j|dd}|r4tj|S|SdS(u Opens a shelf file. If `withclosing` is True, it will be opened with closing, allowing use like: with _open_shelve('somefile',True) as s: ... iNtprotocoli(tshelveR@RR(tshelffnt withclosingRtshelf((s1lib/python2.7/site-packages/astropy/utils/data.pyRTs   cC`stjjtdd}xttjD]|}yQtj|ttjj|dd }|j t tj WdQXWnt k rt jdq,XdSq,Wd}t|j|dS(u Uses the lock directory method. This is good because `mkdir` is atomic at the system call level, so it's thread-safe. iulockupiduwNiuUnable to acquire lock for cache directory ({0} exists). You may need to delete the lock if the python interpreter wasn't shut down properly.(R%R&RwRRR;RtmkdirR@RSR=tgetpidRttimetsleepRRq(tlockdirtiRnR((s1lib/python2.7/site-packages/astropy/utils/data.pyRhs !# cC`stjjtdd}tjj|rutjj|d}tjj|retj|ntj|ndd}t|j |dS(Niulockupidu<Error releasing lock. "{0}" either does not exist or is not u a directory.( R%R&RwRRtRR^trmdirRRq(RtpidfnR((s1lib/python2.7/site-packages/astropy/utils/data.pyRscC`syt\}}Wnittfk r~}d}t|jdkrLdn dt|}tt||jj |t SXt |t }t |jSWdQXdS(u Get the list of URLs in the cache. Especially useful for looking up what files are stored in your cache when you don't have internet access. Returns ------- cached_urls : list List of cached URLs. u/Remote data cache could not be accessed due to iuu: N(RRKRRRlR=R RRRRRRRRtkeys(RRRhRRR((s1lib/python2.7/site-packages/astropy/utils/data.pytget_cached_urlss +(JRt __future__RRRRtexternRtextern.six.movesRRtatexitRRRR%RARRRVR ttempfileRRtwarningsR tR Rtutils.exceptionsR tutils.introspectionR R R8RPRRR7Rt__all__RtConfigNamespaceRR;RR$R+tcontextmanagerR:RkRoR|RRRRRRsR}RRR>RRRRtregisterRRRRRRR(((s1lib/python2.7/site-packages/astropy/utils/data.pyts"                   N9A % (    !  4 = '