ó ¦–Õ\c@`sëddlmZmZmZddlZddlZddlZddlmZddl m Z ddl m Z ddZ ejd ƒZejd ƒZd efd „ƒYZeejd]*?\s+)?href=(["'])(.*?)\1s&(http[s]?://[-a-zA-Z0-9@:%_+.~#?&/=]+)tHTTPFileSystemcB`sneZdZdZed ed„Zed„Zd„Z d„Z d„Z dd d„Z d „Z d „ZRS( s¤ Simple File-System for fetching data via HTTP(S) Unlike other file-systems, HTTP is limited in that it does not provide glob or write capability. t/cK`sI|dk r|nt|_||_||_||_tjƒ|_dS(s¢ Parameters ---------- block_size: int Blocks to read bytes; if 0, will default to raw requests file-like objects instead of HTTPFile instances simple_links: bool If True, will consider both HTML tags and anything that looks like a URL; if False, will consider only the former. same_schema: bool For ls, glob: only return paths having the same schema (http/https) as the original URL. storage_options: key-value May be credentials, e.g., `{'auth': ('username', 'pword')}` or any other parameters passed on to requests N( tNonetDEFAULT_BLOCK_SIZEt block_sizet simple_linkst same_schematkwargstrequeststSessiontsession(tselfR R R tstorage_options((s.lib/python2.7/site-packages/dask/bytes/http.pyt__init__s     c C`stj||j}|jrCtj|jƒtj|jƒ}ntj|jƒ}tƒ}t |ƒ}x;|D]3}t |t ƒr“|d}n|j dƒr$|j rê|jddƒd|jddƒdkr!|j|ƒq!q¤|jddƒj |jddƒƒr¤|j|ƒq¤qq|j dƒrgt|ƒdkrg|j|jd|j|ƒqq|dkrq|jdj|jdƒ|jdƒgƒƒqqqqW|||dh}|rüg|D]/}i|d 6|j|ƒrîd nd d6^qÉStt|ƒƒSdS(Nithttpt:ithttpsRs://s..s../ttnamet directorytfilettype(s..s../R(RtgetR R tex2tfindallttexttextsetRt isinstancettuplet startswithR tsplittaddtreplacetlentschemetnetloctjointrstriptlstriptisdirtlisttsorted( Rturltdetailtrtlinkstouttpartstltu((s.lib/python2.7/site-packages/dask/bytes/http.pytls3s2 %     ,!" 57cC`s t‚dS(s7Make any intermediate directories to make path writableN(tNotImplementedError(RR1((s.lib/python2.7/site-packages/dask/bytes/http.pytmkdirsTscC`stS(N(tTrue(Rtpath((s.lib/python2.7/site-packages/dask/bytes/http.pyR.XscC`stt|t|ƒƒS(N(R0Rt posixpath(RR=((s.lib/python2.7/site-packages/dask/bytes/http.pytglob[strbcK`sž|dkrt‚n|dk r'|n|j}|rOt||j||jS|jjƒ}t|d<|jj||}|j ƒt|j _ |j SdS(shMake a file-like object Parameters ---------- url: str Full URL with protocol mode: string must be "rb" block_size: int or None Bytes to download in one request; use instance value if None. kwargs: key-value Any other parameters, passed to requests calls R@tstreamN( R:RR tHTTPFileRR tcopyR<Rtraise_for_statustrawtdecode_content(RR1tmodeR R tkwR3((s.lib/python2.7/site-packages/dask/bytes/http.pytopen^s     cC`s ddlm}|||jƒS(s;Unique identifier; assume HTTP files are static, unchangingi(ttokenize(t dask.baseRJR (RR1RJ((s.lib/python2.7/site-packages/dask/bytes/http.pytukeyyscC`st|d|j|jS(s!Size in bytes of the file at pathR(t file_sizeRR (RR1((s.lib/python2.7/site-packages/dask/bytes/http.pytsize~sN(t__name__t __module__t__doc__tsepR<RRtFalseR9R;R.R?RIRLRN(((s.lib/python2.7/site-packages/dask/bytes/http.pyRs !    RRRBcB`sªeZdZddd„Zdd„Zd„Zdd„Zd„Zd„Z d „Z d „Z d „Z d „Z d „Zd„Zd„Zd„Zd„Zd„ZRS(sø A file-like object pointing to a remove HTTP(S) resource Supports only reading, with read-ahead of a predermined block-size. In the case that the server does not supply the filesize, only reading of the complete file in one go is supported. Parameters ---------- url: str Full URL of the remote resource, including the protocol session: requests.Session or None All calls will be made within this session, to avoid restarting connections where the server allows this block_size: int or None The amount of read-ahead to do, in bytes. Default is 5MB, or the value configured for the FileSystem creating this file. kwargs: all other key-values are passed to reqeuests calls. cK`s||_||_d|_|dk r-|n tjƒ|_|dk rN|nt|_y%t ||jdt |j|_ Wnnt tj fk r¡d|_ nLtj k rì}|jj}|dks×|dkrà|‚nd|_ nXd|_t|_d|_d|_dS( Nitallow_redirectsiôi‘i“i”R(i‘i“i”(R1R tlocRRRRR t blocksizeRMR<RNt ValueErrort HTTPErrortresponset status_codetcacheRStclosedtstarttend(RR1RR R terrtcode((s.lib/python2.7/site-packages/dask/bytes/http.pyRs(   !        icC`s»|jdkr6||fdd gkr6tdƒ‚n|dkrK|}nH|dkrg|j|}n,|dkrƒ|j|}ntd|ƒ‚|dkr®tdƒ‚n||_|S( stSet file position Parameters ---------- where: int Location to set whence: int (default 0) If zero, set from start of file (value should be positive); if 1, set relative to current position; if 2, set relative to end of file (value shoulf be negative) Returns the position. iis+Cannot seek since size of file is not knownis'Whence must be in [1, 2, 3], but got %ssSeek before start of fileN(ii(ii(RNRRWRU(Rtwheretwhencetnloc((s.lib/python2.7/site-packages/dask/bytes/http.pytseek¹s'      cC`s|jS(sGet current file byte position(RU(R((s.lib/python2.7/site-packages/dask/bytes/http.pyttellÖsiÿÿÿÿcC`sþ|dkrdS|dkr5|jdkr5|jƒS|jdkrl|dkr_tdƒ‚ql|jƒSn|dksŽ|j||jkrš|j}n |j|}|j|jkr½dS|j|j|ƒ|j|j|j||j!}||_|S(s5Read bytes from file Parameters ---------- length: int Read up to this many bytes. If negative, read all content to end of file. If the server has not supplied the filesize, attempting to read only part of the data will raise a ValueError. iRs(File size is unknown, must read all dataN(RUt _fetch_allRNRRWt_fetchR[R](RtlengthR^tdata((s.lib/python2.7/site-packages/dask/bytes/http.pytreadÚs"    "  ! cC`s”|jdkrR|jdkrR||_||j|_|j||jƒ|_n>||jkrß|j||jkr®||_||j|_|j|j|jƒ|_q|j||jƒ}||_||j|_n±||jkr|j|jkrdS||j|jkrQ||_||j|_|j|j|jƒ|_q|j|j||jƒ}||j|_|j||_ndS(s9Set new bounds for data cache and fetch data, if requiredN(R]RR^RVt _fetch_rangeR[RN(RR]R^tnew((s.lib/python2.7/site-packages/dask/bytes/http.pyRgþs,    cC`sƒ|jj|j|j}|jƒ|j}t|ƒ}||jkrpd|_||_ ||_ ||_ nt|ƒ|_ |S(sµRead whole file in one shot, without caching This is only called when size is None or position is still at zero, and read() is called without a byte-count. i( RRR1R RDtcontentR(RVR]R^R[RNRU(RR3R5R7((s.lib/python2.7/site-packages/dask/bytes/http.pyRfs       c C`s]|jjƒ}|jdiƒ}d||df|d<|jj|jd|dt|}|jƒ|jdkr}|j Sd|j krÓt |j dƒ}|||kr¶|j St d|||fƒ‚nd }g}xn|j d dƒD]Z}|rK|j|ƒ|t|ƒ7}|||krLt d |||fƒ‚qLqòPqòWdj|ƒS(s3Download a block of data The expectation is that the server returns only the requested bytes, with HTTP code 206. If this is not the case, we first check the headers, and then stream the output - if the data size is bigger than we requested, an exception is raised. theaderss bytes=%i-%iitRangeRAiÎsContent-Lengths'Got more bytes (%i) than requested (%i)it chunk_sizeiis/Got more bytes so far (>%i) than requested (%i)Ri(R RCtpopRRR1R<RDRZRmRntintRWt iter_contenttappendR(R+( RR]R^R RnR3tclR5tchunk((s.lib/python2.7/site-packages/dask/bytes/http.pyRk.s2$  cC`s d|_|S(Ni(RU(R((s.lib/python2.7/site-packages/dask/bytes/http.pyt __enter__Us cG`s|jƒdS(N(tclose(Rtargs((s.lib/python2.7/site-packages/dask/bytes/http.pyt__exit__YscC`s t‚dS(N(R:(R((s.lib/python2.7/site-packages/dask/bytes/http.pyt__iter__\scC`s t‚dS(N(R:(R((s.lib/python2.7/site-packages/dask/bytes/http.pytwrite`scC`sdS(N((R((s.lib/python2.7/site-packages/dask/bytes/http.pytflushcscC`s t|_dS(N(R<R\(R((s.lib/python2.7/site-packages/dask/bytes/http.pyRxfscC`stS(N(R<(R((s.lib/python2.7/site-packages/dask/bytes/http.pytseekableiscC`stS(N(RS(R((s.lib/python2.7/site-packages/dask/bytes/http.pytwritablelscC`stS(N(R<(R((s.lib/python2.7/site-packages/dask/bytes/http.pytreadableosN(RORPRQRRRdReRjRgRfRkRwRzR{R|R}RxR~RR€(((s.lib/python2.7/site-packages/dask/bytes/http.pyRB‡s"   $   '        cK`s|jƒ}|jdtƒ}|jdiƒ}d|d<|j|d||}|jƒd|jkr|t|jdƒStd|ƒ‚dS(sºCall HEAD on the server to get file size Default operation is to explicitly allow redirects and use encoding 'identity' (no compression) to get the true size of the target. RTRntidentitysAccept-EncodingsContent-Lengths Server did not supply size of %sN( RCRqR<RtheadRDRnRrRW(R1RR tarR‚R3((s.lib/python2.7/site-packages/dask/bytes/http.pyRMss   i(t __future__RRRR>treRRRR?Rt compatibilityRR tcompileR RtobjectRt _filesystemsRBRM(((s.lib/python2.7/site-packages/dask/bytes/http.pyts    r  ì