ó Q˜[c@`s³ddlmZmZmZddlmZddlmZddlm Z m Z e e_ e e_ dgZ defd„ƒYZdefd „ƒYZd efd „ƒYZd S( i(tabsolute_importtdivisiontunicode_literals(tstr(turllib(tparsetrequestuRobotFileParsertRobotFileParsercB`sbeZdZdd„Zd„Zd„Zd„Zd„Zd„Zd„Z d „Z d „Z RS( us This class provides a set of methods to read, parse and answer questions about a single robots.txt file. ucC`s>g|_d|_t|_t|_|j|ƒd|_dS(Ni(tentriestNonet default_entrytFalset disallow_allt allow_alltset_urlt last_checked(tselfturl((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyt__init__s      cC`s|jS(u·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. (R(R((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pytmtime&scC`sddl}|jƒ|_dS(uYSets the time the robots.txt file was last fetched to the current time. iN(ttimeR(RR((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pytmodified/s cC`s2||_tjj|ƒdd!\|_|_dS(u,Sets the URL referring to a robots.txt file.iiN(RRRturlparsethosttpath(RR((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR7s cC`s—ytjj|jƒ}WnOtjjk rj}|jdkrLt|_q“|jdkr“t|_ q“n)X|j ƒ}|j |j dƒj ƒƒdS(u4Reads the robots.txt URL and feeds it to the parser.i‘i“iuutf-8N(i‘i“(RRturlopenRterrort HTTPErrortcodetTrueR R treadRtdecodet splitlines(Rtfterrtraw((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR<s  cC`sAd|jkr-|jdkr=||_q=n|jj|ƒdS(Nu*(t useragentsR R Rtappend(Rtentry((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyt _add_entryIscC`sd}tƒ}xÝ|D]Õ}|sn|dkr@tƒ}d}qn|dkrn|j|ƒtƒ}d}qnn|jdƒ}|dkr–|| }n|jƒ}|s®qn|jddƒ}t|ƒdkr|djƒjƒ|dR@(((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyRs    3 R0cB`s)eZdZd„Zd„Zd„ZRS(uoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cC`s>|dkr| rt}ntjj|ƒ|_||_dS(Nu(RRRR9RR;(RRR;((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR£s cC`s|jdkp|j|jƒS(Nu*(Rt startswith(Rtfilename((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR:ªscC`s|jrdpdd|jS(NuAllowuDisallowu: (R;R(R((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR@­s(RARBRCRR:R@(((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR0 s  R(cB`s2eZdZd„Zd„Zd„Zd„ZRS(u?An entry has one or more user-agents and zero or more rulelinescC`sg|_g|_dS(N(R$R/(R((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR³s cC`sjg}x'|jD]}|jd|dgƒqWx*|jD]}|jt|ƒdgƒq:Wdj|ƒS(Nu User-agent: u u(R$textendR/RR?(RtrettagentR3((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR@·s cC`s]|jdƒdjƒ}x=|jD]2}|dkr9tS|jƒ}||kr#tSq#WtS(u2check if this entry applies to the specified agentu/iu*(R+R-R$RR (RR<RH((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR:¿s   cC`s.x'|jD]}|j|ƒr |jSq WtS(uZPreconditions: - our agent applies to this entry - filename is URL decoded(R/R:R;R(RRER3((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR;Ìs (RARBRCRR@R:R;(((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyR(±s    N(t __future__RRRtfuture.builtinsRtfuture.backportsRtfuture.backports.urllibRt_parseRt_requestt__all__tobjectRR0R((((sBlib/python2.7/site-packages/future/backports/urllib/robotparser.pyts   ˆ