B
j9C\ÁZ ã @ s d Z dZdZdZdZdgZddlZddlZddlZddl Z ddl
Z
dd l m Z m
Z
dd
lmZ dd lmZmZmZmZmZmZmZmZmZmZmZ d d
k G dd„ deƒZ e Z e Z G dd„ de ƒZ G dd„ de ƒZ!G dd„ de"ƒZ#e$dkr
ddlZe ej%ƒZ&e'e& (¡ ƒ dS )aH Beautiful Soup
Elixir and Tonic
"The Screen-Scraper's Friend"
http://www.crummy.com/software/BeautifulSoup/
Beautiful Soup uses a pluggable XML or HTML parser to parse a
(possibly invalid) document into a tree representation. Beautiful Soup
provides methods and Pythonic idioms that make it easy to navigate,
search, and modify the parse tree.
Beautiful Soup works with Python 2.7 and up. It works better if lxml
and/or html5lib is installed.
For more than you ever wanted to know about Beautiful Soup, see the
documentation:
http://www.crummy.com/software/BeautifulSoup/bs4/doc/
z*Leonard Richardson (leonardr@segfault.org)z4.7.1z*Copyright (c) 2004-2019 Leonard RichardsonZMITÚ
BeautifulSoupé Né )Úbuilder_registryÚParserRejectedMarkup)Ú
UnicodeDammit) ÚCDataÚCommentÚDEFAULT_OUTPUT_ENCODINGÚ DeclarationÚDoctypeÚNavigableStringÚ PageElementÚProcessingInstructionÚ ResultSetÚ SoupStrainerÚTagz`You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.zuYou need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).c sð e Zd ZdZdZddgZdZdZd4d d
„Zd d „ Z d
d„ Z
e dd„ ƒZ dd„ Z
dd„ Zddi fdd„Zefdd„Zdd„ Zdd „ Zd d „ Zd d „ Zefd!d"„Zd5d#d$„Zd%d&„ Zd6d(d)„Zd*d+„ Zd7d,d-„Zd.d/„ Z d0e d1f‡ fd2d3„ Z ‡ Z S )8r a
This class defines the basic interface called by the tree builders.
These methods will be called by the parser:
reset()
feed(markup)
The tree builder may call these methods from its feed() implementation:
handle_starttag(name, attrs) # See note about return value
handle_endtag(name)
handle_data(data) # Appends to the current data node
endData(containerClass=NavigableString) # Ends the current data node
No matter how complicated the underlying parser is, you should be
able to build a tree using 'start tag' events, 'end tag' events,
'data' events, and "done with data" events.
If you encounter an empty-element tag (aka a self-closing tag,
like HTML's
tag), call handle_starttag and then
handle_endtag.
z
[document]ZhtmlZfastz
aí No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system ("%(parser)s"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
The code that caused this warning is on line %(line_number)s of the file %(filename)s. To get rid of this warning, pass the additional argument 'features="%(parser)s"' to the BeautifulSoup constructor.
Ú Nc
s– dˆ krt d¡ dˆ kr*ˆ d= t d¡ dˆ krBˆ d= t d¡ dˆ krZˆ d= t d¡ d ˆ krrˆ d = t d
¡ ‡ fd d „}|pŠ|d
dƒ}|p˜|ddƒ}|r¶t|tƒr¶t d¡ d}tˆ ƒdkrÞtˆ ¡ ƒ ¡ } td| ƒ‚|dkr|}
t|tƒrü|g}|dkst|ƒdkr| j }t
j |Ž } | dkr@t dd
|¡ ƒ‚| ƒ }|
|jks|
|jks|jrld} nd} d}
yt d¡}
W n tk
r˜ Y nX |
r®|
j}|
j}n
tj}d}| d¡}|rè| ¡ }| d¡rè|dd … }|rt|||j| d }t j| j| d d || _ |j| _| j| _ tƒ | _ || _ | j | ¡ t!|d ƒrZ| "¡ }nÎt|ƒd!kr(t|t#ƒr~d"|ks”t|tƒr(d#|kr(t|tƒr¶t$j%j&s¶| 'd$¡}n|}d%}yt$j% (|¡}W n$ t)k
rò } zW dd}~X Y nX |r t|tƒr| 'd$¡}t d&| ¡ | *|¡ xZ| j j+|||d'D ]D\| _,| _-| _.| _/| 0¡ y| 1¡ P W n t2k
r| Y nX q.deprecated_argumentZparseOnlyTheseÚ
parse_onlyZ fromEncodingÚ
from_encodingzlYou provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.Nr z2__init__() got an unexpected keyword argument '%s'zjCouldn't find a tree builder with the features you requested: %s. Do you need to install a parser library?ú,ZXMLZHTMLr Ú__file__)z.pycz.pyoéÿÿÿÿ)ÚfilenameÚ line_numberÚparserÚ markup_typeé )Ú
stacklevelÚreadé ó <ú<Úutf8Fzw"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.)Úexclude_encodings)4r r Ú
isinstanceÚstrÚlenÚlistÚkeysÚpopÚ TypeErrorÚDEFAULT_BUILDER_FEATURESr ÚlookupÚFeatureNotFoundÚjoinÚNAMEZALTERNATE_NAMESÚis_xmlÚsysÚ _getframeÚ
ValueErrorÚ f_globalsÚf_linenoÚ__dict__ÚgetÚlowerÚendswithÚdictÚNO_PARSER_SPECIFIED_WARNINGÚbuilderZ known_xmlZ _namespacesr Zinitialize_soupÚhasattrr& ÚbytesÚosÚpathÚsupports_unicode_filenamesÚencodeÚexistsÚ ExceptionÚ_check_markup_is_urlZprepare_markupÚmarkupÚoriginal_encodingZdeclared_html_encodingZ contains_replacement_charactersÚresetÚ_feedr Úsoup)ÚselfrN ÚfeaturesrD r r r+ r r ÚargZoriginal_featuresZ
builder_classr# ZcallerÚglobalsr! r ZfnlÚvaluesZpossible_filenameÚis_fileÚer )r r Ú__init__V sØ '
zBeautifulSoup.__init__c C s& t | ƒ| d¡| jdd}| j|_|S )Nzutf-8)rD r )ÚtyperJ rD rO )rS Úcopyr r r Ú__copy__# s zBeautifulSoup.__copy__c C s& t | jƒ}d|kr"| jjs"d |d< |S )NrD )rB r> rD Z picklable)rS Údr r r Ú __getstate__/ s
zBeautifulSoup.__getstate__c sx t ˆ tƒrd}d}nt ˆ tƒr(d}d}ndS t‡ fdd„|D ƒƒrt|ˆ krtt ˆ tƒrbˆ dd ¡}nˆ }t d
| ¡ dS ) z—
Check if markup looks like it's actually a url and raise a warning
if so. Markup can be unicode or str (py2) / bytes (py3).
ó )s http:s https:ú )zhttp:zhttps:Nc 3 s | ]}ˆ |¡V qd S )N)Ú
startswith)Ú.0Úprefix)rN r r ú E s z5BeautifulSoup._check_markup_is_url..zutf-8Úreplacez¿"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client like requests to get the document behind the URL, and feed that document to Beautiful Soup.)r, rF r- ÚanyÚdecoder r )rN ZspaceZcant_start_withZdecoded_markupr )rN r rM 6 s
z"BeautifulSoup._check_markup_is_urlc C s@ | j ¡ | j | j¡ | ¡ x| jj| jkr:| ¡ q"W d S )N) rD rP ZfeedrN ÚendDataÚ
currentTagÚnameÚ
ROOT_TAG_NAMEÚpopTag)rS r r r rQ R s
zBeautifulSoup._feedc C sJ t | | | j| j¡ d| _| j ¡ g | _d | _g | _g | _ |
| ¡ d S )Nr ) r rZ rD rl ZhiddenrP Ú current_datarj ÚtagStackÚ preserve_whitespace_tag_stackÚpushTag)rS r r r rP \ s
zBeautifulSoup.resetc K s | |¡ td| j||||ƒS )z+Create a new tag associated with this soup.N)Úupdater rD )rS rk Ú namespaceÚnsprefixÚattrsZkwattrsr r r Únew_tagf s
zBeautifulSoup.new_tagc C s ||ƒS )z7Create a new NavigableString associated with this soup.r )rS ÚsÚsubclassr r r Ú
new_stringk s zBeautifulSoup.new_stringc C s t dƒ‚d S )Nz4BeautifulSoup objects don't support insert_before().)ÚNotImplementedError)rS Ú successorr r r Ú
insert_beforeo s zBeautifulSoup.insert_beforec C s t dƒ‚d S )Nz3BeautifulSoup objects don't support insert_after().)rz )rS r{ r r r Ú insert_afterr s zBeautifulSoup.insert_afterc C s@ | j ¡ }| jr(|| jd kr(| j ¡ | j r:| j d | _| jS )Nr )ro r1 rp rj )rS Útagr r r rm u s
zBeautifulSoup.popTagc C sN | j d k r| j j |¡ | j |¡ | jd | _ |j| jjkrJ| j |¡ d S )Nr )rj ÚcontentsÚappendro rk rD Zpreserve_whitespace_tagsrp )rS r~ r r r rq ~ s
zBeautifulSoup.pushTagc C s˜ | j r”d | j ¡}| jsPd}x |D ]}|| jkr"d}P q"W |rPd|krLd}nd}g | _ | jr‚t| jƒdkr‚| jjr~| j |¡s‚d S ||ƒ}| |¡ d S )Nr TFÚ
ra r )
rn r6 rp Ú ASCII_SPACESr r. ro ÚtextÚsearchÚobject_was_parsed)rS ZcontainerClassrn Z
strippableÚiÚor r r ri ‡ s&
zBeautifulSoup.endDatac C s˜ |dkr| j }|dk r |}n| j}d } }}t|tƒrX|j}|j}|j}|dkrX|j}|jdk }| |||||¡ || _|j
|¡ |r”| |¡ dS )z Add an object to the parse tree.N) rj Ú_most_recent_elementr, r Ú next_elementÚ next_siblingÚprevious_siblingÚprevious_elementZsetupr r€ Ú_linkage_fixer) rS r‡ ÚparentZmost_recent_elementrŒ r‰ r‹ rŠ Zfixr r r r… ¥ s$
z BeautifulSoup.object_was_parsedc C sÀ |j d }|j d }|}||krX|jdk rX||_|j}|dk rL||k rLd|_||_d|_d|_t|tƒrx|j rx| d¡}d|_d|_|}x2|dkr–P n |jdk r²|j|_||j_P |j}qŠW dS )z,Make sure linkage of this fragment is sound.r r NF) r rŽ r‰ rŒ r‹ rŠ r, r Z_last_descendant)rS ZelÚfirstZchildZ
descendantZprev_elÚtargetr r r r Á s0
z BeautifulSoup._linkage_fixerTc C sn || j krdS d}t| jƒ}xLt|d ddƒD ]8}| j| }||jkr^||jkr^|r\| ¡ }P | ¡ }q.W |S )zÜPops the tag stack up to and including the most recent
instance of the given tag. If inclusivePop is false, pops the tag
stack up to but *not* including the most recent instqance of
the given tag.Nr r r )rl r. ro Úrangerk rd rm )rS rk rt Z inclusivePopZmost_recently_poppedÚ
stack_sizer† Útr r r Ú _popToTagé s
zBeautifulSoup._popToTagc C s„ | ¡ | jr6t| jƒdkr6| jjs2| j ||¡s6dS t| | j||||| j| j ƒ}|dkr^|S | j dk rp|| j _
|| _ | |¡ |S )a Push a start tag on to the stack.
If this method returns None, the tag was rejected by the
SoupStrainer. You should proceed as if the tag had not occurred
in the document. For instance, if this was a self-closing tag,
don't call handle_endtag.
r N) ri r r. ro rƒ Z
search_tagr rD rj rˆ r‰ rq )rS rk rs rt ru r~ r r r Úhandle_starttag s
z BeautifulSoup.handle_starttagc C s | ¡ | ||¡ d S )N)ri r” )rS rk rt r r r Ú
handle_endtag s zBeautifulSoup.handle_endtagc C s | j |¡ d S )N)rn r€ )rS Údatar r r Ú handle_data s zBeautifulSoup.handle_dataFZminimalc sN | j r$d}|dkrd| }d| }nd}|s2d}nd}|tt| ƒ |||¡ S )zlReturns a string or Unicode representation of this document.
To get Unicode, pass None for encoding.r Nz encoding="%s"z
r )r8 Úsuperr rh )rS Z pretty_printZeventual_encodingZ formatterZ
encoding_partrd Z indent_level)Ú __class__r r rh # s
zBeautifulSoup.decode)r NNNNN)NN)NT)N) Ú__name__Ú
__module__Ú __qualname__Ú__doc__rl r3 r‚ rC rZ r] r_ Ú staticmethodrM rQ rP rv r ry r| r} rm rq ri r… r r” r• r– r˜ r rh Ú
__classcell__r r )rš r r 6 s:
M
(
c s e Zd ZdZ‡ fdd„Z‡ ZS )ÚBeautifulStoneSoupz&Deprecated interface to an XML parser.c s( d|d< t d¡ tt| ƒj||Ž d S )NZxmlrT zxThe BeautifulStoneSoup class is deprecated. Instead of using it, pass features="xml" into the BeautifulSoup constructor.)r r r™ r¡ rZ )rS Úargsr )rš r r rZ ? s zBeautifulStoneSoup.__init__)r› rœ r rž rZ r r r )rš r r¡ < s r¡ c @ s e Zd ZdS )Ú StopParsingN)r› rœ r r r r r r£ G s r£ c @ s e Zd ZdS )r5 N)r› rœ r r r r r r5 J s r5 Ú__main__))rž Ú
__author__Ú __version__Z
__copyright__Z __license__Ú__all__rG Úrer9 Ú tracebackr rD r r Zdammitr Úelementr r r r
r r r
r r r r r Z_sZ_soupr¡ rL r£ r; r5 r› ÚstdinrR ÚprintZprettifyr r r r Ú s8 4