九Python之HTML的解析（网页抓取一） -

zhdkn

浏览: 103679 次
性别:
来自: 北京

最近访客更多访客>>

suchiheng6

zhuxinyu

yueguifang1

aa00aa00

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

九Python之HTML的解析（网页抓取一）

博客分类：

Python之旅

Python HTMLParser html的解析

对html的解析是网页抓取的基础，分析抓取的结果找到自己想要的内容或标签以达到抓取的目的。

HTMLParser是python用来解析html的模块。它可以分析出html里面的标签、数据等等，是一种处理html的简便途径。 HTMLParser采用的是一种事件驱动的模式，当HTMLParser找到一个特定的标记时，它会去调用一个用户定义的函数，以此来通知程序处理。它主要的用户回调函数的命名都是以handler_开头的，都是HTMLParser的成员函数。当我们使用时，就从HTMLParser派生出新的类，然后重新定义这几个以handler_开头的函数即可。这几个函数包括：

handle_startendtag 处理开始标签和结束标签
handle_starttag 处理开始标签，比如<xx>
handle_endtag 处理结束标签，比如</xx>
handle_charref 处理特殊字符串，就是以&#开头的，一般是内码表示的字符
handle_entityref 处理一些特殊字符，以&开头的，比如  
handle_data 处理数据，就是<xx>data</xx>中间的那些数据
handle_comment 处理注释
handle_decl 处理<!开头的，比如<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.01 Transitional//EN”
handle_pi 处理形如<?instruction>的东西

1. 基本解析，找到开始和结束标签

from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):

    def handle_starttag(self, tag, attrs):
        print("Encountered the beginning of a %s tag" %(tag))

    def handle_endtag(self, tag):
        print ("Encountered the end of a %s tag" %(tag))

if __name__ == '__main__':
    a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'

    m = MyHTMLParser()
    #传入要分析的html模块
    m.feed(a)

运行结果

Encountered the beginning of a html tag
Encountered the beginning of a head tag
Encountered the beginning of a title tag
Encountered the end of a title tag
Encountered the beginning of a body tag
Encountered the beginning of a a tag
Encountered the end of a a tag
Encountered the end of a body tag
Encountered the end of a html tag

2. 解析html的超链接和链接显示的内容

from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):
    def __init__(self):
        HTMLParser.__init__(self)
        self.flag=None

    def handle_starttag(self, tag, attrs):
    # 这里重新定义了处理开始标签的函数
        if tag == 'a':
    # 判断标签<a>的属性
            self.flag='a'
            for name,value in attrs:
                if name == 'href':
                    print("href:"+value)

    def handle_data(self,data):
        if self.flag == 'a':
            print("data:"+data)

if __name__ == '__main__':
    a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'
    my = MyHTMLParser()
    my.feed(a)

运行结果

href:http: //www.163.com
data:链接到163

3. 完整解析html元素，拼接后原样输出

from html.parser import HTMLParser
import html.entities

class BaseHTMLProcessor(HTMLParser):
	def reset(self):
		# extend (called by HTMLParser.__init__)
		self.pieces = []
		HTMLParser.reset(self)

	def handle_starttag(self, tag, attrs):
		# called for each start tag
		# attrs is a list of (attr, value) tuples
		# e.g. for <pre class="screen">, tag="pre", attrs=[("class", "screen")]
		# Ideally we would like to reconstruct original tag and attributes, but
		# we may end up quoting attribute values that weren't quoted in the source
		# document, or we may change the type of quotes around the attribute value
		# (single to double quotes).
		# Note that improperly embedded non-HTML code (like client-side Javascript)
		# may be parsed incorrectly by the ancestor, causing runtime script errors.
		# All non-HTML code must be enclosed in HTML comment tags (<!-- code -->)
		# to ensure that it will pass through this parser unaltered (in handle_comment).
		strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
		self.pieces.append("<%(tag)s%(strattrs)s>" % locals())

	def handle_endtag(self, tag):
		# called for each end tag, e.g. for </pre>, tag will be "pre"
		# Reconstruct the original end tag.
		self.pieces.append("</%(tag)s>" % locals())

	def handle_charref(self, ref):
		# called for each character reference, e.g. for "&#160;", ref will be "160"
		# Reconstruct the original character reference.
		self.pieces.append("&#%(ref)s;" % locals())

	def handle_entityref(self, ref):
		# called for each entity reference, e.g. for "&copy;", ref will be "copy"
		# Reconstruct the original entity reference.
		self.pieces.append("&%(ref)s" % locals())
		# standard HTML entities are closed with a semicolon; other entities are not
		if entities.entitydefs.has_key(ref):
			self.pieces.append(";")

	def handle_data(self, text):
		# called for each block of plain text, i.e. outside of any tag and
		# not containing any character or entity references
		# Store the original text verbatim.
		self.pieces.append(text)

	def handle_comment(self, text):
		# called for each HTML comment, e.g. <!-- insert Javascript code here -->
		# Reconstruct the original comment.
		# It is especially important that the source document enclose client-side
		# code (like Javascript) within comments so it can pass through this
		# processor undisturbed; see comments in unknown_starttag for details.
		self.pieces.append("<!--%(text)s-->" % locals())

	def handle_pi(self, text):
		# called for each processing instruction, e.g. <?instruction>
		# Reconstruct original processing instruction.
		self.pieces.append("<?%(text)s>" % locals())

	def handle_decl(self, text):
		# called for the DOCTYPE, if present, e.g.
		# <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
		#	 "http://www.w3.org/TR/html4/loose.dtd">
		# Reconstruct original DOCTYPE
		self.pieces.append("<!%(text)s>" % locals())

	def output(self):
		"""Return processed HTML as a single string"""
		return "".join(self.pieces)

if __name__ == '__main__':
    a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'
    
    bhp =BaseHTMLProcessor()
    bhp.feed(a)
    print(bhp.output())

运行结果

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>

0
顶

2
踩

分享到：

十Python之Http Web服务（网页抓取二） | 八Python正则表达式

2013-03-05 08:29
浏览 10587
评论(0)
分类:编程语言
查看更多

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

九Python之HTML的解析（网页抓取一）

评论

发表评论

相关推荐

最近访客 更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论

九Python之HTML的解析（网页抓取一）

评论

发表评论

相关推荐

十一Python网页抓取实战（网页抓取三）

十Python之Http Web服务（网页抓取二）

八Python正则表达式

七Python异常和文件处理（二）

六Python异常和文件处理（一）

五Python对象和面向对象（二）

四Python对象和面向对象（一）

三Python内置函数

二Python内置数据类型

一Python函数及对象

Python写在开始

最近访客更多访客>>