python实现抓取某站的sitemap.xml，获取url、title和描述

Klock0828

感谢分享，论坛因你而精彩！

806785900

感谢楼主分享，用xpath解析重写一下！

[Python] 纯文本查看 复制代码

import requests
import re
import parsel

# 请求XML文件
url = 'https://www.xxxxxxxxxxxxxxxx/sitemap.xml'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)

# 解析XML文件
selector = parsel.Selector(response.text)
urls = selector.xpath('//loc/text()').getall()

# 匹配URL的后缀名为.html的网页并获取title、description和keywords
pattern = re.compile(r'.*\.html$')
for url in urls:
    if re.match(pattern, url):
        response = requests.get(url, headers=headers)
        selector = parsel.Selector(response.text)
        title = selector.css('title::text').get()
        keywords = selector.xpath('//meta[@name="keywords"]/@content')
        description = selector.xpath('//meta[@name="description"]/@content')

        # 保存到txt文件
        with open('output.txt', 'a', encoding='utf-8') as f:
            f.write('Title: {}\n'.format(title))
            f.write('Keywords: {}\n'.format(keywords.get()))
            f.write('Description: {}\n'.format(description.get()))
            f.write('URL: {}\n\n'.format(url))

Alexwhich

好像得另外下载解析库才能解析xml数据

jiuzhou888

学习了，感谢大神的原创，必须顶上去。

echoaku

不错，实用，收藏了