python实现抓取某站的sitemap.xml，获取url、title和描述 - 吾爱破解 - 52pojie.cn

oneai 发表于 2023-5-29 15:22

Klock0828 发表于 2023-5-29 18:36

感谢分享，论坛因你而精彩！

806785900 发表于 2023-5-29 20:39

感谢楼主分享，用xpath解析重写一下！
import requests
import re
import parsel

# 请求XML文件
url = 'https://www.xxxxxxxxxxxxxxxx/sitemap.xml'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)

# 解析XML文件
selector = parsel.Selector(response.text)
urls = selector.xpath('//loc/text()').getall()

# 匹配URL的后缀名为.html的网页并获取title、description和keywords
pattern = re.compile(r'.*\.html$')
for url in urls:
if re.match(pattern, url):
   response = requests.get(url, headers=headers)
   selector = parsel.Selector(response.text)
   title = selector.css('title::text').get()
   keywords = selector.xpath('//meta[@name="keywords"]/@content')
   description = selector.xpath('//meta[@name="description"]/@content')

   # 保存到txt文件
   with open('output.txt', 'a', encoding='utf-8') as f:
         f.write('Title: {}\n'.format(title))
         f.write('Keywords: {}\n'.format(keywords.get()))
         f.write('Description: {}\n'.format(description.get()))
         f.write('URL: {}\n\n'.format(url))

Alexwhich 发表于 2023-5-29 22:25

好像得另外下载解析库才能解析xml数据

jiuzhou888 发表于 2023-5-30 16:19

学习了，感谢大神的原创，必须顶上去。

echoaku 发表于 2023-5-30 16:30

不错，实用，收藏了

页: [1]

吾爱破解 - 52pojie.cn's Archiver