怎么让url发生变化继续运行 向下运行 不能运行的报none值

lihu5841314

[Asm] 纯文本查看 复制代码

import  requests
import os
from  lxml import  etree
headers ={
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36"
}

url = "https://www.runoob.com/python/python-tutorial.html"
page_text = requests.get(url=url,headers=headers).text
tree =etree.HTML(page_text)
a_list = tree.xpath('.//div[@id="leftcolumn"]/a')[4:23]
if not os.path.exists('./cainiao'):
    os.mkdir('./cainiao')
for a in a_list:
    detail_url = "https://www.runoob.com"+a.xpath('./@href')[0]
    name = a.xpath('./text()')[0]
    name= name.strip()
    # print(detail_url,name)
    detail_page = requests.get(url=detail_url,headers=headers).text
    detail_tree = etree.HTML(detail_page)
    detail_text = detail_tree.xpath('.//div[@id="content"]//text()')
    path = './cainiao/' + name+ ".txt"
    for  text in detail_text:
        with open(path,"a",encoding="utf-8")  as  pf:
             pf.write(text)
    print(name,'下载成功')

南归不NG

想把菜鸟教程下载到本地?

一线码农

异常处理 不会？

lihu5841314

南归不NG 发表于 2021-5-28 16:44
想把菜鸟教程下载到本地?

详情页的url有变化  我只能把不变的切片  不然就就报错  用scrapy框架爬  就不会  无效的url直接就过了

fanvalen

两种方法
一种try excpet
另一种再可能出现none的地方加if 判断len是否>0 是就跳过本次continue

lihu5841314

fanvalen 发表于 2021-5-28 17:03
两种方法
一种try excpet
另一种再可能出现none的地方加if 判断len是否>0 是就跳过本次continue

我的想法也差不多

datalai

try处理可以吗