吾爱破解 - 52pojie.cn

 找回密码
 注册[Register]

QQ登录

只需一步,快速开始

查看: 1832|回复: 1
收起左侧

[会员申请] 申请会员ID:夜星.py

[复制链接]
吾爱游客  发表于 2022-9-17 12:01
1、申请ID:夜星.py
2、个人邮箱:zwm14755773273@outlook.com
3、原创技术文章:
python利用线程池快速下载某趣阁小说
1.首先我们先导入我们需要的包
import requests
from lxml import etree
import os
from concurrent.futures import ThreadPoolExecutor #线程池
1.然后从一本书中获取各个章节的链接
url = 'https://www.biquge9.com/book/3808/'
res = requests.get(url=url)
t = etree.HTML(res.text)
i = t.xpath('//div[@class="listmain"]/dl/dd/a/@href')
o = i[:10]
p = i[11:]
b = t.xpath('//div[@class="listmain"]/dl/span/dd/a/@href')
z = o + b + p
ur = []
for v in z:
    ur.append('https://www.biquge9.com' + v)
3.定义一个函数从章节链接中获取内容
def download(url,file):
    re = requests.get(url=url)
    re.encoding = 'utf-8'
    if not os.path.exists(file):
        os.mkdir(file)
    tr = etree.HTML(re.text)
    name = tr.xpath('//div[@class="book reader"]//div[@class="content"]/h1/text()')[0]

    path = file + '/' + name + '.txt'
    fp = open(path, 'w', encoding='utf-8')
    q = tr.xpath('//*[@id="chaptercontent"]/text()')
    for nei in q:
        fp.write(nei + '\n')
        print(name + '爬取成功')
    fp.close()
总体思路就这样,然后创建线程池快速爬取小说

下面这是源码
import requests
from lxml import etree
import time
import os
from concurrent.futures import ThreadPoolExecutor
def download(url,file):
    re = requests.get(url=url)
    re.encoding = 'utf-8'
    if not os.path.exists(file):
        os.mkdir(file)
    tr = etree.HTML(re.text)
    name = tr.xpath('//div[@class="book reader"]//div[@class="content"]/h1/text()')[0]

    path = file + '/' + name + '.txt'
    fp = open(path, 'w', encoding='utf-8')
    q = tr.xpath('//*[@id="chaptercontent"]/text()')
    for nei in q:
        fp.write(nei + '\n')
        print(name + '爬取成功')
    fp.close()
if __name__ == '__main__':
    '''获取章节内容url'''
    print('新笔趣阁的网址为:https://www.biquge9.com')
    url = str(input('请输入新笔趣阁中书的网址链接:'))
    file = input('请输入文件夹的名字:')
    # url = 'https://www.biquge9.com/book/3808/'
    res = requests.get(url=url)
    t = etree.HTML(res.text)
    i = t.xpath('//div[@class="listmain"]/dl/dd/a/@href')
    o = i[:10]
    p = i[11:]
    b = t.xpath('//div[@class="listmain"]/dl/span/dd/a/@href')
    z = o + b + p
    ur = []
    for v in z:
        ur.append('https://www.biquge9.com' + v)
    with ThreadPoolExecutor(50)as f:
        for url in ur:
            f.submit(download,url=url,file=file)

发帖前要善用论坛搜索功能,那里可能会有你要找的答案或者已经有人发布过相同内容了,请勿重复发帖。

Hmily 发表于 2022-9-19 11:25
抱歉,未能达到申请要求,申请不通过,可以关注论坛官方微信(吾爱破解论坛),等待开放注册通知。
您需要登录后才可以回帖 登录 | 注册[Register]

本版积分规则

返回列表

RSS订阅|小黑屋|处罚记录|联系我们|吾爱破解 - LCG - LSG ( 京ICP备16042023号 | 京公网安备 11010502030087号 )

GMT+8, 2024-11-15 01:25

Powered by Discuz!

Copyright © 2001-2020, Tencent Cloud.

快速回复 返回顶部 返回列表