百度小说全文爬取

aiyamaya

输入小说的ID号，该ID号可在 中获得。
默认存放路径D:/小说
下载出现1KB文件情况，主要是因为收费章节的原因。目前只能下载免费章节（收费如何下载，还没弄明白）

[Python] 纯文本查看 复制代码

import time,os
import requests
import asyncio
import aiohttp
import aiofiles

async def get_chapters_ids(n_id):
    book_url = f'https://dushu.baidu.com/api/pc/getCatalog?data=%7B"book_id":{n_id}%7D'
    t_start = int(time.time())
    tasks =[]
    with requests.get(book_url) as respon:
        dic = respon.json()
        for i in dic['data']['novel']['items']:
            title = i['title']
            chapter_id = i['cid']
            tasks.append(asyncio.create_task(get_chapters(n_id,title,chapter_id)))
        await asyncio.wait(tasks)
    t_over = int(time.time())
    print('下载完毕!')
    print('共用时:',t_over-t_start,'秒')

async def get_chapters(n_id,title,chapter_id):
    chapter_url = f'https://dushu.baidu.com/api/pc/getChapterContent?data=%7B"book_id":"{n_id}","cid":"{n_id}|{chapter_id}","need_bookinfo":1%7D'
    # print(chapter_url)
    async with aiohttp.ClientSession() as req:
        async with req.get(chapter_url) as respon:
            dic = await respon.json()
            async with aiofiles.open(f'D:\小说\{title}.txt',mode='w',encoding='utf-8') as f:
                await f.write(dic['data']['novel']['content'])
    print(title,'下载完成')

if __name__ =='__main__':
    if not os.path.exists(r'd:\小说'):
        os.mkdir(r'd:\小说')
    novel_id = input('输入小说编号:')
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(get_chapters_ids(novel_id))

waweiggfnh

测试能下载，但是大都是1KB，内容不全

aiyamaya

waweiggfnh 发表于 2022-5-10 09:12
测试能下载，但是大都是1KB，内容不全

1kb的是收费内容，忘记说了，这个只能下载免费章节

Yun3369

试试看，感谢楼主。

erhu70b

怎么用呢

woawapj

学习一下了

ZhuanZhuYuIT

感谢分享

bilibini

感谢分享，先收藏起，以后应该可以用到

haiyangnanzi

小白不会用，哎

tianxinnuhuo

这个就不会用了

zhongzhi72

小白不会用，哎

ych13846701169

太需要了，谢谢分享