50行java代码爬取某站整部小说保存到本地

甘霖之霜 · 发表于 2023-4-13 21:57

本帖最后由甘霖之霜于 2023-4-13 22:00 编辑

思路
从书的起始页出发，先获取整部书的目录
根据目录来到对应的章节的详情页，然后开爬。

开敲
1.先导入依赖

[XML] 纯文本查看 复制代码

 
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.15.4</version>
        </dependency>

        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.11.0</version>
        </dependency>

2.获取目录
参数说明一下 root是书目录起始页 next是目录的下一页 dir是收集各章节网址的集合

[Java] 纯文本查看 复制代码

private static void getDir(String root,String next,List<String> dir) throws Exception {
       Document document = Jsoup.connect(next).get();
       Elements elements = document.select("a[href$=\".html\"]");
       List<String> list = elements.eachAttr("href");
       list.remove(0);
       if (elements.last().text().equals("下一页")){
            String nextPage = list.get(list.size() - 1);
            nextPage = root +  nextPage.substring(nextPage.lastIndexOf("/") + 1);
            list.remove(list.size() - 1);
            if (elements.get(elements.size() - 2).text().equals("上一页")){
                list.remove(list.size() - 1);
            }
            dir.addAll(list);
            getDir(root,nextPage,dir);
            return;
       }
       if ((elements.last().text().equals("上一页"))){
           list.remove(list.size() - 1);
       }
       dir.addAll(list);
   }

3.根据目录获取章节信息写入文件
参数说明：dir刚才收集的目录 root是书目录起始页 writer用于将书写到文件中

[Asm] 纯文本查看 复制代码

private static void getContent(List<String> dir,String root, Writer writer) throws Exception {
        StringBuilder temp = new StringBuilder();
        for (String url : dir) {
           Document document = Jsoup.connect(root + url).get();
           String title = document.select("h1").text() + "\n";
           System.out.println(title);
           Elements content = document.select("div[id=\"content\"]");
           String text = content.toString();
           int i = text.indexOf("&");
           if (i != -1){
               text = text.substring(i);
           }
           text = text.replaceAll("    ","").replaceAll("<br><br>","").replaceAll("</div>","");
           temp.append(title + text);
       }
       IOUtils.write(temp,writer);
       writer.close();
       IOUtils.close();
   }

总的代码

[Java] 纯文本查看 复制代码

public class Soup {
    public static void main(String[] args) throws Exception {
        String url = "https://www.bbiquge.net/book/132488/";
        String fileName = Jsoup.connect(url).get().select("h1").text();
        fileName = fileName.replace("/","") + ".txt";
        File file = new File(fileName);
        Writer writer =new FileWriter(file,true);
        List<String> dir = new ArrayList<>();
        getDir(url,url,dir);
        getContent(dir,url,writer);
    }
   private static void getDir(String root,String next,List<String> dir) throws Exception {
       Document document = Jsoup.connect(next).get();
       Elements elements = document.select("a[href$=\".html\"]");
       List<String> list = elements.eachAttr("href");
       list.remove(0);
       if (elements.last().text().equals("下一页")){
            String nextPage = list.get(list.size() - 1);
            nextPage = root +  nextPage.substring(nextPage.lastIndexOf("/") + 1);
            list.remove(list.size() - 1);
            if (elements.get(elements.size() - 2).text().equals("上一页")){
                list.remove(list.size() - 1);
            }
            dir.addAll(list);
            getDir(root,nextPage,dir);
            return;
       }
       if ((elements.last().text().equals("上一页"))){
           list.remove(list.size() - 1);
       }
       dir.addAll(list);
   }
   private static void getContent(List<String> dir,String root, Writer writer) throws Exception {
        StringBuilder temp = new StringBuilder();
        for (String url : dir) {
           Document document = Jsoup.connect(root + url).get();
           String title = document.select("h1").text() + "\n";
           System.out.println(title);
           Elements content = document.select("div[id=\"content\"]");
           String text = content.toString();
           int i = text.indexOf("&");
           if (i != -1){
               text = text.substring(i);
           }
           text = text.replaceAll("    ","").replaceAll("<br><br>","").replaceAll("</div>","");
           temp.append(title + text);
       }
       IOUtils.write(temp,writer);
       writer.close();
       IOUtils.close();
   }
}

十分钟写出来的，运行起来效率感人也是，希望各位能说些优化方案学习一下

前途难定 · 发表于 2023-4-13 22:39

String url = "https://www.bbiquge.net/book/132488/";这个网址换成https://www.qidian.com/这个刑不刑啊

luxingyu329 · 发表于 2023-4-14 00:25

我来凑凑热闹，手生了，回忆回忆

[Python] 纯文本查看 复制代码

import time
import requests
import parsel

headers = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.35"
}
for page in range(1, 24):
    url = f"https://www.bbiquge.net/book/132488/index_{page}.html"
    response = requests.get(url, headers)
    if response.status_code == 200:
        content = response.text
        # print(content)
        sel = parsel.Selector(content)
        title_url = sel.css(".zjlist a::attr(href)").extract()
        # title_text = sel.css(".zjlist a::text").extract()    
        for j in range(0, len(title_url)):    
            response_d = requests.get(f"https://www.bbiquge.net/book/132488/{title_url[j]}", headers).text
            # print(f"https://www.bbiquge.net/book/132488/{title_url[j]}"f"https://www.bbiquge.net/book/132488/{title_url[j]}")
    
            content_d = parsel.Selector(response_d)
            text = content_d.css("#content *::text").extract()
            title = content_d.css("h1::text").extract_first()
   
            print(title, "   保存中......")    
            with open(“title.txt", "w", encoding="utf-8") as fw:
                for i in text[3:-1]:
                    # print(i.strip())
                    fw.write(i.strip() + "\n")
            time.sleep(5)

brheart92 · 发表于 2023-4-14 11:09

写的不错，学习了。推荐优化：如果网站测试没有反爬设计，推荐爬取章节内容用线程池。速度会更快一些。毕竟现在的小说动不动就是上千章

huangdanhuakui · 发表于 2023-4-13 22:37

厉害厉害

前途难定 · 发表于 2023-4-13 22:37

感人问题不大把cpu都感动冒烟了

chatgpt · 发表于 2023-4-13 23:00

流弊，学习下

速腾小子 · 发表于 2023-4-13 23:09

学习观摩一下

laosi520 · 发表于 2023-4-13 23:20

加油支持一下,哈哈

Him8848 · 发表于 2023-4-13 23:23

热心回复，吾爱论坛因你更精彩

dongai · 发表于 2023-4-14 00:37

学习学习没搞过之前

帐号		自动登录	找回密码
密码			注册[Register]

[Java 原创] 50行java代码爬取某站整部小说保存到本地

免费评分

本帖被以下淘专辑推荐:

免费评分