百度网盘资源搜索爬虫源代码开源

gude · 发表于 2016-11-20 17:03

本帖最后由 gude 于 2017-1-10 22:24 编辑

10.1无聊写的百度网盘爬虫
一共爬了1100w数据做的网盘搜索网站 http://www.fastsoso.cn/
源码: https://github.com/gudegg/yunSpider
直接下载编译好的使用

下载适合自己系统的版本
修改config.ini配置文件，并且放到程序运行的目录下
linux: chmod +x spider,然后直接./spider运行或者nohup ./spider 1>log.out 2>err.out &以后台方式运行;win:直接点击spider.exe运行
下载地址：https://github.com/gudegg/yunSpider/releases

说明里面也有java源码

神枪泡泡丶 · 发表于 2016-11-20 17:18

落落问一句爬虫是啥。。。

gude · 发表于 2016-11-23 16:09

yhxi1714 发表于 2016-11-23 15:21
OkhttpUtil能否类提供一下？？

[Java] 纯文本查看 复制代码

public class OkhttpUtil {
    public static final OkHttpClient OK_HTTP_CLIENT = new OkHttpClient();
    public static final MediaType MEDIA_TYPE_JSON = MediaType.parse("application/json; charset=utf-8");
    public static final MediaType MEDIA_TYPE_TEXT = MediaType.parse("text/x-markdown; charset=utf-8");

    static {
        //cookie状态保持
        CookieManager cookieManager = new CookieManager();
        CookieHandler.setDefault(cookieManager);
        cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
        OK_HTTP_CLIENT.setCookieHandler(cookieManager);
    }


    public static ResponseBody syncGet(String url) {
        Request request = new Request.Builder().url(url).build();
        return getResponseBody(request);
    }

    public static ResponseBody syncGet(String url, Headers headers) {
        Request request = null;
        if (headers != null) {
            request = new Request.Builder().url(url).headers(headers).build();
        } else {
            request = new Request.Builder().url(url).build();
        }
        return getResponseBody(request);
    }

    public static ResponseBody syncGet(String url, Headers headers, Map<Object, Object> params) {
        Request.Builder builder = new Request.Builder();
        StringBuilder sb = new StringBuilder(url);
        judgeParams(params, sb);
        if (headers != null) {
            builder = builder.url(sb.toString()).headers(headers);
        } else {
            builder = builder.url(sb.toString());
        }
        Request request = builder.build();
        return getResponseBody(request);
    }


    public static void asynGet(String url, Headers headers, Map<Object, Object> params, Callback callback) {
        Request.Builder builder = new Request.Builder();
        StringBuilder sb = new StringBuilder(url);
        judgeParams(params, sb);
        if (headers != null) {
            builder = builder.url(sb.toString()).headers(headers);
        } else {
            builder = builder.url(sb.toString());
        }
        Request request = builder.build();
        OK_HTTP_CLIENT.newCall(request).enqueue(callback);
    }

    private static void judgeParams(Map<Object, Object> params, StringBuilder sb) {
        if (params != null) {
            int params_size = params.size();
            if (params_size > 0) {
                int index = 0;
                sb.append("?");
                for (Map.Entry entry : params.entrySet()) {
                    index++;
                    if (index != params_size) {
                        sb.append(entry.getKey()).append("=").append(entry.getValue()).append("&");
                    } else {
                        sb.append(entry.getKey()).append("=").append(entry.getValue());
                    }

                }
            }
        }
    }

    private static ResponseBody getResponseBody(Request request) {
        try {
            Response response = OK_HTTP_CLIENT.newCall(request).execute();
            if (response.isSuccessful()) {
                return response.body();
            }
        } catch (IOException e) {
            e.printStackTrace();
            return null;
        }
        return null;
    }

    public static ResponseBody syncPostForm(String url, Headers headers, Map<String, String> params) {
        Request.Builder builder = new Request.Builder();
        if (params != null && params.size() > 0) {
            FormEncodingBuilder formEncodingBuilder = new FormEncodingBuilder();
            for (Map.Entry<String, String> entry : params.entrySet()) {
                formEncodingBuilder.add(entry.getKey(), entry.getValue());
            }
            RequestBody requestBody = formEncodingBuilder.build();
            builder.post(requestBody);
        }
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);
    }

    public static ResponseBody syncPostByJson(String url, Headers headers, String json) {
        Request.Builder builder = new Request.Builder();
        RequestBody requestBody = RequestBody.create(MEDIA_TYPE_JSON, json);
        builder.post(requestBody);
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);
    }

    public static ResponseBody syncPostByString(String url, Headers headers, String str) {
        Request.Builder builder = new Request.Builder();
        RequestBody requestBody = RequestBody.create(MEDIA_TYPE_TEXT, str);
        builder.post(requestBody);
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);

    }

    public static ResponseBody syncPostByStream(String url, Headers headers, InputStream is) throws IOException {
        Request.Builder builder = new Request.Builder();
        RequestBody requestBody = RequestBody.create(MEDIA_TYPE_TEXT, ByteStreams.toByteArray(is));
        builder.post(requestBody);
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);
    }

    public static ResponseBody syncPostByFile(String url, Headers headers, File file) {
        Request.Builder builder = new Request.Builder();
        RequestBody requestBody = RequestBody.create(MEDIA_TYPE_TEXT, file);
        builder.post(requestBody);
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);
    }

    public static ResponseBody syncPost(String url, RequestBody requestBody, Headers headers) {
        Request.Builder builder = new Request.Builder();
        builder.post(requestBody);
        if (headers != null) {
            builder = builder.url(url).headers(headers);
        } else {
            builder = builder.url(url);
        }
        Request request = builder.build();
        return getResponseBody(request);
    }
}

featmellwo · 发表于 2016-11-20 17:19

同问爬虫是啥？有什么用？

adalyb · 发表于 2016-11-20 17:21

哈哈，这个要顶

缥缈的心情 · 发表于 2016-11-20 17:23

JAVA写的么. 电脑里没有环境啊. 只有PYTHON的

gude · 发表于 2016-11-20 17:25

featmellwo 发表于 2016-11-20 17:19
同问爬虫是啥？有什么用？

爬到数据就能做搜索站啊我怕了1100w 做的https://www.zgdgude.cn/

gude · 发表于 2016-11-20 17:26

缥缈的心情发表于 2016-11-20 17:23
JAVA写的么. 电脑里没有环境啊. 只有PYTHON的

java和golang 2个版本

yelidewo · 发表于 2016-11-20 17:27

有图吗?这是...用来做什么的啊...

featmellwo · 发表于 2016-11-20 17:28

gude 发表于 2016-11-20 17:25
爬到数据就能做搜索站啊我怕了1100w 做的https://www.zgdgude.cn/

。。好吧我还是对你网站比较感兴趣23333

15820956473 · 发表于 2016-11-20 17:36

不错。留个v ok？

帐号		自动登录	找回密码
密码			注册[Register]

[其他原创] 百度网盘资源搜索爬虫源代码开源

点评

免费评分

本帖被以下淘专辑推荐:

免费评分