东方财富股吧爬虫

han5562877 发表于 2023-1-4 14:51

# 东方财富帖子爬取—以上证指数吧为例

## 概述
最近在学习文本情感分析, 找了很多作为案例的论坛贴吧作为爬取对象

本次的对象是东方财富的股吧

!(https://huohuowai-1304849801.cos.ap-shanghai.myqcloud.com//%E5%9B%BE%E5%BA%8A202301041445114.png)

## 代码

```python
import requests
from lxml import etree
from time import sleep
import random
from fake_useragent import UserAgent

ua = UserAgent()

for p in range(4710):
try:
   url = f'http://guba.eastmoney.com/list,zssh000001_{p + 1}.html'
   headers = {'User-Agent': ua.random}
   page = requests.get(url=url, headers=headers).content.decode('utf-8')
   tree = etree.HTML(page)

   for li in tree.xpath('//div[@id="articlelistnew"]/div'):
         try:
            popper = li.xpath('./span[@class="l4 a4"]/a/@data-popper')
            if popper not in ['3006113720930996', '3006113720930996', '7428111481466798', '6712111507146464', '6255325874333310']:
               read = li.xpath('./span[@class="l1 a1"]/text()')
               com = li.xpath('./span[@class="l2 a2"]/text()')
               title = li.xpath('./span[@class="l3 a3"]/a/text()')
               date = li.xpath('./span[@class="l5 a5"]/text()')
               row = f'{read},{com},{title},{date}'
               print(row)
               with open('上证指数吧.csv', 'a', encoding='utf-8-sig') as f:
                     f.write(row + '\n')
         except Exception as e:
            print(str(e))
except Exception as e:
   print(str(e))
finally:
   print(str(p))
   sleep(0.1 + random.random() / 10)
```

## 运行状况

!(https://huohuowai-1304849801.cos.ap-shanghai.myqcloud.com//%E5%9B%BE%E5%BA%8A202301041449578.png)

## 运行结果

!(https://huohuowai-1304849801.cos.ap-shanghai.myqcloud.com//%E5%9B%BE%E5%BA%8A202301041450535.png)

白云云云 发表于 2023-1-4 17:43

nuoan 发表于 2023-1-4 15:43
出现ModuleNotFoundError: No module named 'fake_useragent'，好像用pip install 也装不上fake_useragent ...

刚测试不用ua也没有问题, 4,7,12行删掉, 13行内headers=headers删掉, 可以正常运行

echoxx 发表于 2023-1-4 15:50

nuoan 发表于 2023-1-4 15:43
出现ModuleNotFoundError: No module named 'fake_useragent'，好像用pip install 也装不上fake_useragent ...

可以给错误贴出来看看

qwerdf4567 发表于 2023-1-4 15:12

牛市要来了么:lol

micksoft 发表于 2023-1-4 15:27

从哪里可以抓取到5日线10日线季线年线，这种数据？

nuoan 发表于 2023-1-4 15:43

出现ModuleNotFoundError: No module named 'fake_useragent'，好像用pip install 也装不上fake_useragent，怎么办？

Dumeng 发表于 2023-1-4 16:02

看看小姐姐的家

FIzz001 发表于 2023-1-4 16:19

爬完可以成为股神不

feiyu361 发表于 2023-1-4 16:58

爬完可以成为巴菲特么?

whsstc 发表于 2023-1-4 16:58

要成为股神

xiaoshu1688 发表于 2023-1-4 17:04

癌股哪里有牛市呢

页: [1] 2 3

吾爱破解 - 52pojie.cn's Archiver

东方财富 股吧 爬虫

东方财富股吧爬虫