怎么把遍历出的数据清洗一下保存在表格里面好看一些，还想求助一下怎么爬多页
本帖最后由 c672569644 于 2022-12-10 00:13 编辑import requests
import csv
# url = 'https://stock.xueqiu.com/v5/stock/screener/fund/list.json?type=18&parent_type=1&order=desc&order_by=percent&page=1&size=30'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.62',
'Cookie': 's=ci12owalds;device_id=bf170c1a5ad2e8f024c7daaa8ea85226; '
'Hm_lvt_1db88642e346389874251b5a1eded6e3=1664207247,1664350847; xq_a_token=df4b782b118f7f9cabab6989b39a24cb04685f95; xqat=df4b782b118f7f9cabab6989b39a24cb04685f95; xq_r_token=3ae1ada2a33de0f698daa53fb4e1b61edf335952; xq_id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJ1aWQiOi0xLCJpc3MiOiJ1YyIsImV4cCI6MTY3MjE4Njc1MSwiY3RtIjoxNjY5ODkyMDk3MDA1LCJjaWQiOiJkOWQwbjRBWnVwIn0.R7xVHyE3IoYwNn_YuJIIsuApsxdEf0e-cFwXSzaJRaMWLoHVBua77D5y3SBKbd7EUEZx7BbSn_Ip9JgIO2F66vVWxvMEc1hO6IyAIy-Sz3KXHF4FOeJsAevLtXRV2JW2MfQZW2KaPXpNJSFpy7t15ER-1K4jI9wd9kpYPsl8c3du3m4pSp7TKd-fhMwXFYseOIlASUIg-Mp-zdzUDbPIjm6vV9enbnK_30Cg-jnsXFVb3QUnijVYVAVRuX5kLFQbXKpMUnW4KorKVPf0TNZgM7Hx0UshevE0n3tWLsBErV_W3NJ_lc6NYtTQraxSmBUFaVcahBi1xbilHPwjsBkh9w; u=691669892154166',
'Origin': 'https://xueqiu.com',
'Referer': 'https://xueqiu.com/hq', }
url = 'https://stock.xueqiu.com/v5/stock/screener/fund/list.json?type=18&parent_type=1&order=desc&order_by=percent&page=1&size=30'
response = requests.get(url=url, headers=headers)
json = response.json()
data1 = json['data']['list']
for i in data1:
print()
f = open('123.csv', mode='a', encoding='utf-8', newline='')
csv_writer = csv.writer(f)
csv_writer.writerow()
import requests
import openpyxl
MAX_PAGE = 3
IS_SETTING_TITLE = False
# url = 'https://stock.xueqiu.com/v5/stock/screener/fund/list.json?type=18&parent_type=1&order=desc&order_by=percent&page=1&size=30'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.62',
'Cookie': 's=ci12owalds;device_id=bf170c1a5ad2e8f024c7daaa8ea85226; '
'Hm_lvt_1db88642e346389874251b5a1eded6e3=1664207247,1664350847; xq_a_token=df4b782b118f7f9cabab6989b39a24cb04685f95; xqat=df4b782b118f7f9cabab6989b39a24cb04685f95; xq_r_token=3ae1ada2a33de0f698daa53fb4e1b61edf335952; xq_id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJ1aWQiOi0xLCJpc3MiOiJ1YyIsImV4cCI6MTY3MjE4Njc1MSwiY3RtIjoxNjY5ODkyMDk3MDA1LCJjaWQiOiJkOWQwbjRBWnVwIn0.R7xVHyE3IoYwNn_YuJIIsuApsxdEf0e-cFwXSzaJRaMWLoHVBua77D5y3SBKbd7EUEZx7BbSn_Ip9JgIO2F66vVWxvMEc1hO6IyAIy-Sz3KXHF4FOeJsAevLtXRV2JW2MfQZW2KaPXpNJSFpy7t15ER-1K4jI9wd9kpYPsl8c3du3m4pSp7TKd-fhMwXFYseOIlASUIg-Mp-zdzUDbPIjm6vV9enbnK_30Cg-jnsXFVb3QUnijVYVAVRuX5kLFQbXKpMUnW4KorKVPf0TNZgM7Hx0UshevE0n3tWLsBErV_W3NJ_lc6NYtTQraxSmBUFaVcahBi1xbilHPwjsBkh9w; u=691669892154166',
'Origin': 'https://xueqiu.com',
'Referer': 'https://xueqiu.com/hq', }
wb = openpyxl.Workbook()
wk = wb.active
for page in range(1, MAX_PAGE + 1):
url = f'https://stock.xueqiu.com/v5/stock/screener/fund/list.json?type=18&parent_type=1&order=desc&order_by=percent&page={page}&size=30'
response = requests.get(url=url, headers=headers)
json_data = response.json()
data = json_data['data']['list']
if not IS_SETTING_TITLE:
wk.append(list(data.keys()))
IS_SETTING_TITLE = True
for v in data:
wk.append()
wb.save("stock.xlsx")
观察每页url 差异，向不同页的url发请求可以实现抓取多页
观察响应结果页面结构，用正则或其他方式解析可以实现精准获取数据
看了也不明，了解中 你想要在表格里好看些 那就只能是json格式的数据了 请问直接把url贴进浏览器地址里，打不开网页的呢？
https://stock.xueqiu.com/v5/stock/screener/fund/list.json?type=18&parent_type=1&order=desc&order_by=percent&page=1&size=30
O2H2O 发表于 2022-12-6 09:22
请问直接把url贴进浏览器地址里，打不开网页的呢？
https://stock.xueqiu.com/v5/stock/screener/fund/lis ...
这个我也不懂你自己开个帖子问问 雨陌 发表于 2022-12-6 00:30
import requests
import openpyxl
感谢大佬 O2H2O 发表于 2022-12-6 09:22
请问直接把url贴进浏览器地址里，打不开网页的呢？
https://stock.xueqiu.com/v5/stock/screener/fund/lis ...
要带上cookie，因为这个是一个内部的接口。 雨陌 发表于 2022-12-6 20:39
要带上cookie，因为这个是一个内部的接口。
好的，谢谢了{:1_893:}我还要多学习~
页:
[1]