批处理之家 - Powered by Discuz! Board

标题: [问题求助] 请教大家一个关于python scrapy的问题 [打印本页]

作者: adneywt 时间: 2024-12-14 20:10 标题: 请教大家一个关于python scrapy的问题

File "C:\python\Lib\site-packages\scrapy_splash\middleware.py", line 19, in <module>
from scrapy_splash.responsetypes import responsetypes
  File "C:\python\Lib\site-packages\scrapy_splash\responsetypes.py", line 41, in <module>
responsetypes = SplashResponseTypes()
  File "C:\python\Lib\site-packages\scrapy\responsetypes.py", line 49, in __init__
self.classes[mimetype] = load_object(cls)
                           ~~~~~~~~~~~^^^^^
  File "C:\python\Lib\site-packages\scrapy\utils\misc.py", line 71, in load_object
mod = import_module(module)
  File "C:\python\Lib\importlib\__init__.py", line 88, in import_module
return _bootstrap._**_import(name[level:], package, level)
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\python\Lib\site-packages\scrapy_splash\response.py", line 11, in <module>
from scrapy_splash.utils import headers_to_scrapy
  File "C:\python\Lib\site-packages\scrapy_splash\utils.py", line 12, in <module>
from scrapy.utils.python import unicode_to_str as to_bytes
ImportError: cannot import name 'unicode_to_str' from 'scrapy.utils.python' (C:\python\Lib\site-packages\scrapy\utils\python.py)

请问这个报错是什么问题呢  麻烦大家帮忙看一下。

作者: Five66 时间: 2024-12-15 00:25

使用时还是安装时报错
使用时报错换个Import顺序或者是否缺了依赖或者各种版本是否一致
安装时报错看看少了依赖或者各种版本是否一致

作者: adneywt 时间: 2024-12-15 19:18

回复 2# Five66

感觉是版本报错，但是我直接改成低版本了这个错误会没有但是又会出现其他的错误。

有没有一种办法可以直接解决所有的错误呢

作者: Five66 时间: 2024-12-15 23:44

不知道你啥环境啥py版本啥scrapy版本又是在啥情况下报错 ,如果是网上整合的或embed就不要自己安装py,其他的py得按照其他的来

建议
全用最新的 ,去网上找找安装步骤 ,一步一步照着来
或者
全手动,自己去下py去下scrapy去下依赖
或者
用linux

作者: adneywt 时间: 2024-12-16 16:00

回复 4# Five66

  你好由于上传不了附件源码请看下面

from typing import Iterable

import scrapy
from scrapy import Request
from scrapy_splash.request import SplashRequest

lua_source = """
function main(splash, args)
  assert(splash:go(args.url))
  assert(splash:wait(2))
  -- 准备一个js函数. 预加载
  -- jsfun是splash预留的专门为了js代码和lua代码结合准备的
  get_btn_display = splash:jsfunc([[
function(){
   return document.getElementsByClassName('load_more_btn')[0].style.display;
   }
]])

  while(true)
  do
splash:runjs("document.getElementsByClassName('load_more_btn')[0].scrollIntoView(true)")
splash:select(".load_more_btn").click()
splash:wait(1)
--判断load_more_btn是否是none
display = get_btn_display()
if(display == 'none')
   then
      break
   end

  end

  return splash:html()  --直接返回页面源代码
end

"""

class WangyiSpider(scrapy.Spider):
name = "wangyi"
allowed_domains = ["163.com"]
start_urls = ["https://news.163.com"]
# 重写start_request
def start_requests(self):
      yield SplashRequest(
         url=self.start_urls[0],
         callback=self.parse,
         endpoint="execute", # 终端表示你要执行哪一个splash服务
         args={
            "lua_source": lua_source
         }
      )

def parse(self, response):
      print(response.text)

py版本3.13.0
scrapy版本 2.12.0
都是最新版本，他这个源码是不是需要旧版本。

作者: Five66 时间: 2024-12-16 21:39

回复 5# adneywt

按 1 楼的报错来看 ,应该是
from scrapy_splash.request import SplashRequest
这句的问题
大概是 scrapy_splash 用的是支持 py2.7 的 scrapy
试试更新一下 scrapy_splash 这个包包

作者: adneywt 时间: 5 天前 20:27

回复 6# Five66

pip install --upgrade scrapy_splash

更新后
Requirement already satisfied: scrapy_splash in c:\python\lib\site-packages (0.9.0)

然后运行程序还是报错下面

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\python\Scripts\scrapy.exe\__main__.py", line 7, in <module>
sys.exit(execute())
         ~~~~~~~^^
  File "C:\python\Lib\site-packages\scrapy\cmdline.py", line 187, in execute
cmd.crawler_process = CrawlerProcess(settings)
                        ~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\python\Lib\site-packages\scrapy\crawler.py", line 424, in __init__
super().__init__(settings)
~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\python\Lib\site-packages\scrapy\crawler.py", line 295, in __init__
self.spider_loader: SpiderLoader = self._get_spider_loader(settings)
                                    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\python\Lib\site-packages\scrapy\crawler.py", line 289, in _get_spider_loader
return cast("SpiderLoader", loader_cls.from_settings(settings.frozencopy()))
                              ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\python\Lib\site-packages\scrapy\spiderloader.py", line 81, in from_settings
return cls(settings)
  File "C:\python\Lib\site-packages\scrapy\spiderloader.py", line 36, in __init__
self._load_all_spiders()
~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\python\Lib\site-packages\scrapy\spiderloader.py", line 65, in _load_all_spiders
for module in walk_modules(name):
               ~~~~~~~~~~~~^^^^^^
  File "C:\python\Lib\site-packages\scrapy\utils\misc.py", line 98, in walk_modules
submod = import_module(fullpath)
  File "C:\python\Lib\importlib\__init__.py", line 88, in import_module
return _bootstrap._**_import(name[level:], package, level)
         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _**_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\pythonbase\news\news\spiders\wangyi.py", line 5, in <module>
from scrapy_splash.request import SplashRequest
  File "C:\python\Lib\site-packages\scrapy_splash\__init__.py", line 10, in <module>
from .dupefilter import SplashAwareDupeFilter, splash_request_fingerprint
  File "C:\python\Lib\site-packages\scrapy_splash\dupefilter.py", line 12, in <module>
from scrapy.utils.request import request_fingerprint
ImportError: cannot import name 'request_fingerprint' from 'scrapy.utils.request' (C:\python\Lib\site-packages\scrapy\utils\request.py)

作者: Five66 时间: 4 天前 00:15

回复 7# adneywt

Scrapy 2.12貌似移除了request_fingerprint
而scrapy_splash 0.9还在用request_fingerprint ,大概还没支持Scrapy 2.12

试试换回旧版的Scrapy
或者
等scrapy_splash支持
或者试试
配置里设置并定义一个过滤器
注意还得运行splash服务

欢迎光临批处理之家 (http://bathome.net./)