Scrapy - 使用 TwistedScheduler 时出现 ReactorAlreadyInstalledError

我有以下 Python 代码来启动 APScheduler/TwistedScheduler cronjob 来启动蜘蛛。

使用一只蜘蛛不是问题，而且效果很好。然而，使用两个蜘蛛会导致错误：twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed.

我确实找到了一个相关问题 https://stackoverflow.com/questions/71548957/twisted-internet-error-reactoralreadyinstallederror-reactor-already-installed, using CrawlerRunner作为解决方案。但是，我正在使用 TwistedScheduler 对象，所以我不知道如何使用多个 cron 作业（多个add_job()).

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from apscheduler.schedulers.twisted import TwistedScheduler

from myprojectscraper.spiders.my_homepage_spider import MyHomepageSpider
from myprojectscraper.spiders.my_spider import MySpider

process = CrawlerProcess(get_project_settings())
# Start the crawler in a scheduler
scheduler = TwistedScheduler(timezone="Europe/Amsterdam")
# Use cron job; runs the 'homepage' spider every 4 hours (eg. 12:10, 16:10, 20:10, etc.)
scheduler.add_job(process.crawl, 'cron', args=[MyHomepageSpider], hour='*/4', minute=10)
# Use cron job; runs the full spider every week on the monday, tuesday and saturday at 4:35 midnight
scheduler.add_job(process.crawl, 'cron', args=[MySpider], day_of_week='mon,thu,sat', hour=4, minute=35)
scheduler.start()
process.start(False)

https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

还有另一个 Scrapy 实用程序可以对爬行过程提供更多控制：scrapy.crawler.CrawlerRunner。这个类是一个薄包装器，封装了一些简单的帮助器来运行多个爬虫，但它不会以任何方式启动或干扰现有的反应器。
如果您的应用程序已经在使用 Twisted 并且您想在同一个反应器中运行 Scrapy，建议您使用 CrawlerRunner 而不是 CrawlerProcess。

https://docs.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process https://docs.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process

默认情况下，当您运行 scrapycrapy 时，Scrapy 为每个进程运行一个蜘蛛。然而，Scrapy 支持使用内部 API 在每个进程运行多个蜘蛛。

from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.utils.log import configure_logging
from twisted.internet import reactor
from apscheduler.schedulers.twisted import TwistedScheduler

from myprojectscraper.spiders.my_homepage_spider import MyHomepageSpider
from myprojectscraper.spiders.my_spider import MySpider

configure_logging()

runner = CrawlerRunner(get_project_settings())
scheduler = TwistedScheduler(timezone="Europe/Amsterdam")
# Use cron job; runs the 'homepage' spider every 4 hours (eg. 12:10, 16:10, 20:10, etc.)
scheduler.add_job(runner.crawl, 'cron', args=[MyHomepageSpider], hour='*/4', minute=10)
# Use cron job; runs the full spider every week on the monday, tuesday and saturday at 4:35 midnight
scheduler.add_job(runner.crawl, 'cron', args=[MySpider], day_of_week='mon,thu,sat', hour=4, minute=35)

deferred = runner.join()
deferred.addBoth(lambda _: reactor.stop())

scheduler.start()
reactor.run()  # the script will block here until all crawling jobs are finished
scheduler.shutdown()