There are times when we want to test spiders with different configurations/environments but have to deploy spiders to different projects and then test the spiders in respective projects. But the recent SHUB release (v2.8.2) has made it possible to test multiple environments in single project. To make use of this feature, you can have different settings.py with required configurations and then while scheduling the spider provide the location of settings that has to be used.
Please follow below steps to deploy project with different settings:
- Ensure that recent version of SHUB is installed. You can upgrade SHUB using "pip install --upgrade shub"
- Then schedule job with a flag "-e SCRAPY_SETTINGS_MODULE=<settings-module-path>", like
shub schedule localinfo -e SCRAPY_SETTINGS_MODULE=testspiders.settings
For any further queries feel free to contact Support.