To enable Crawlera in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the left panel select Addons Setup (under ADDONS):
Then click on Enable on the Crawlera addon:
The addon will appear under your Enabled Addons where you can configure its settings listed below.
|CRAWLERA_URL||proxy URL (default: |
|CRAWLERA_ENABLED||tick the checkbox to enable Crawlera|
|CRAWLERA_APIKEY||Crawlera API key|
|CRAWLERA_MAXBANS||number of consecutive bans (503s) to ignore before closing the spider (default: |
|CRAWLERA_DOWNLOAD_TIMEOUT||timeout for requests (default: |
NOTE #1: To enable Crawlera using the scrapy-crawlera middleware, please see: Using Crawlera with Scrapy.
NOTE #2: If your project is deployed with a custom Docker image, the ADDONS section will not be displayed. In this case Crawlera is enabled in the spider, via the middleware and settings.py file.