Scrapy Cloud Addons

Here you'll find all about the Addons available on Scrapy Cloud.

Images Storage addon
The Images addon downloads images from extracted image URLs and stores them into an Amazon S3 storage. The addon is enabled by updating the IMAGES_STORE set...
Thu, 1 Feb, 2018 at 12:59 PM
Auto Throttle addon
The Auto Throttle addon makes spiders crawl the target sites with more caution, by dynamically adjusting request concurrency and delay according to the site...
Mon, 27 Mar, 2017 at 1:46 PM
Monitoring addon
⚠ Note that the Monitoring addon is unsupported and may be decommissioned in 2017 The Monitoring addon lets you monitor your spiders, generate reports a...
Wed, 3 May, 2017 at 5:49 PM
Delta Fetch addon
⚠ Important: You’ll need to enable the DotScrapy Persistence add on for DeltaFetch to work. The purpose of this addon is to ignore requests to pages c...
Wed, 5 Apr, 2017 at 11:48 PM
Page Storage addon
If seeing the logs it's not enough, the Page Storage Addon could help seeing the responses Scrapy Cloud is getting from the job's crawl. 1 - G...
Sat, 25 Mar, 2017 at 12:35 AM
Query Cleaner addon
The Query Cleaner addon can be used to clean up the request URL GET query parameters at the output of the spider in accordance with the patterns provided by...
Sat, 25 Mar, 2017 at 1:20 AM
DotScrapy Persistence addon
This addon keeps the content of the .scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It a...
Sat, 25 Mar, 2017 at 12:25 AM
Magic Fields addon
Sometimes, you need to add certain fields to your scraped data that can be derived from the context. For example, you may need a timestamp for when an item ...
Mon, 27 Mar, 2017 at 1:04 PM
Crawlera addon
To employ Crawlera in Scrapy Cloud projects the Crawlera addon is used. Go to Settings > Addons > Crawlera to activate. Settings CRAWLERA_URL...
Thu, 9 Nov, 2017 at 3:40 PM