Scrapy Cloud Addons

Here you'll find all about the Addons available on Scrapy Cloud.

Images Storage addon
The Images addon downloads images from extracted image URLs and stores them into an Amazon S3 storage. The addon is enabled by updating the IMAGES_STORE set...
Thu, 1 Feb, 2018 at 12:59 PM
Auto Throttle addon
The Auto Throttle addon makes spiders crawl the target sites with more caution, by dynamically adjusting request concurrency and delay according to the site...
Mon, 27 Mar, 2017 at 1:46 PM
Monitoring addon
⚠ Note that the Monitoring addon is unsupported and may be decommissioned in 2017 The Monitoring addon lets you monitor your spiders, generate reports a...
Wed, 3 May, 2017 at 5:49 PM
Delta Fetch addon
⚠ The Delta Fetch addon is the Scrapinghub dashboard is deprecated and will be removed soon. You can use the same functionality by using the deltafetch libr...
Fri, 9 Mar, 2018 at 3:12 PM
Page Storage addon
If seeing the logs it's not enough, the Page Storage Addon could help seeing the responses Scrapy Cloud is getting from the job's crawl. 1 - G...
Sat, 25 Mar, 2017 at 12:35 AM
Query Cleaner addon
The Query Cleaner addon can be used to clean up the request URL GET query parameters at the output of the spider in accordance with the patterns provided by...
Sat, 25 Mar, 2017 at 1:20 AM
DotScrapy Persistence addon
This addon keeps the content of the .scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It a...
Sat, 25 Mar, 2017 at 12:25 AM
Magic Fields addon
Sometimes, you need to add certain fields to your scraped data that can be derived from the context. For example, you may need a timestamp for when an item ...
Mon, 27 Mar, 2017 at 1:04 PM
Crawlera addon
To enable Crawlera in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the left panel select Addons Setup (under ...
Thu, 22 Mar, 2018 at 7:20 PM