Scrapy Cloud Advanced Topics

Here you'll find articles on advanced settings and features of Scrapy Cloud.

Downloading and processing images
NOT TO BE CONFUSED WITH THE IMAGES ADDON Scrapy provides reusable item pipelines for downloading images attached to a particular item (for example, when...
Mon, 30 Apr, 2018 at 11:12 PM
Syncing your .scrapy folder to an S3 bucket using DotScrapy Persistence
NOT TO BE CONFUSED WITH THE DOTSCRAPY PERSISTENCE ADDON The purpose of this guide is to keep the content of the .scrapy directory in a persistent store,...
Fri, 4 Jan, 2019 at 8:31 PM
Machine learning with Scrapy and MonkeyLearn
NOT TO BE CONFUSED WITH THE MONKEYLEARN ADDON The purpose of this guide is to bring machine learning technology to the data that you extract through Scr...
Thu, 14 Jun, 2018 at 10:21 AM
Deploy Project and run Spiders with settings of different environments
There are times when we want to test spiders with different configurations/environments but have to deploy spiders to different projects and then test th...
Thu, 18 Apr, 2019 at 6:34 PM
Deploying Custom Docker Image with Selenium on Scrapy Cloud
See shub documentation for the custom Docker image deployment instructions. Chrome Dockerfile Example for Chrome FROM scrapinghub/scrapinghub-st...
Wed, 20 May, 2020 at 6:02 AM