Scrapy Cloud Advanced Topics

Here you'll find articles on advanced settings and features of Scrapy Cloud.

Syncing your .scrapy folder to an S3 bucket using DotScrapy Persistence
NOT TO BE CONFUSED WITH THE DOTSCRAPY PERSISTENCE ADDON The purpose of this guide is to keep the content of the .scrapy directory in a persistent store,...
Fri, 4 Jan, 2019 at 8:31 PM
Machine learning with Scrapy and MonkeyLearn
NOT TO BE CONFUSED WITH THE MONKEYLEARN ADDON The purpose of this guide is to bring machine learning technology to the data that you extract through Scr...
Thu, 14 Jun, 2018 at 10:21 AM
Errors while deploying Custom Image to Scrapy Cloud.
While deploying custom Docker images to Scrapy Cloud there're some known issues. We are actively working on getting it resolved, but until it's co...
Mon, 15 Oct, 2018 at 3:49 AM
Deploy Project and run Spiders with settings of different environments.
There are times when we want to test spiders with different configurations/environments but have to deploy spiders to different projects and then test th...
Thu, 24 Jan, 2019 at 8:57 AM