Scrapy Cloud FAQ

Which services are available on Scrapy Cloud?
Note: Portia is no longer available for new users. It has been disabled for all the new organisations from August 20, 2018 onward. Here is a diagram tha...
Thu, 18 Oct, 2018 at 11:38 AM
What is a Scrapy Cloud Unit?
Scrapy Cloud jobs run in containers. These containers can be of different sizes defined by Scrapy Cloud units. A Scrapy Cloud provides: 1 GB of RAM 2....
Wed, 26 Jul, 2017 at 3:38 PM
Why do I see only one Scrapy Cloud unit in Dashboard after purchasing an extra unit?
When you subscribe to Scrapinghub, 1 Scrapy Cloud unit is given for free. This free unit has features such as data retention for 7 days and a run time limit...
Tue, 19 Sep, 2017 at 5:40 AM
Can I use an HTTP cache on Scrapy Cloud?
Yes, you can. To do that, you have to enable Scrapy's HTTP cache extension by setting HTTPCACHE_ENABLED to True in your project settings. Th...
Fri, 2 Jun, 2017 at 11:30 PM
What are the IP addresses used by Scrapy Cloud?
Scenario: You have e.g. a MongoDB server that you would like your spiders to write to, and you would like to open access to that server only from Scrapy Clo...
Wed, 26 Apr, 2017 at 1:35 PM
Can I run Scrapy Cloud on my environment?
Scrapy Cloud is only offered as a hosted service. If you're interested in hosting your own version of Scrapy Cloud, get in touch and let us know.
Wed, 26 Apr, 2017 at 1:29 PM
What are the differences between running a spider locally and on Scrapy Cloud?
Here are a few things that work differently on Scrapy Cloud compared to a default Scrapy configuration. All these can be overridden of course:  AutoThro...
Wed, 26 Apr, 2017 at 1:34 PM
Where can I find my activation (or re-activation) link?
A common issue is not to receive your activation link when you subscribe to a new account in Scrapinghub or when you forgot your password and trying to send...
Wed, 26 Apr, 2017 at 1:37 PM
How can I migrate my Python eggs to Scrapy Cloud 2.0?
Scrapinghub will support eggs within the Scrapinghub Dashboard through the end of 2016. After that, project dependencies outside of the Scrapy Cloud stack m...
Wed, 26 Apr, 2017 at 1:33 PM
Scrapy Cloud vs Scrapyd (using Heroku)
Here at Scrapinghub we are big fans of Heroku. When people ask what Scrapy Cloud is about we sometimes tell people that "it's like Heroku, but for...
Tue, 7 Nov, 2017 at 9:35 PM