Crawlera Basics
Crawlera is a smart HTTP/HTTPS downloader that routes your requests through a pool of IP addresses, introducing delays and discarding IPs where necessary to...
Thu, 2 Apr, 2020 at 11:14 AM
When using Crawlera, it's important to keep in mind the following best practices: Set download timeout One of the most common problems our users h...
Tue, 19 Jun, 2018 at 8:30 PM
First, go to your Crawlera Dashboard in Scrapinghub and click the 'Create Account' button: Next, under the 'Create New Crawlera Acco...
Fri, 6 Jul, 2018 at 4:54 PM
Crawlera performs HTTPS requests using CONNECT method, transparently tunnelling packets over default HTTPS port 443 to and from the destination server in a ...
Wed, 21 Aug, 2019 at 8:54 AM
Crawlera can process POST requests, and they're considered as a single requests, like GET requests. This is commonly used for crawling search results a...
Wed, 26 Apr, 2017 at 3:55 PM
Crawlera is a proxy rotater which helps in evading bans from the target website by adjusting delays, handling cookies and managing IPs. When Crawlera is add...
Wed, 30 Oct, 2019 at 9:11 AM
You are receiving this error because you are using Crawlera service for requesting content that is larger than 500MB. To improve the stability and performa...
Fri, 1 Feb, 2019 at 6:01 AM
Crawlera will by default use a random desktop-like user agent with every request and discard the User-Agent header that the user sends. This is to further h...
Mon, 10 Feb, 2020 at 5:51 PM
Introducing our new Crawlera Usage Stats dashboard. New and improved interface that provides better visibility and customization for your Crawlera stats mon...
Mon, 9 Mar, 2020 at 12:12 PM
Crawlera headers can be configured in Scrapy per project, per spider and per request. Configuring the Headers Per Project List the headers in the Sc...
Wed, 29 Apr, 2020 at 7:12 PM