Crawlera Basics

Getting started with Crawlera
Crawlera is a smart HTTP/HTTPS downloader that routes your requests through a pool of IP addresses, introducing delays and discarding IPs where necessary to...
Wed, 29 Mar, 2017 at 5:53 PM
Crawlera best practices
When using Crawlera, it's important to keep in mind the following best practices: Set download timeout One of the most common problems our users h...
Thu, 21 Dec, 2017 at 12:04 PM
Restricting Crawlera IPs to a specific region
First, go to your Crawlera Dashboard in Scrapinghub and click the 'Create Account' button: Next, under the 'Create New Crawlera Accou...
Thu, 2 Nov, 2017 at 2:08 PM
Fetching HTTPS pages with Crawlera
To fetch HTTPS web pages through Crawlera you need to download and install the certificate file for Crawlera Certificate Authority or disable SSL certificat...
Wed, 26 Apr, 2017 at 3:52 PM
Sending POST requests with Crawlera
Crawlera can process POST requests, and they're considered as a single requests, like GET requests. This is commonly used for crawling search results a...
Wed, 26 Apr, 2017 at 3:55 PM
Customizing Crawlera User Agents
Crawlera will by default use a random user-agent with every request and discard the User-Agent header that the user sends. This is to further help prevent y...
Wed, 26 Apr, 2017 at 3:30 PM