Crawlera is a smart HTTP/HTTPS downloader that routes your requests through a pool of IP addresses, introducing delays and discarding IPs where necessary to evade anti-crawling measures. Crawlera is useful if you want to save yourself the hassle of tinkering with download delays and concurrent requests, dealing with cookies, user-agents, referrers to avoid getting banned.

To use Crawlera you need to follow these steps:

  1. subscribe to Crawlera through the Scrapinghub dashboard
  2. get your API key from the Scrapinghub dashboard
  3. use your API key to make a request

Subscribing to Crawlera

To subscribe to Crawlera, follow these steps:

  1. Go to your organization Billing page
  2. select Crawlera
  3. select the plan you would like to subscribe to

Follow the checkout page instructions by entering your billing details and payment information, and click Subscribe:

Getting your API key

Your Crawlera account will now be accessible from your organization home page,

Go to Crawlera → Overview and click the gear icon next to your account username in the left sidebar. You will see instructions there on how to configure Crawlera in your web crawler or HTTP client.

For more information, please refer to the Crawlera official documentation.

Testing your API key

You can test your credentials using curl:

curl -x -U <API key>: