Crawlera is a smart HTTP/HTTPS downloader that routes your requests through a pool of IP addresses, introducing delays and discarding IPs where necessary to evade anti-crawling measures. Crawlera is useful if you want to save yourself the hassle of tinkering with download delays and concurrent requests, dealing with cookies, user-agents, referrers to avoid getting banned.

To use Crawlera you need to follow these steps:

  1. subscribe to Crawlera through the Scrapinghub dashboard
  2. get your API key from the Scrapinghub dashboard
  3. use your API key to make a request

Subscribing to Crawlera

To subscribe to Crawlera, follow these steps:

  1. Go to your organization Billing page
  2. select Crawlera
  3. select the plan you would like to subscribe to

Scroll down to the bottom of the page, review the order and amount due and Click on Update Subscriptions. 

Please enter the Billing details (Billing address and payment details) to set up the payment method.

Getting your API key

Your Crawlera account will now be accessible from your organization home page,

Go to Crawlera → Usage Stats and click on Setup in the left sidebar. You will see instructions there on how to configure Crawlera in your web crawler or HTTP client.

For more information, please refer to the Crawlera official documentation.

Testing your API key

You can test your credentials using curl:

curl -x -U <API key>: