Crawlera Advanced Topics
With Crawlera, each request is routed through a different outgoing IPs. To make requests using the same IP you need to use Sessions. Each session will inter...
Mon, 30 Jul, 2018 at 5:06 PM
When a request fails for some reason, Crawlera will include an error code in the X-Crawlera-Error header. Please refer to the API documentation to understan...
Wed, 26 Apr, 2017 at 3:58 PM
X-Crawlera-Profile is a header that is meant to replace X-Crawlera-UA. It not only sets the User-Agent but also applies a set of headers which are actually ...
Wed, 4 Jul, 2018 at 8:40 PM
When Crawlera gets a ban from a target website, it automatically retries the request from another proxy IP. By default, Crawlera re-tries 5 times to ret...
Wed, 22 Aug, 2018 at 5:08 PM
Overview Crawling with a headless browser is different from traditional approaches. Conventional spiders give you control over the requests and sequence...
Mon, 18 Mar, 2019 at 3:59 PM
We are changing the behavior of a custom header (X-Crawlera-Profile) that can be sent along with your request to Crawlera. This change is expected to go out...
Fri, 1 Feb, 2019 at 2:50 PM