Crawlera Advanced Topics

Understanding Crawlera Errors
When a request fails for some reason, Crawlera will include an error message in the X-Crawlera-Error header. Please refer to the API documentation to unders...
Mon, 12 Oct, 2020 at 3:41 PM
Desktop and Mobile Profiles
X-Crawlera-Profile is a header that is meant to replace X-Crawlera-UA. It not only sets the User-Agent but also applies a set of headers which are actually ...
Wed, 4 Jul, 2018 at 8:40 PM
How To Deal With Bans ('503' Responses From Crawlera)
When Crawlera gets a ban from a target website, it automatically retries the request from another proxy IP.  By default, Crawlera re-tries 5 times to ret...
Mon, 12 Oct, 2020 at 3:54 PM
How to use crawlera with headless browsers
⚠️  Note: Headless browser support is only available in Advanced and Enterprise plans. Overview Crawling with a headless browser is different from t...
Wed, 13 May, 2020 at 9:08 PM
Changes to the behavior of X-Crawlera-Profile header
We are changing the behavior of a custom header (X-Crawlera-Profile) that can be sent along with your request to Crawlera. This change is expected to go out...
Fri, 1 Feb, 2019 at 2:50 PM
Using Crawlera Sessions to make multiple requests from the same IP
⚠️  Note: Sessions are only available on Crawlera Advanced, C50, C100, C200 and Enterprise plans. With Crawlera, each request is routed through a differ...
Wed, 15 Apr, 2020 at 2:10 PM