Crawlera will by default use a random desktop-like user agent with every request and discard the User-Agent header that the user sends. This is to further help prevent your crawler from being blocked.


With the X-Crawlera-Profile header, you can choose a browser-based (X-Crawlera-Profile: desktop) or mobile-based user agent (X-Crawlera-Profile: mobile). To send your own user agent, use X-Crawlera-Profile: pass.


Warning

Passing your own user agent is allowed but strongly discouraged, unless the goal is to alter the content being delivered by the websites (not circumventing bans). Please also note that this header is only available for users on C50 plan and above (including Enterprise).