Crawlera will by default use a random user-agent with every request and discard the User-Agent header that the user sends. This is to further help prevent your crawler from being blocked.

With the X-Crawlera-UA header, you can choose a browser-based or mobile-based User-Agent.


Passing your own user agent is allowed but strongly discouraged for users on C50 plan and above (including Enterprise), unless the goal is to alter the content being delivered by the websites (not circumventing bans).