Crawlera will by default use a random desktop-like user agent with every request and discard the User-Agent header that the user sends. This is to further help prevent your crawler from being blocked.


With the X-Crawlera-Profile header, you can choose a browser-based (X-Crawlera-Profile: desktop) or mobile-based user agent (X-Crawlera-Profile: mobile).


To send your own user agent, use X-Crawlera-Profile: pass. Passing your own user agent is only available for users on the Enterprise plan.