We are changing the behavior of a custom header (X-Crawlera-Profile) that can be sent along with your request to Crawlera. This change is expected to go out on Feb 5th 2019 between 12:00-14:00 UTC


Current Behavior


When X-Crawlera-Profile is sent (say, with values desktop or mobile) along with the proxy request, Crawlera sends a default set of headers that match the desktop or mobile user in the outgoing request to the target website. However, when X-Crawlera-Profile is sent along with other HTTP headers such as Accept or Accept-Language, those headers end up overriding Crawlera’s behavior of choosing the best set of browser profile headers. As a result, Crawlera will pass through the other HTTP headers to the target website.


New Behavior


When X-Crawlera-Profile is sent along with other HTTP headers such as Accept or Accept-Language, those headers will not override Crawlera’s behavior of what set of browser profile headers will be sent to the target website. Crawlera will not pass through the other HTTP headers to the target website unless it receives an additional header X-Crawlera-Profile-Pass listing the headers that need to be overridden. 


For example, in the following request, Crawlera will send Accept-Language:de_DE to the outgoing request to the target website because the header X-Crawlera-Profile-Pass is present in the request with the name of the header (Accept-Language) that needs to be overridden. If X-Crawlera-Profile-Pass is not included in the request, then Crawlera will send what it usually sends for the browser profile and ignore the Accept-Language:de_DE header. 


curl -x proxy.crawlera.com:8010 -U $MYAPIKEY: \
-H X-Crawlera-Profile:desktop -H Accept-Language:de_DE \
-H X-Crawlera-Profile-Pass:Accept-Language https://httpbin.org/headers


Please note that multiple header names can be passed for X-Crawlera-Profile-Pass header with comma separated values. 


curl -x proxy.crawlera.com:8010 -U $MYAPIKEY: \
-H X-Crawlera-Profile:desktop -H Accept-Language:de_DE \
-H Accept:text/html -H X-Crawlera-Profile-Pass:Accept-Language,Accept \
https://httpbin.org/headers


Those header names will be overridden only if they are not empty in the request. If you ask Crawlera to override Accept header but do not send any Accept header, then Crawlera will include the default Accept header for the browser profile.