Crawlera headers can be configured in Scrapy per project, per spider and per request.


Configuring the Headers Per Project


List the headers in the Scrapy project's settings.py file:


DEFAULT_REQUEST_HEADERS = {
    "X-Crawlera-Profile": "desktop",
    "X-Crawlera-Cookies": "disable",
}


Configuring the Headers Per Spider


If the project contains several spiders, but the headers should only be used by a select one, make use of custom_settings variable in the spider class file:


class SomeSpider(scrapy.Spider):
    name = 'somename'
    custom_settings = {
        "DEFAULT_REQUEST_HEADERS": {
            "X-Crawlera-Profile": "desktop",
            "X-Crawlera-Cookies": "disable",
        }
    }


Configuring the Headers Per Request


If the headers should only be used by a particular request in a particular method of the spider class (for instance, in start_requests, but not in parse_results), list the headers in the headers parameter of the scrapy.Request object:


def start_requests(self):
    headers = {
        "X-Crawlera-Profile": "desktop",
        "X-Crawlera-Cookies": "disable",
    }
    for start_url in self.start_urls:
        yield scrapy.Request(
            url=start_url,
            callback=self.parse_results,
            headers=headers)