Start a new topic

Error 404 when using Session ID

Hi, 


I got 404 error for my second request when i use session id.


I am creating Session Id with my first request

headers={'X-Crawlera-Session': 'create'},


getting session id in response

session_id = response.headers.get('X-Crawlera-Session', '')


and sending second request

scrapy.FormRequest.from_response(

 response, 

 headers={'X-Crawlera-Session': session_id}, 

 formdata={...}, 

 callback=self.parseResult)


When crawlera is disabled, everything works fine


I see it's POST request. Try adding the header "X-Crawlera-Cookies: disable". If that doesn't work, I would suggest you open a ticket (Help > Contact Support) and provide the data you are sending on the request.

I got --> [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404...>: HTTP status code is not handled or not allowed


Everything works fine when crawlera is disabled

A 404 typically means URL not found, are you sure that's the error code you're getting?

Login to post a comment