Start a new topic

How to handle 302 redirects?

I read that Crawlera treats a 302 redirect as a successful request, but what if it's actually an anti-spider response from the server? This happened to me when I tried to use the POST method, only to be rebuffed and redirected to an authentication page. Is there a way to manually ask Crawlera to retry with a new IP address when that happens?

2020-06-24 20:37:56 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET> from <POST>
2020-06-24 20:38:00 [scrapy.core.engine] DEBUG: Crawled (401) <GET> (referer: None)


Login to post a comment