Start a new topic
Answered

Crawlera 503 Ban

Hi everyone,


I'm using a C10 plan in order to crawl a french website.


But every request are in 503, with those headers :


  

HTTP/1.1 200 OK

HTTP/1.1 503 Service Unavailable
Connection: close
Date: Sun, 25 Jun 2017 01:19:48 GMT
Proxy-Connection: close
Transfer-Encoding: chunked
X-Crawlera-Error: banned
X-Crawlera-Slave: 107.173.143.85:3128
X-Crawlera-Version: 1.30.16-c3ff4a

Website crawl ban

  


What does exactly means ?

Will Crawlera be permanently banned from this domain ?


What can I do ?


Thank you.



Best Answer

Hello,


Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.


And from your logs, I can see that you have made successful responses as well.

1 Comment

Answer

Hello,


Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.


And from your logs, I can see that you have made successful responses as well.

Login to post a comment