0
Answered
Nayak 1 week ago in Crawlera • updated by Pablo Vaz (Support Engineer) 6 days ago 1

Hi,


We want to make webrequest for one of popular domain continuesly, we created background service and requesting domain using .Net HttpWebRequest.
We got the problem of IP Ban. I just saw the Crawlera.
I just saw the post How to use Crawlera in C# .Net
var myProxy = new WebProxy("http://proxy.crawlera.com:8010");
myProxy.Credentials = new NetworkCredential("<API KEY>", "");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("some domain url");
request.Proxy = myProxy;
request.PreAuthenticate = true;

If i make a webrequest for any website repeatedly, if IP ban occurs, will crawlera automatically handle this request with different IP? or do we need to send same request once again?


If it handles automatically then Do we need to send any header information apart from above listed code.

Regards,

Nayak


Answer

Answer
Answered

Hi Nayak, yes Crawlera will automatically retry. After 5 times (by default) if the site still banning IPs Crawlera will give you a ban status and will try with another request.

Even though Crawlera should protect you against bans, sometimes it runs out of capacity and will return a 503 response. Because of this, we recommend you retry 503 responses up to 5 times. Consider using the x-crawlera-next-request-in to retry more efficiently.

Kind regards,

Pablo

Answer
Answered

Hi Nayak, yes Crawlera will automatically retry. After 5 times (by default) if the site still banning IPs Crawlera will give you a ban status and will try with another request.

Even though Crawlera should protect you against bans, sometimes it runs out of capacity and will return a 503 response. Because of this, we recommend you retry 503 responses up to 5 times. Consider using the x-crawlera-next-request-in to retry more efficiently.

Kind regards,

Pablo