Start a new topic
Answered

Website redirects

Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?


I am getting this header =>


HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache


and then being redirected to => distil_r_blocked.html


Thanks, Simon


Best Answer

Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.

If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.


Have a nice day!


Pablo


 


Answer

Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.

If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.


Have a nice day!


Pablo


 

So crawlera cannot get around Distil? 

Login to post a comment