I'm using crawlera's c50 to crawl a bunch of websites and most of them are fine with it, except there are some websites which return captcha in response with 200 response code. In simple words the websites know that this is not a user but a crawler scraping data.
These websites are using fun captcha.
I am facing this issue with the following websites:
How am I supposed to bypass this situation?
Crawlera should be able to handle this on its own right?
How the websites are detecting that I am using a crawler to scrap data when I have different proxy on each request and with a valid user agent from crawlera?
As far as I know that websites can detect bots based on: