Start a new topic
Answered

How is it using crawlera for a website which is secured by Cloudflare?

Hello there,


I have been scraping a website with Cloudflare security system. I would like to subscribe to a plan here for Crawlera but I also saw some bad comments about Crawlera.


Since I am a student I do not want to waste my money which is not so much.

So is there anyone who had experience scraping a well secured website with Crawlera, especially protected with Cloudflare and having captchas?


Cheers!


Best Answer

Cloudfare employs 2-3 different flavors of bot protection. Some of them can be addressed by using the `cfscrape` library - https://github.com/Anorov/cloudflare-scrape

Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.

1 Comment

Answer

Cloudfare employs 2-3 different flavors of bot protection. Some of them can be addressed by using the `cfscrape` library - https://github.com/Anorov/cloudflare-scrape

Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.

Login to post a comment