Start a new topic
Answered

Crawlera very slow (Puppeteer & Pages Jaunes)

I use crawlera with Puppeteer for https://www.pagesjaunes.fr

My code is similar to the official example: https://support.scrapinghub.com/support/solutions/articles/22000220800-using-crawlera-with-puppeteer


I have test with multiple possibility. I have this result for load the page:

Crawlera Proxy

 test 1: 90294.232ms 

 test 2: 129361.872ms 

 test 3: 176996.368ms

VPN

 test 1: 12641.482ms 

 test 2: 5216.013ms

Normal

 test 1: 5418.336ms


Have you an idea ? A reason ?


Thanks !


Best Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.


Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.

Thanks for your response Nestor. 

I desactivate the headless mode for all my examples. 


Can you tell me more for "try disabling loading resources through the proxy" ? How do you do this ?


Thanks

Login to post a comment