0
Answered
Алексей 1 month ago in Scrapy Cloud • updated by Pablo Vaz (Support Engineer) 1 month ago 3

I'm used these spider configuration:

CONCURRENT_REQUESTS = 1

DOWNLOAD_TIMEOUT = 300

AUTOTHROTTLE_MAX_DELAY = 60

AUTOTHROTTLE_ENABLED = false

CONCURRENT_REQUESTS_PER_DOMAIN = 1

AUTOTHROTTLE_START_DELAY = 20

And get result, for 1 minute scraping:


downloader/response_status_count/20016
downloader/response_status_count/30111
downloader/response_status_count/503663

How to slow down scraping? I need about 1 page for 2-5 second

Answer

Answer
Answered

Hey Алексей!

You can set for instance: DOWNLOAD_DELAY = 5 (Which is 5 secs of delay).

Have you considered to use Crawlera? It can improve our crawling methods in order to give you more concurrent requests.

Let me know if you need further help.

Regards,

Pablo

GOOD, I'M SATISFIED
Satisfaction mark by Алексей 1 month ago
Answer
Answered

Hey Алексей!

You can set for instance: DOWNLOAD_DELAY = 5 (Which is 5 secs of delay).

Have you considered to use Crawlera? It can improve our crawling methods in order to give you more concurrent requests.

Let me know if you need further help.

Regards,

Pablo

Thanks, seems it worked!

I would gladly took advantage of Crawlera, but I need so many requests... it is too expensive for me :(

You are welcome Алексей!

Best regards and thanks for your quick feedback!