Start a new topic

Connection pool is full, discarding connection

Hello,

my crawler is downloading images it has found to an AWS S3 bucket.

However, after only 30 seconds of crawling it starts spitting out this warning:

[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>


I have configured it according to this documentation:

https://docs.scrapy.org/en/latest/topics/media-pipeline.html


Are there any settings I can change to increase the connection pool size?

Thanks for any help!
Login to post a comment