Start a new topic

Error due to "Rejected message because it was too big: ITM"

Dear Support Team,


I encountered this error when using Scrapy Cloud to run my job:

Rejected message because it was too big: ITM {...


I am sure that the request is fired successfully. This error should be caused by large response, which is about 2M. But this is the size I need to store.


I followed the advices in the following link, i.e. enabling Page Storage addon, and configured a Amazon S3 storage using Feed Export:

https://support.scrapinghub.com/support/discussions/topics/22000009523


 However, the same error is throw. May i know how to solve this issue?

1 Comment

I have the same issue. My scraper produces just one result item, which is a 100MB JSON tree representing a website's data. This works fine locally for me.


What limits are enforced? Is there any way to use Scrapy Cloud with an item like this?

Login to post a comment