There's a limit of 1MB for serialised items in Scrapy Cloud (this includes log and request entries). Data pre-processing/normalising is recommended before storing the items (consider writing a pipeline for item sanitising, placed after all other pipelines, or compressing the data with zlib, for instance).
Modified on: Tue, 13 Feb, 2018 at 3:57 PM
Did you find it helpful?Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.