0
Answered
dyv 3 months ago in Scrapy Cloud • updated by Pablo Vaz (Support Engineer) 3 weeks ago 1

My spiders return inconsistent results. I am working on the website, https://www.ifixit.com/. How can I have same number of items every time I run the spider?

Answer

Answer
Answered

Hi dyv, 


Unfortunately this is not possible to guarantee. Even with the best plans offered and the most strictly monitoring, some items got missing due other parameters beyond our control. Mainly due the backend issues or changes from the target domain side.


Many of our clients consider using DotScrapy persistence add-on to have a more robust control on the items extracted.


Best regards,


Pablo

Answer
Answered

Hi dyv, 


Unfortunately this is not possible to guarantee. Even with the best plans offered and the most strictly monitoring, some items got missing due other parameters beyond our control. Mainly due the backend issues or changes from the target domain side.


Many of our clients consider using DotScrapy persistence add-on to have a more robust control on the items extracted.


Best regards,


Pablo