If seeing the logs it's not enough, the Page Storage Addon could help seeing the responses Scrapy Cloud is getting from the job's crawl.


1 - Go to https://app.scrapinghub.com/p/PROJECT_ID/addons/page_storage , enable it and configure the settings:



Page storage mode:

  • Cache: Items expire after a month
  • Versioned Cache: Multiple copies are retained, and each one expires after a month


2 - Stored pages are found as collections and they are found:

https://app.scrapinghub.com/p/PROJECT_ID/collections/
 





3 - Each stored page could be downloaded as JSON object or viewed from dash. In order to check the html in a browser, the contents of body field should be saved as html in a new file and open in any browser. 


Fields available per stored page as JSON:


body: html code of the page
_encoding: 
cookies:
url: url of the response
_jobid: job id where the response came from