To avoid infinite crawling jobs, Portia spiders are finished when they scrape less than 200 items in the last 3600 seconds. You can change those values via the following spider settings:


* SLYCLOSE_SPIDER_CHECK_PERIOD: the period of time (defaults to 3600 seconds)

* SLYCLOSE_SPIDER_PERIOD_ITEMS: the minimimum amount of items that the spider must have scraped in the last SLYCLOSE_SPIDER_CHECK_PERIOD to keep running (defaults to 200)



To set the SLYCLOSE_SPIDER_CHECK_PERIOD and SLYCLOSE_SPIDER_PERIOD_ITEMS you would need to navigate to the Job settings of the spider, and then set the values under the Scrapy raw settings as shown in example below.



And then click on SAVE. After save, the settings would appear under scrapy settings as well.



The settings will be overridden for the spider. Now when the spider will run, it will consider these settings as well.