This addon keeps the content of the
.scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It allows spiders to share data between different runs, keeping a state or any kind of data that needs to be persisted.
.scrapy directory is well known in Scrapy and a few extensions use it to keep a state between runs. The canonical way to work with the
.scrapy directory is by calling the
scrapy.utils.project.data_path function, as illustrated in the following example:
from scrapy.utils.project import data_path mydata_path = data_path() # ... use mydata_path to store or read data which will be persisted among runs ...
DOTSCRAPY_ENABLED- enables or disables DotScrapy addon (either project-wide or per spider)