This addon keeps the content of the .scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It allows spiders to share data between different runs, keeping a state or any kind of data that needs to be persisted.

The .scrapy directory is well known in Scrapy and a few extensions use it to keep a state between runs. The canonical way to work with the .scrapy directory is by calling the scrapy.utils.project.data_path function, as illustrated in the following example:

from scrapy.utils.project import data_path

mydata_path = data_path()

# ... use mydata_path to store or read data which will be persisted among runs ...

Supported settings

  • DOTSCRAPY_ENABLED  - enables or disables DotScrapy addon (either project-wide or per spider)