Start a new topic
Answered

Passing command line arguments to CrawlerProcess

I have a single Scrapy crawler that I pass arguments to as such. How can I pass all the same command line arguments to this crawler process using the same for example consider the command line command to run a spider "scrapy crawl example -o data.jl -t jsonlines -s JOBDIR=/crawlstate". Now if it want to run two instances of the same spider reading from the same request queue and making an output to the same output file, while resuming from their respective states how should I do it ?

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

# 'followall' is the name of one of the spiders of the project.
process.crawl(#How do I Pass arguments like -o data.jl -t jsonlines -s JOBDIR=/crawlstate here?)
process.start() # the script will block here until the crawling is finished

Best Answer

Hi Aki,


Unfortunately we don't provide coding assistance through our Support channels which is exclusive for platform related issues. Please raise your scrapy questions in scrapy-users mailing list or you can also post to StackOverflow with a 'scrapy' tag.
I hope you understand.

Have a nice day!


Pablo

1 Comment

Answer

Hi Aki,


Unfortunately we don't provide coding assistance through our Support channels which is exclusive for platform related issues. Please raise your scrapy questions in scrapy-users mailing list or you can also post to StackOverflow with a 'scrapy' tag.
I hope you understand.

Have a nice day!


Pablo

Login to post a comment