Is there any way to create or modify periodic jobs programmatically? I'd like to create a process where I autogenerate spiders and commit them to github, where they are pulled down automatically by ScrapingHub.
Then I'd like to script modifying a periodic job to add new spiders to the job to be run on a periodic basis.
This is a very popular request and we are already working on an API for Periodic Jobs, it should be ready sometime this year. For now, the only option is to do it via the UI.
This is a very popular request and we are already working on an API for Periodic Jobs, it should be ready sometime this year. For now, the only option is to do it via the UI.
a
acowpy
said
1 day ago
Hi, any update on when this feature will be available?
Simon Mosk-Aoyama
Hello,
Is there any way to create or modify periodic jobs programmatically? I'd like to create a process where I autogenerate spiders and commit them to github, where they are pulled down automatically by ScrapingHub.
Then I'd like to script modifying a periodic job to add new spiders to the job to be run on a periodic basis.
Is this possible? The Jobs API only seems to be for one-off jobs (https://doc.scrapinghub.com/api/jobs.html).
thanks!
This is a very popular request and we are already working on an API for Periodic Jobs, it should be ready sometime this year. For now, the only option is to do it via the UI.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
This is a very popular request and we are already working on an API for Periodic Jobs, it should be ready sometime this year. For now, the only option is to do it via the UI.
acowpy
Hi, any update on when this feature will be available?
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 246 topics