Start a new topic

Easy Ways to Run Multiple Spiders?

From what I can tell, you can't just check the box next to all of your spiders and hit a run button (unless I'm missing it). What are some easy ways to run all spiders for a project?

Hitting "Run" and making sure I manually pick each spider in the drop-down list is annoying when you have many spiders. Right now, I just view each spider in a new tab and hit Run on each one. Any better ideas with the dashboard? Or am I going to have to work with the API for this?

1 person has this question

I would like to add that manually choosing spiders from the drop-down list stops working after about 9 spiders or so, so then you have to do it again and make sure you don't pick the same spiders as before. If it kept going until you actually had no spiders remaining, it would be a fair option.

Hello. Any update on this?

The UI uses the API to schedule the jobs, so why not use the API if you need to run multiple spiders?


Alternatively, you can use Periodic Jobs: https://support.scrapinghub.com/support/solutions/articles/22000200419-scheduling-periodic-jobs

Login to post a comment