The Jobs dashboard can be used to monitor and control your spiders.

To schedule a spider for running select the spider in the RunĀ box and click the Schedule button.

Jobs table

If you click on any project, you go to the Jobs table where you will see the spider jobs that are completed, running or pending to run.

  • Pending Jobs - the list of jobs that are waiting to start. Jobs may not start immediately after being scheduled, sometimes they must wait for resources to be available. If you need better capacity guarantees, consider purchasing a dedicated server.
  • Running Jobs - the list of jobs that are currently running.
  • Completed Jobs - the latest completed jobs.

Job Page

Clicking on any job ID (see the picture above) will take you to the job page, where you can see stats about the job alongside items scraped from it: