Start a new topic
Answered

Web hook when sppider/job completed

Is there any way to define a web hook to be called when a spider/job is completed?


Best Answer

Hi, not sure to understand exactly but could this be helpful?


https://helpdesk.scrapinghub.com/support/solutions/articles/22000200451-getting-notifications-on-certain-events


Best regards,


Pablo


1 person has this question

If not, could this be added to a feature request list for Scrapy Cloud?

Answer

Hi, not sure to understand exactly but could this be helpful?


https://helpdesk.scrapinghub.com/support/solutions/articles/22000200451-getting-notifications-on-certain-events


Best regards,


Pablo

It does not help with automation. The question was about a webhook to be called when the spider is completed. The purpose would be to extract data automatically via the API without the need to constantly call the API and check for changes. 


A spider can take anywhere from 2 minutes to 24 hours. If we poll the API for changes every 2 minutes, it adds up to a lot of calls with an empty result.

I was also trying to figure out if this was possible, and I found one way to accomplish webhook functionality is to use the closed() method of a spider:


https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=closed#scrapy.spiders.Spider.closed


This closed() method will be called after the spider is done, so you can make a request to another url here. And since you can pass spiders arguments, you can provide a custom webhook url to notify you when the spider is completed. Hope that helps.


2 people like this
Login to post a comment