0
Answered
Seok-chan Ahn 1 week ago in Scrapy Cloud • updated by Pablo Vaz (Support Engineer) 6 days ago 2

Hi, I am using Scrapy Cloud

I want to customize my Slack Incoming WebHooks to get error information during execution of jobs in scrapy cloud.

Could I get error notifications in real-time or after job completion?
I want some polling web hooks or something else


Answer

Answer

Hey Seok more updates for you from our best developers.

About Notifications, perhaps is not the suitable option due the limitations on adding comments by people and not by the system.

Using Jobs API only allows for probing what the status of a job, so if you want to know what's up with a job, you currently have to do regular polling, which is the opposite of what you want (a webhook for when the event happens).

So basically more than an Answer I probably give more headaches :) Sorry about that.

It has been said, that there're some projects to review things like you propose and some of our developers could be motivated to see our users asking to improve notifications in real time.

Another suggestion provided for some ancient masters here in SH, is to simply implement a python script to check job status as posted here: https://blog.scrapinghub.com/2016/09/28/how-to-run-python-scripts-in-scrapy-cloud/

I think that this approach will be very useful by now...

Hope this time give you a better solution.

Kind regards!

Answered

Hey Seok-chan Ahn,

Yes you could use our extensive collection of APIs to perform all type of duties. Please take a time to explore different options, I strongly recommend in your case to use Jobs API:

https://doc.scrapinghub.com/scrapy-cloud.html#app-scrapinghub-com

Also check this article, it may be helpful for your purposes:

http://help.scrapinghub.com/scrapy-cloud/notifications

Kind regards.

Answer

Hey Seok more updates for you from our best developers.

About Notifications, perhaps is not the suitable option due the limitations on adding comments by people and not by the system.

Using Jobs API only allows for probing what the status of a job, so if you want to know what's up with a job, you currently have to do regular polling, which is the opposite of what you want (a webhook for when the event happens).

So basically more than an Answer I probably give more headaches :) Sorry about that.

It has been said, that there're some projects to review things like you propose and some of our developers could be motivated to see our users asking to improve notifications in real time.

Another suggestion provided for some ancient masters here in SH, is to simply implement a python script to check job status as posted here: https://blog.scrapinghub.com/2016/09/28/how-to-run-python-scripts-in-scrapy-cloud/

I think that this approach will be very useful by now...

Hope this time give you a better solution.

Kind regards!