Start a new topic
Answered

Find the difference between 2 periodic jobs items

I have a periodic job that collects items.

Lets assume that Job number N found the items:

A,B,C,D


Lets assume that Job number N+1 found the items:

B,C,D,Z,T


So we can see that item A is gone and items Z,T found in run N+1


Is there a way that I can calculate the items difference between jobs N and N+1 ?


Best Answer

Hey Balder-man,


I think it can be done through these steps:


1. Using Delta Fetch Addon to avoid repeated items between N and N+1 spider

2. The difference now will be just items collected in (N+1) for any N from 1 to inf. Because every item collected in N+1 is not present in the previous one.


Best regards,


Pablo


Hi

I was reading https://blog.scrapinghub.com/2016/07/20/scrapy-tips-from-the-pros-july-2016/ and I am not sure it can work for me.


I want the spider to do all requests in run N but drop all items that were found in run N-1.

In my example above  - only Z,T should be valid items since they are "new"

How can I do that?


Thanks

I will check this addon

Answer

Hey Balder-man,


I think it can be done through these steps:


1. Using Delta Fetch Addon to avoid repeated items between N and N+1 spider

2. The difference now will be just items collected in (N+1) for any N from 1 to inf. Because every item collected in N+1 is not present in the previous one.


Best regards,


Pablo

Login to post a comment