Start a new topic
Answered

how to set the portia crawler not to add repetitive data in job items ?

how we should set the Portia crawler to realize repetitive items and not to add them into the job items ?


Best Answer

You'll need to enable the Deltafetch addon as instructed here: https://helpdesk.scrapinghub.com/support/solutions/articles/22000200411-delta-fetch-addon. As explained in the article, please make sure to also enable DotScrapy Persistence addon.


Hi,


Do you mean from previous crawls?

yes from previous crawls that have done it in another timeĀ 

I mean compare current crawl data with dataset of previous done job and don't add the repetitive items again

Answer

You'll need to enable the Deltafetch addon as instructed here: https://helpdesk.scrapinghub.com/support/solutions/articles/22000200411-delta-fetch-addon. As explained in the article, please make sure to also enable DotScrapy Persistence addon.

Login to post a comment