Start a new topic
Answered

Use 2 spiders on same website

Hi,


Is it possible to use 2 spiders on the same website ?


I need to scrape 500 products for 10 items each - once a week.

And also need to scrape the same 500 products for only 1 item each - once a day.


Is this possible with Portia ?




Best Answer

Yes it is possible to create 2 spiders for the same website.


Answer

Yes it is possible to create 2 spiders for the same website.

Hi nestor,


Great, how do I do this with Portia ?

Create your first spider to the website, publish it. Then on the left panel click on Show all spiders, then change the URL to the website you want to create it and click on New spider.

Thanks nestor..that works!

Would you be interested in optimizing some spiders that I creates in 2018... so they run as fast as possible ?

You should start by disabling Autothrottle if you want it to run as fast as possible. Although if it leads to bans, you should consider using Crawlera for IP rotation: https://scrapinghub.com/crawlera

Login to post a comment