Start a new topic
Answered

Limiting Link Depth?

I don't need my spiders to go very far into the sites that I'm looking at. From what I understand, the link depth can be limited by setting "DEPTH_LIMIT = 1" or the like in settings.py of the project's code. But how can I do something like that through Scraping Hub?

Or am I going to have to make use of Scrapy?


Best Answer

You can add that setting to your project settings in Scrapinghub: https://app.scrapinghub.com/p/projectid/job-settings/standard


Answer

You can add that setting to your project settings in Scrapinghub: https://app.scrapinghub.com/p/projectid/job-settings/standard


1 person likes this

Excellent, thank you!

You're welcome :)

Login to post a comment