Let us help you decide which technology best fits you


So, you just started creating a Scrapy Cloud project and got a daunting choice: "Build spider with Portia or Scrapy?". Let us help you decide between them.



Both Portia and Scrapy are technologies used to build web crawlers. The biggest difference between them is: Portia is a visual scraping tool that doesn't require any coding skills from you, while Scrapy is an application framework that requires you to code your web crawlers.


So, if you are not a developer, go straight with Portia.


To help you decide between them, we have outlined a small feature comparison based on common requirements:


Requirement
Portia
Scrapy
Can be used by non-developers
Scrape data based on user selection

Scrape data using CSS selectors

Scrape data using XPath expressions

Scrape data from multiple pages

Scrape multiple items of data from a single page

Navigate a website based on crawling rules

Authenticate into a website via login forms

HTTP basic authentication

Form submissions and POST requests

Scrape pages based on JavaScript

Scrape based on JavaScript that require user actions

Scrape data from infinite scrolling pages

Extract data from reverse engineered APIs

Full customization of your crawler behavior

Run both locally and on Scrapy Cloud