Hey everyone. Kinda new to the Scrapy/web scraping world.
So at the moment I'm working for a security (alarms and cameras) company and need to extract all the products their suppliers sell to them. The suppliers won't provide price lists with certain information or in a certain format that I can use to import into this company's new program for managing parts/quotes/etc, so I have to scrape their sites myself. This way I can pull exactly what we need and organise it into the right columns in Excel.
With these sites, the prices are not available if you just visit the site, which is where my problem lies. You have to login to see the price field since each company gets a different discount.
I've successfully extracted products and prices from 3 or so supplier's websites. However, I've started on the next supplier and Portia won't pass through the login screen.
I've filled in the "Perform login" data as I have with the previous jobs I ran but the page either sits there (when I'm testing it as I build the scraper) or returns a "Page not found" error when I actually run it.
The site is https://www.hills.com.au/user/login. I appreciate that support may be limited as it is a login screen and I'm not in a position to share the login details. But I'm just wondering if anyone has the knowledge to see if the website is running something that Portia doesn't like, or isn't compatible with. Or if anyone has any other suggestions it would be greatly appreciated.
I've attached the log from the job and the login refusal seems to happen at line 23.
Thank you advance (and sorry for the long winded message).
Actually the "Page not found" error might have come from something else. But the first bit about it sitting there is still 100% relevent. I clicked Sign In on the site before I wrote my story above, and it still hasn't gone through.