Start a new topic
Answered

How to set RandomUserAgentMiddleware ?

Hey ,

I am testing Scrapinghub and try to get by . I find all the option really good , I am missing something very important.  How can I  rotate the user agent ?

Is there a way to set up the  DOWNLOADER_MIDDLEWARES  in order to rotate  the user agent ?   If not , is there any  alternative ?

Thank you in advance

Rémi


Best Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud


Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

Thank you for answer nestor .  About the dependencies all fine (if the library would not have not been in the requirement file I would have got an error on it .... ) .  And about the setting.py , that is also good  since I followed the instruction on Pypi  (https://pypi.org/project/scrapy-fake-useragent-fix/)

I guess  something is messing ......

When I got it I post it here :)

Hey ,  Update on my issue.

I change the stack in scrapinghub,yml to run  same the python  version  that I have locally.  The  fake-useragent  library just did not start and did not log anything . That  is why I though  it came form  scrapinghub. 

It is enabled, check log line 6 of your latest jobs and you'll see: 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'. I don't think it logs anything else.


1 person likes this
Login to post a comment