Start a new topic
Answered

Add parser to requirements.txt

Hi,


I've been following https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud as a guide to include my dependencies in the requirements.txt file, but I'm having some issues.


I'm trying to use urllib.parse to analyse and clean up some URLs. First, I added the module urllib3 to my requirements.txt file as follows:

urllib3==1.22

When I try to deploy the project to Scrapinghub, I see the following error: 

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

{"message": "Internal error", "status": "error"}
Deploy log location: /var/folders/sy/5lq7m7v523v7wqkgymgbs8sm0000gn/T/shub_deploy_f9uckshv.log
Error: Deploy failed: b'{"message": "Internal error", "status": "error"}'

Then, I tried to generate the requirements.txt using pip freeze > requirements.txt, just to be sure that I was doing everything the right way. Now I get the following error while trying to deploy the project to Scrapinghub:

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}

{"message": "Requirements error", "status": "error"}
Deploy log location: /var/folders/sy/5lq7m7v523v7wqkgymgbs8sm0000gn/T/shub_deploy_z25sl9b6.log
Error: Deploy failed: b'{"message": "Requirements error", "status": "error"}'

May be this is a stupid question, but... what should I put on my requirements.txt to import urllib.parse? :_)


Thanks!!


Best Answer

The error seems to be due to  project==1.0 in the requirements.txt.

Collecting project==1.0 (from -r /app/requirements.txt (line 25)) 
Downloading project-1.tar.gz 
Complete output from command python setup.py egg_info: 
---------------------------------------- 
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

 

Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy. 




Nop, sorry @thriveni, it wasn't that. I have removed project from requiremens.txt, but I still see the same error with urllib.parse when deploying to Scrapinghub:

  

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

I've just found the problem, and it is kind of stupid and a newbie one... I was having issues with the Python version :_).


Just changed my imports to:


try:
    from urllib.parse import urlparse, parse_qs, urlencode
except ImportError:
    from urlparse import urlparse, parse_qs
    from urllib import urlencode

Issue solved. Sorry for all the inconveniences, thanks for the help!!

Answer

The error seems to be due to  project==1.0 in the requirements.txt.

Collecting project==1.0 (from -r /app/requirements.txt (line 25)) 
Downloading project-1.tar.gz 
Complete output from command python setup.py egg_info: 
---------------------------------------- 
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

 

Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy. 



Any ideas about this? :)


Thanks!

Hi! 


Thanks for your reply :).


Here are the contents of my requirements.txt, generated using pip freeze: 

apiclient==1.0.3
attrs==16.3.0
certifi==2017.7.27.1
cffi==1.9.1
chardet==3.0.4
click==6.7
constantly==15.1.0
cryptography==1.6
cssselect==1.0.0
docker-py==1.10.6
docker-pycreds==0.2.1
fake-useragent==0.1.7
google-api-python-client==1.6.1
httplib2==0.9.2
hubstorage==0.23.5
idna==2.6
incremental==16.10.1
lxml==3.6.4
numpy==1.11.3
oauth2client==4.0.0
olefile==0.44
pandas==0.19.2
parsel==1.1.0
Pillow==4.2.1
project==1.0
pycparser==2.17
PyDispatcher==2.0.5
pyOpenSSL==16.2.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML==3.12
queuelib==1.4.2
requests==2.18.4
retrying==1.3.3
rsa==3.4.2
scrapinghub==2.0.1
Scrapy==1.2.1
scrapy-fake-useragent==1.1.0
scrapy-splash==0.7
service-identity==16.0.0
shub==2.7.0
simplejson==3.10.0
six==1.10.0
tqdm==4.15.0
treq==17.3.1
Twisted==16.6.0
uritemplate==3.0.0
urllib3==1.22

 And these are the contents from my scrapinghub.yum file:

project: 270487
requirements:
  file: ./requirements.txt

The parse module is included in urllib, so I guess it is enough to add it to the requirements.txt.


Thanks!!

Hello,


From the  previous Failed deploy the error was related to Module name "parse". You would need to include the Module "Parse" to requirements.txt.

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

 

Can you share the Requirements.txt that you are using.

Login to post a comment