Start a new topic
Answered

Shub deploy fails, requirements error (I have requirements.txt configured)

I've spent hours trying to find a solution to this.

When trying to do a shub deploy, this happens: 

Entrypoint container is created successfully
>>> Checking python dependencies
awscli 1.10.52 has requirement botocore==1.4.42, but you have botocore 1.10.1.
Warning: Pip checks failed, please fix the conflicts.
{"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /var/folders/qc/dzs6cch91lq21cb50q2c7zhm0000gn/T/shub_deploy_e0o02o4v.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'


 My requirements.txt looks like this:

Distance==0.1.3
nltk==3.2.5
Scrapy==1.5.0
tweepy==3.5.0
scrapinghub==2.0.3
setuptools==36.5.0
gensim==3.4.0
msgpack_python==0.5.6
PyDispatcher==2.0.5


 My scrapinghub.yml looks like this:

project: 285313

requirements:
  file: requirements.txt

stacks:
    default: scrapy:1.1-py3


 It seems that the awscli version is being overrided somehow, as the awscli version that I have installed on my computer is updated. 


Pip freeze, awscli==1.15.1 (on my computer), but when I do a shub deploy, it says that I have awscli==1.10.52


Best Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.


Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.

This worked, thank you!

I'm glad it worked.

Login to post a comment