Start a new topic
Answered

String field not saved

Hi everybody,


I just ran my first spider on scrapinghub cloud. Everything is fine except one field "description" is not being saved. The spider runs fine on my machine and can save everything in a csv file. The spider does not need javascript rendering, it simply fetches everything from a script tag. 
The only guess I have right now is that there might be a limit on the length of the string. Though the descriptions are not that big, around 400-500 characters each.

Has anybody encountered similar problem before or at least knows how to approach debugging it.


Thanks 


Best Answer

ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python) 


oh yeah, forgot to mention that you can change scrapinghub python version in scrapinghub.yml:

stacks:
  default: scrapy:1.3-py3

 

Answer

ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python) 

Login to post a comment