[Scons-dev] SCons 3.0, sconsign files and Py2 vs Py3.. how to handle sconsign imcompatabilities

[RW]

Okay so changing
pickle.loads(p)
to
pickle.loads(p, encoding='bytes')

seems like a partial fix
if there's a .sconsign.dblite file saved in python 2 with pickle protocol 2
python 3 won't crash now, instead it'll just rebuild the sources / ignore
the current state

going the opposite way from python 3 to python 2 gives me the error
scons: *** [mkdocs.yml] ValueError : unsupported pickle protocol: 4

I suppose you could force it to always use pickle protocol 2 (not sure how
to do that)
the end result would be that it would always rebuild when going from py2 ->
py3 or py3 -> py2
so it wouldn't crash but ignore the build state (perhaps the file hashes
are calculated differently?)

pickle protocol 4 might be a lot quicker than pickle protocol 2 btw I'm not
sure
maybe there should be a way of setting it as an option within SConstruct

Many Thanks
Richard

On 10 September 2017 at 13:49, RW <garlicbready at googlemail.com> wrote:

> Hi Bill,
> the ascii related error (the one you get when generating using py2 first
> then py3 second)
> seems to be caused by
> line 119, dblite.py, p = f.read()
>
> python 2 reads then decodes a file to a string by default
> but python 3 reads the file as binary without decoding.
> it then feeds it into pickle.loads()
> line 122, dblite.py, self._dict = pickle.loads(p)
>
> I spotted some comments about something similar here
> https://stackoverflow.com/questions/28218466/unpickling-
> a-python-2-object-with-python-3
> https://stackoverflow.com/questions/30469575/how-to-
> pickle-and-unpickle-to-portable-string-in-python-3
> I tried some simple decodes of the f.read to get it to work but I think it
> might be a case of changing from pickle.loads to pickle.load with an
> encoding parameter
>
> For a future version of scons (past 3.0) it might be worth investigating
> performance of different storage types.
> Based on some of these links I think pickle 4 might be the fastest, then
> json then pickle 2
> https://stackoverflow.com/questions/2259270/pickle-or-json
> https://konstantin.blog/2010/pickle-vs-json-which-is-faster/
> https://stackoverflow.com/questions/23582489/python-pickle-protocol-choice
>
> Many Thanks
> Richard
>
> On 10 September 2017 at 03:13, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> Greetings,
>>
>> This is (I hope) the last issue gating 3.0 release.
>>
>> *If we set the default pickle version to 2. (Currently set to
>> HIGHEST_AVAILABLE, which means 4 for py3.5+ and 2 for py2.7)*
>>
>> If you run scons with py2.7 and then run again with py3.x, you'll get the
>> following:
>> ...
>> scons: *** [$DISTDIR/scons-local-3.0.0.alpha.20170821.tar.gz]
>> UnicodeDecodeError : 'ascii' codec can't decode byte 0x80 in position 0:
>> ordinal not in range(128)
>>
>> If you run scons with py3 and then rerun with py2.7, you'll get the
>> following:
>>  * It will rebuild everything you just built with py3
>>
>> *If we set the default pickle version  to HIGHEST_AVAILABLE, which means
>> 4 for py3.5+ and 2 for py2.7*
>>
>>
>> Here's the options I see to deal with this:
>> 1. Always append .py3 to PY3 scons runs' sconsign files
>> 2. Always append .py# to sconsign files based on what version of python
>> was used, ignore pre-existing .sconsigns from older versions
>> 3. Do something complicated:
>>    a. Leave pickle version as HIGHEST_AVAILABLE, detect version of pickle
>> by reading first two bytes. If incompatible version detected, change file
>> name to .py# and try reading that one.  Always write to .sconsign? Or write
>> to file read.
>>    b. ?
>>
>> Thoughts?
>>
>> -Bill
>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170910/ba5214a4/attachment.html>