[Scons-dev] SCons 3.0, sconsign files and Py2 vs Py3.. how to handle sconsign imcompatabilities

[RW]

Hi Bill,
the ascii related error (the one you get when generating using py2 first
then py3 second)
seems to be caused by
line 119, dblite.py, p = f.read()

python 2 reads then decodes a file to a string by default
but python 3 reads the file as binary without decoding.
it then feeds it into pickle.loads()
line 122, dblite.py, self._dict = pickle.loads(p)

I spotted some comments about something similar here
https://stackoverflow.com/questions/28218466/unpickling-a-python-2-object-with-python-3
https://stackoverflow.com/questions/30469575/how-to-pickle-and-unpickle-to-portable-string-in-python-3
I tried some simple decodes of the f.read to get it to work but I think it
might be a case of changing from pickle.loads to pickle.load with an
encoding parameter

For a future version of scons (past 3.0) it might be worth investigating
performance of different storage types.
Based on some of these links I think pickle 4 might be the fastest, then
json then pickle 2
https://stackoverflow.com/questions/2259270/pickle-or-json
https://konstantin.blog/2010/pickle-vs-json-which-is-faster/
https://stackoverflow.com/questions/23582489/python-pickle-protocol-choice

Many Thanks
Richard

On 10 September 2017 at 03:13, Bill Deegan <bill at baddogconsulting.com>
wrote:

> Greetings,
>
> This is (I hope) the last issue gating 3.0 release.
>
> *If we set the default pickle version to 2. (Currently set to
> HIGHEST_AVAILABLE, which means 4 for py3.5+ and 2 for py2.7)*
>
> If you run scons with py2.7 and then run again with py3.x, you'll get the
> following:
> ...
> scons: *** [$DISTDIR/scons-local-3.0.0.alpha.20170821.tar.gz]
> UnicodeDecodeError : 'ascii' codec can't decode byte 0x80 in position 0:
> ordinal not in range(128)
>
> If you run scons with py3 and then rerun with py2.7, you'll get the
> following:
>  * It will rebuild everything you just built with py3
>
> *If we set the default pickle version  to HIGHEST_AVAILABLE, which means 4
> for py3.5+ and 2 for py2.7*
>
>
> Here's the options I see to deal with this:
> 1. Always append .py3 to PY3 scons runs' sconsign files
> 2. Always append .py# to sconsign files based on what version of python
> was used, ignore pre-existing .sconsigns from older versions
> 3. Do something complicated:
>    a. Leave pickle version as HIGHEST_AVAILABLE, detect version of pickle
> by reading first two bytes. If incompatible version detected, change file
> name to .py# and try reading that one.  Always write to .sconsign? Or write
> to file read.
>    b. ?
>
> Thoughts?
>
> -Bill
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170910/18d31c95/attachment-0001.html>