[Scons-dev] Py2/Py3 sconsign compatibility

[Mats Wichmann]

Happy New Year, all!


I put a brief outline of this on IRC, but it's probably better
to include it in a list mailing.

If you switch python versions in a build area, you'll get some
surprising results - perhaps unwanted, perhaps just a "don't
do that".  I'm not saying this a normal thing to do, but what if
there's a shared build area?  I didn't go looking for this, just
stumbled across it in pursuing reproducers for something different.
[my Fedora system is set up so "scons" uses Python 2, so even
though I'm working primarly with Py3, if I forget and just type
"scons" instead of the more explicit stuff shown in the examples
below I get the Py2 version]

First - 3 to 2, starting with clean tree (no sconsign file)

===
$ python3 ~/scons.dev/src/script/scons.py
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o src/hello.o -c src/hello.c
gcc -o src/hello src/hello.o
scons: done building targets.
$ python2 ~/scons.dev/src/script/scons.py
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [SConstruct] ValueError : unsupported pickle protocol: 4
scons: building terminated because of errors.
===

This seems pretty reasonable - with Py3, scons has chosen the
newest pickle protocol of 4 (3 would be used if needed compat with
older Py3 versions, but that is not needed since scons doesn't
support earlier Py3 anyway), so Py2 cannot proceed.  I maybe might
have expected it to bail earlier, but that's not a real problem.

Then - 2 to 3, starting with clean tree (no sconsign file)

===
$ python2 ~/scons.dev/src/script/scons.py
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o src/hello.o -c src/hello.c
gcc -o src/hello src/hello.o
scons: done building targets.
$ python3 ~/scons.dev/src/script/scons.py
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o src/hello.o -c src/hello.c
gcc -o src/hello src/hello.o
scons: done building targets.
===

This seems less normal.  The second run, when there should be
nothing to build, has still rebuilt everthing.  The sconsign
file, which started as a protocol 2 pickle, is now rewritten as
a protocol 4 pickle:

===
$ python2 ~/scons.dev/src/script/sconsign.py .sconsign.dblite
sconsign: ignoring invalid `dblite' file `.sconsign.dblite': unsupported
pickle protocol: 4
===

But it also has problems:

===
$ python3 ~/scons.dev/src/script/sconsign.py .sconsign.dblite
Traceback (most recent call last):
  File "/home/mats/scons.dev/src/script/sconsign.py", line 635, in <module>
    Do_SConsignDB(Map_Module.get(dbm_name, dbm_name), dbm)(a)
  File "/home/mats/scons.dev/src/script/sconsign.py", line 504, in __call__
    for dir in sorted(db.keys()):
TypeError: '<' not supported between instances of 'str' and 'bytes'
===

And this is because the data is now weird:

===
>>> import pickle
>>> with open('.sconsign.dblite', 'rb') as f:
...     p = pickle.load(f)
...
>>> from pprint import pprint
>>> pprint(p)      # output trimmed for brevity
{b'.':
b'\x80\x02}q\x01U\nSConstructq\x02cSCons.SConsign\nSConsignEntry\nq\x03)'
       ... ,
 b'/bin':
b'\x80\x02}q\x01U\x03gccq\x02cSCons.SConsign\nSConsignEntry\nq\x03'
          ... ,
 b'src': b'\x80\x02}q\x01(U\nSConscriptq\x02cSCons.SConsign\nSConsignEntr'
         ... ,
 '.': b'\x80\x04\x95(\x01\x00\x00\x00\x00\x00\x00}\x94\x8c\nSConstruc'
      ... ,
 '/bin':
b'\x80\x04\x95N\x01\x00\x00\x00\x00\x00\x00}\x94\x8c\x03gcc\x94\x8c'
         ... ,
 'src': b'\x80\x04\x95\xe5\x04\x00\x00\x00\x00\x00\x00}\x94(\x8c\nSConscri'
        ... }
===

That is: in reading the Py2-created sconsign, Py3 scons has
kept the old keys - the ones that are strings ('.', '/bin',
'src'), although rewriting the values as bytes objects, and
added its own new keys, which are bytes (b'.', b'/bin', b'src').
Since it apparently doesn't use the old entries at all it decided
to rebuild everything.  And left an unusable sconsign file -
the attempt to sort the data for "consistent presentation"
cannot proceed because of the mix of keys.

Should we do anything about this?
Maybe just consider sconsigns from different versions as
incompatible and bail?  Or try to be a little smarter in
the Py3 case?