[Scons-dev] RFC: Proposal for remote caching

[Adam Gross]

The random python comparisons I have seen online with python 3 is that SHA-1 is faster than MD5. With python 2 I don’t think that was the case. Remote caching wouldn’t come earlier than 4.0, so it’s reasonable to assume py3 anyway.

I am not sure about SHA-256.

>> Also there needs to be a reasonable solution to (de)serializing which hash is used to sconsign.

Doing it based on length would be fine. The Bazel remote cache change we are going to upstream (adding support for MD5 and SHA-1) does that because the maintainer preferred its simplicity.

From: Scons-dev <scons-dev-bounces at scons.org> On Behalf Of Bill Deegan
Sent: Monday, December 9, 2019 6:51 PM
To: SCons developer list <scons-dev at scons.org>
Subject: Re: [Scons-dev] RFC: Proposal for remote caching

re switching hashes.
Do we have any perf comparisons for MD5 vs SHA-256 in general and in SCons?
While I think adding SHA-256 has value, I'd be hesitant to make it the default and/or remove MD5.
Also there needs to be a reasonable solution to (de)serializing which hash is used to sconsign.

On Mon, Dec 9, 2019 at 1:34 PM Mats Wichmann <mats at wichmann.us<mailto:mats at wichmann.us>> wrote:

> 2.1.3 Changes Needed To Bazel Remote Cache Server
>
> Currently the Bazel remote cache server only supports SHA-256 for requests (e.g. GET http://bazel-cache.corp.int/cache/ac/<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7C0f3e205b7aae4d9514a208d77d02bdb4%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115323036911396&sdata=bUEJneQzYW8VcTrfN4s9uWCawwRWp%2B%2FixwrZOuPSsag%3D&reserved=0><sha_256_hash>), while SCons by default uses MD5. As part of this project, VMware will be contributing code to the upstream Bazel remote cache server project to support MD5 and SHA-1. We have received confirmation from the project maintainer that (1) it is acceptable to do this and (2) no prefix is needed for these alternative hashing formats. As a result, the requests SCons would make would be of the form http://bazel-cache.corp.int/cache/ac/<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7C0f3e205b7aae4d9514a208d77d02bdb4%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115323036911396&sdata=bUEJneQzYW8VcTrfN4s9uWCawwRWp%2B%2FixwrZOuPSsag%3D&reserved=0><md5_hash> or http://bazel-cache.corp.int/cache/ac/<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7C0f3e205b7aae4d9514a208d77d02bdb4%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115323036921389&sdata=jIg1xouKCbwfFxKG4eOatqg4sjWmsj0RBvVX0w1XqgA%3D&reserved=0><sha1_hash>. As mentioned before, see the Threat Modeling section at the end of this page for more discussion on hash formats.

I'm not sure we should actually further push md5.  While it's not
intended to be used for security purposes (and yes I read the section on
that), we've already run into users who are not allowed to use it no
matter what (there's a pending patch to failover to sha1 to address one
of those users' concerns)... and there are fast algorithms in the SHA-2
family as well as ones that didn't quite make the SHA-3 choice (namely,
Blake) which are quite fast on Python.  It may be time to transition?

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
https://pairlist2.pair.net/mailman/listinfo/scons-dev<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7Cgrossag%40vmware.com%7C0f3e205b7aae4d9514a208d77d02bdb4%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115323036921389&sdata=EKLzHg0N8xYEmvYX%2FtvWfeYpHAyN5mNx3COMD55rO9Y%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20191210/0af3785f/attachment.html>