[Scons-dev] RFC: Proposal for remote caching

[Adam Gross]

>> It just seems odd to me to add md5 support anywhere that doesn't have it, when you usually get flamed by someone when you mention md5 these days :)

The Bazel remote caching maintainer indicated that he is fine accepting a patch to support MD5.

-----Original Message-----
From: Scons-dev <scons-dev-bounces at scons.org> On Behalf Of Mats Wichmann
Sent: Tuesday, December 10, 2019 10:51 AM
To: scons-dev at scons.org
Subject: Re: [Scons-dev] RFC: Proposal for remote caching

On 12/10/19 6:50 AM, Adam Gross via Scons-dev wrote:
> The random python comparisons I have seen online with python 3 is that
> SHA-1 is faster than MD5. With python 2 I don’t think that was the case. 
> Remote caching wouldn’t come earlier than 4.0, so it’s reasonable to 
> assume py3 anyway.
> 
> I am not sure about SHA-256.

256 is a bit slower. Here are some random relative timings from a Linux host, excluding the sha3* family (which are all slower):

==== md5        [4.895055413013324, 4.794695725897327, 4.788602610118687]
==== sha1       [4.8523716980125755, 4.790286544011906, 4.779321678215638]
==== sha256     [5.980340243084356, 6.025372178060934, 5.983012914890423]
==== sha512     [7.033806097926572, 7.030697694979608, 7.019461386138573]
==== blake2b_20 [4.735298512969166, 4.768527464941144, 4.754039308987558]
==== blake2b    [4.883447706932202, 4.856357607059181, 4.866529860068113]
==== blake2s_20 [4.393687708070502, 4.392260011984035, 4.4033572028856725]
==== blake2s    [4.266355097061023, 4.233708509942517, 4.228863182011992]
==== sha224     [5.833340571029112, 5.849389663897455, 5.8033752529881895]
==== sha384     [6.833396520931274, 6.844970899866894, 6.844409989891574]


On a Windows host the timings fall into roughly the same ranking, although blake2s wins by more over md5/sha1 (it's more like 40% faster than 12% faster)

This isn't particularly important, I'm just following up.

It just seems odd to me to add md5 support anywhere that doesn't have it, when you usually get flamed by someone when you mention md5 these days :)


> 
>  >> Also there needs to be a reasonable solution to (de)serializing 
> which hash is used to sconsign.
> 
> Doing it based on length would be fine. The Bazel remote cache change 
> we are going to upstream (adding support for MD5 and SHA-1) does that 
> because the maintainer preferred its simplicity.
> 
> *From:* Scons-dev <scons-dev-bounces at scons.org> *On Behalf Of *Bill 
> Deegan
> *Sent:* Monday, December 9, 2019 6:51 PM
> *To:* SCons developer list <scons-dev at scons.org>
> *Subject:* Re: [Scons-dev] RFC: Proposal for remote caching
> 
> re switching hashes.
> 
> Do we have any perf comparisons for MD5 vs SHA-256 in general and in SCons?
> 
> While I think adding SHA-256 has value, I'd be hesitant to make it the 
> default and/or remove MD5.
> 
> Also there needs to be a reasonable solution to (de)serializing which 
> hash is used to sconsign.
> 
> On Mon, Dec 9, 2019 at 1:34 PM Mats Wichmann <mats at wichmann.us 
> <mailto:mats at wichmann.us>> wrote:
> 
> 
>      > 2.1.3 Changes Needed To Bazel Remote Cache Server
>      >
>      > Currently the Bazel remote cache server only supports SHA-256 for
>     requests (e.g. GET https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0
>     <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0><sha_256_hash>),
>     while SCons by default uses MD5. As part of this project, VMware
>     will be contributing code to the upstream Bazel remote cache server
>     project to support MD5 and SHA-1. We have received confirmation from
>     the project maintainer that (1) it is acceptable to do this and (2)
>     no prefix is needed for these alternative hashing formats. As a
>     result, the requests SCons would make would be of the form
>     https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0
>     <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0><md5_hash>
>     or https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0
>     <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbazel-cache.corp.int%2Fcache%2Fac%2F&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=mBZsTOVikjfTp8XyfnKF6eLvoSHSqbjloeNjPZowKR4%3D&reserved=0><sha1_hash>.
>     As mentioned before, see the Threat Modeling section at the end of
>     this page for more discussion on hash formats.
> 
>     I'm not sure we should actually further push md5.  While it's not
>     intended to be used for security purposes (and yes I read the
>     section on
>     that), we've already run into users who are not allowed to use it no
>     matter what (there's a pending patch to failover to sha1 to address one
>     of those users' concerns)... and there are fast algorithms in the SHA-2
>     family as well as ones that didn't quite make the SHA-3 choice (namely,
>     Blake) which are quite fast on Python.  It may be time to transition?
> 
>     _______________________________________________
>     Scons-dev mailing list
>     Scons-dev at scons.org <mailto:Scons-dev at scons.org>
>     https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=9dNy1%2FcxKJdSNBuTbRWTbpSAr%2BiS4UEfHvAr%2FqTZsAA%3D&reserved=0
>     
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpai
> rlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7Cgr
> ossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b
> 4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=9dNy1%2FcxKJ
> dSNBuTbRWTbpSAr%2BiS4UEfHvAr%2FqTZsAA%3D&reserved=0>
> 
> 
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpair
> list2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7Cgro
> ssag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4
> aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=9dNy1%2FcxKJd
> SNBuTbRWTbpSAr%2BiS4UEfHvAr%2FqTZsAA%3D&reserved=0
> 

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7Cgrossag%40vmware.com%7Ce501729801484fcc03e608d77d88c3bc%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637115900200121208&sdata=9dNy1%2FcxKJdSNBuTbRWTbpSAr%2BiS4UEfHvAr%2FqTZsAA%3D&reserved=0