[Scons-dev] Subprocess issue on Linux?

[Kenny, Jason L]

Hi guys,

I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well.

The problem is this. We have been building a given huge product here at Intel on RHEL 4.8. IT is old and finally time has come to start moving to a newer system. What was found when we moved to a newer RHEL 5.10 ( I know bleeding edge :) ) we found that the build was twice as slow. Looking into it more it was found that it was the use of fork() in subprocess. It was found that posix_spawn() fixed this. Attached is a monkey patch that is up for internal review to modify subprocess module a little to fix this for Unix based systems.

Now I am not an expert on Linux internal details. However I thought fork() was the way process got spawned on Linux. Ie fork() was the win32 CreateProcess(). I am clearly wrong, but this seems to be a problem I would not expect to have seen on Linux. The root of the problem seems to be that the system takes a long time to setup the fork when you have a large build ( and we have a very large build ) because of the memory usage (2-6GB) needed by SCons alone. Using a different method to create a process that does not try to "clone" memory seems to be the fix that gets the build times to be about ~2x+ faster.

Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost.

Thanks
Jason


Also I attached a test file that to show fork() is the problem. You can see the difference importing the stubprocess.py file and running the test again.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140401/3fc9ec01/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: subprocess_test.py
Type: application/octet-stream
Size: 1183 bytes
Desc: subprocess_test.py
Url : <http://two.pairlist.net/pipermail/scons-dev/attachments/20140401/3fc9ec01/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stubprocess.py
Type: application/octet-stream
Size: 8520 bytes
Desc: stubprocess.py
Url : <http://two.pairlist.net/pipermail/scons-dev/attachments/20140401/3fc9ec01/attachment-0001.obj>