[Scons-dev] env.Clone() vs an overide environment

Kenny, Jason L jason.l.kenny at intel.com
Wed Feb 13 12:22:02 EST 2013

In parts I make a Clone() per Part. So For a product that I build here we have this with -debug=memory -debug=count ( lots of data at end of mail)

The think to note form the mess below is that from
0 5 5 5 Environment.Base
0 506 2350 2350 Environment.EnvironmentClone
0 0 5228 5228 Environment.OverrideEnvironment

Parts for a speed reason, tries to clone environments as much as it can. However given the Scons does not really have a Environment.EnvironmentClone class, this is just a note to the debug code here that this is a cloned environment. Since this code is a full copy ( in general) of the Environment.Base there seems to be a lot of remove here to improve memory usage. For example you will see stuff like:
0 291 102722 102722 parts.common.namespace

This object in Parts is just a dictionary that has some tweaks to allow us to say stuff like:

Env['MSVC']['VERSION'] or
Env['MSVC'].VERSION
Or env.subst("${MSVC.VERSION}")

And have everything work correctly. I have to make a copy of this for every Clone() as I don't know if someone will tweak data in it. I have not tried yet to make a readonly version of it, so it does not get copied, but this object like many other that might get modified are great examples of the values we have in the environment that could be greatly reduced if we had a smarter copy on write-ish setup. I think this would greatly simplify the Clone logic, if we made a real EnvironmentClone object. I am not saying this is a silver bullet, but part of a set of stuff to change. ( another item I want to tweak to get better data on is the subst string classes in Scons. I believe these objects may be taking more mem than needed, but I don't have any data in this run here). Of the 1.6 GB I have before Scons start building all the code. I believe a good chunk of that could be reduced. I don't know yet of any way for Python to tell me how much memory an object is taking. This gets complex in the environment object as it hold pointers to other items, but some items in it we don't want to count, such as the fs attribute, as the points to a container of all the file based Nodes that are known. The environment being mostly a dictionary, by itself probably does not take a lot of mem, but set of stuff it hold, probably does when you add it together. Given the logic is general to copy everything, so the user can modify it safely, it seem making copies of only the things that could be modified because the user changed it seems like a good change to consider.

Jason

Data from a real product run. ( I not going to what product...)

Memory before reading SConscript files: 15294464
Memory after reading SConscript files: 360620032
Memory before Parts processed 360620032
Memory Before post logic queue processing 1625415680
Memory After post logic queue processed 1625415680
Memory after Parts processed 1625415680
Memory before building targets: 1625415680
Memory after building targets: 3891105792
Object counts:
pre- post- pre- post-
read read build build Class
0 1 1 1 <logger>text.text
9 90 253 255 Action.CommandAction
0 79 177 177 Action.CommandGeneratorAction
9 53 177 177 Action.FunctionAction
9 83 204 204 Action.LazyAction
0 84 187 187 Action.ListAction
0 1 1 1 BoolVariable
2 275 732 732 Builder.BuilderBase
0 79 177 177 Builder.CompositeBuilder
0 0 0 0 Builder.OverrideWarner
0 7 7 7 ColorTextStream
0 1 1 1 Console
0 8 8 8 ConsoleColor
0 5 5 5 Environment.Base
0 506 2350 2350 Environment.EnvironmentClone
0 0 5228 5228 Environment.OverrideEnvironment
0 7 7 7 Event
2 131 52982 54448 Executor.Executor
0 0 0 36375 Executor.Null
0 2 14527 14527 Node.Alias.Alias
1 1 1 1 Node.FS
3 254 78722 100915 Node.FS.Base
1 129 8173 9617 Node.FS.Dir
0 0 52814 71607 Node.FS.Entry
1 124 17733 19689 Node.FS.File
1 1 2 2 Node.FS.RootDir
3 256 93294 115487 Node.Node
0 0 45 45 Node.Python.Value
0 0 0 0 QueueLogger
0 2 2 2 SCons.Builder.ListEmitter
0 0 0 3 SCons.Subst.ListSubber
0 5933 26089 26089 SCons.Util.CLVar
0 0 4058 15842 SCons.Util.NodeList
0 1 1 1 Settings
0 5 5 5 Variable
0 1 1 1 Variables
0 9 9 9 parts.Variables.BoolVariable.BoolVariable
0 1 1 1 parts.Variables.EnumVariable.EnumVariable
0 10 10 10 parts.Variables.ListVariable.ListVariable2
0 131 131 131 parts.Variables.variable.Variable
0 7 7 7 parts.color.ConsoleColor
0 595 595 595 parts.common.DelayVariable
0 0 1751 1751 parts.common._make_rel
0 0 13732 13732 parts.common._make_reld
0 291 102722 102722 parts.common.namespace
0 2 2 2 parts.config._ConfigurationSet
0 4 4 4 parts.config.configuration
0 0 0 0 parts.console.Cursor
0 0 2520 2520 parts.dependent_ref.dependent_ref
0 0 67 67 parts.dependson.ComponentEnv
0 1 1 1 parts.engine.parts_addon
0 155 155 155 parts.events.Event
0 0 0 0 parts.functors.map_build_context
0 0 2521 2521 parts.functors.map_depends
0 0 1 1 parts.loadlogic.all.All
0 0 0 0 parts.logger.QueueLogger
0 0 0 0 parts.mappers.Base
0 0 0 0 parts.mappers.TempFileMunge
0 0 0 0 parts.mappers.part_id_export_mapper
0 0 0 0 parts.mappers.part_id_mapper
0 0 0 0 parts.mappers.part_mapper
0 0 0 0 parts.mappers.part_name_mapper
0 0 0 0 parts.mappers.part_shortname_mapper
0 0 0 0 parts.mappers.part_subst_mapper
0 0 0 0 parts.mappers.relpath_mapper
0 0 67 67 parts.node_helpers._AbsDir
0 0 67 67 parts.node_helpers._AbsFile
0 253 93291 115484 parts.overrides.UniqueList
0 7 7 7 parts.overrides.scanner.PartPathDirsWrapper
0 0 0 0 parts.overrides.tool.Parts_Tool
0 151 735 735 parts.part_logger.part_logger
0 0 0 0 parts.part_logger.part_nil_logger
0 151 5952 5952 parts.part_logger.part_spawner
0 151 735 735 parts.part_logger.parts_text_logger
0 0 0 0 parts.part_logger.pipeRedirector
0 0 0 0 parts.part_logger.process_wait
0 1 1 1 parts.part_manager.part_manager
0 0 2520 2520 parts.part_ref.part_ref
0 0 80 80 parts.pattern.Pattern
0 194 799 799 parts.platform_info.SystemPlatform
0 0 2520 2520 parts.pnode.dependent_info.dependent_info
0 151 735 735 parts.pnode.part.part
0 1 1 1 parts.pnode.pnode_manager.manager
0 151 735 735 parts.pnode.section.build_section
0 0 2520 2520 parts.requirement.REQ
0 0 0 0 parts.requirement.requirement
0 0 0 0 parts.requirement.requirement_internal
0 52 52 52 parts.requirement.requirement_set
0 1 1 1 parts.section.section
0 10 10 10 parts.settings.deprecated
0 0 0 0 parts.settings.string_tester
0 0 2520 2520 parts.target_type.target_type
0 59 59 59 parts.tools.Common.Finders.EnvFinder
0 30 30 30 parts.tools.Common.Finders.PathFinder
0 25 25 25 parts.tools.Common.Finders.RegFinder
0 50 50 50 parts.tools.Common.Finders.ScriptFinder
0 34 34 34 parts.tools.Common.ToolInfo.ToolInfo
0 8 8 8 parts.tools.Common.ToolSetting.ToolSetting
0 0 0 0 parts.tools.GnuCommon.android.posix_scanner
0 9 9 9 parts.tools.GnuCommon.android.win_scanner
0 0 0 0 parts.tools.GnuCommon.binutils.BinutilInfo
0 1 1 1 parts.tools.GnuCommon.binutils.BinutilsSetupWrapper
0 4 4 4 parts.tools.GnuCommon.common.GnuInfo
0 29 29 29 parts.tools.IntelCommon.common.IntelcInfo
0 0 0 0 parts.tools.IntelCommon.filescanner.file_scanner11
0 0 0 0 parts.tools.IntelCommon.filescanner.file_scanner12
0 0 0 0 parts.tools.IntelCommon.filescanner.file_scanner9_10
0 7 7 7 parts.tools.IntelCommon.regscanner.reg_scanner
0 6 6 6 parts.tools.IntelCommon.regscanner.reg_scanner2
0 16 16 16 parts.tools.IntelCommon.regscanner.reg_scanner_v12
0 47 1820 1820 parts.version.VersionPart
0 212 2763 2763 parts.version.version
0 91 1997 1997 parts.version.version_range
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20130213/a190f452/attachment.html>

More information about the Scons-dev mailing list