[Scons-dev] Announcement: Core changes for v2.4 ahead, Switching the Node class to using slots...

[Kenny, Jason L]

So adding a __dict__ to __slots__ will add a <type 'dict'> object. This only get assign dynamic attributes. So there is probably some memory mapped to it for the core dictionary. A test would have to be done to see how much worse it would make it. Most of any memory waste comes from add stuff to it.

Looking at the Parts code I think I can redo the code states without the Node object having __dict__ added to them. The FileSymbolicNode object we add is subclass so all it state is fine, and I think we use __slots__ for that already in Parts.

I am at a point I would rather monkey patch to quickly deal and show value in a fix, and want to move more towards get it fixed in SCons. So let me fix up Parts to use a _memo, or attribute object with some namespace object to work around this... I think it should be easy to get working in a hour or two...

FYI.. all the tweak I have being made can be seen at 

https://bitbucket.org/dragon512/parts-jason

so you might want to look at that source as it will have all the fixes before I push it to the main trunk. (or whatever trunk is in git talk...)

Jason


-----Original Message-----
From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of Dirk Bächle
Sent: Tuesday, March 31, 2015 2:41 PM
To: SCons developer list
Subject: Re: [Scons-dev] Announcement: Core changes for v2.4 ahead, Switching the Node class to using slots...

Jason,

On 31.03.2015 21:06, Kenny, Jason L wrote:
> [...]
>
> So let me think this out load here. The main issues is that I have properties that I am adding to the node objects and this fails with __slots__ being defined without a __dict__,  because I was adding a cache value to the object, vs having to look up the value on a different object ( ie the attributes object) as this could conflict with who-knows-what that was added with the Tag()/MetaTag() API by the user.
>
> The plus side for having __dict__ added to the base is that it makes 
> it easy to see what is being added to a node and it makes it easy to 
> prototype new stuff in a system. Keep in mind that point of __slots__ 
> is not about removing __dict__, but allowing python to be efficient in 
> pre-allocating memory as an array vs that of independent items as they 
> would be in the __dict__ object. Ideally Cpython should be doing 
> something else here, so __slots__ add no value. ( hmm somebody should 
> fix this...) But until then __slots__ add value for this reason. Add 
> __dict__ by default only mean that the code as any normal python 
> object. One can argue that the restriction we add by using slots in 
> add new member dynamically is a good thing or a bad thing. I just want 
> to point out that the use of __slots__ means that the memory usage is 
> reduced because of block array allocation ( vs lots of small alloc), 
> and the memory layout of the object is more machine friendly ( as any 
> CPU like array for quick acce
 s
s, and hate everything else). This is why we have a space and speed improvement in SCons with the nodes as the node are the prime data structure when we build.
>

I'm mainly interested in reducing memory consumption, and dictionaries require a lot of bytes. So, if you can add your "__dict__" 
without directly assigning a fully-blown dict to any Node instance by default...you win me over in the same second.
But this would require some sort of lazy initialization like I did for the abspath/tpath attributes, and I have the feeling that that's not directly what you're after. ;)

> Let me look at what I can do on my end, with a fresh set of eyes.. if 
> you still feel that adding __dict__ is a bad thing. ( will miss the 
> clean property syntax )
>
> There are a few other fixes I have here in parts/overrides/nodes.py that would be nice to have added to the main base. Could you look at these and see if any seem OK for you to add to the Node classes?
>

Definitely, I'll have a look.

Regards,

Dirk

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev