cayenne-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Zeigler <robert.zeig...@roxanemy.com>
Subject Re: Cayenne object storage / memory usage
Date Sat, 17 Jun 2017 06:01:28 GMT
I’m also a little surprised at the 1/2-ing… what were the values being stored? I suppose
in theory, many values are relatively “small”, memory-wise, so having the overhead of
also storing the key could ~double the memory use, but if you’re storing large values, I
wouldn’t expect the utilization to drop as dramatically. What were your data values (type
and length distribution for strings)?

Thanks!

Robert

> On Jun 10, 2017, at 6:49 AM, Michael Gentry <blacknext@gmail.com> wrote:
> 
> Hi John,
> 
> I'm a little surprised that map-based storage is over 2x worse in memory
> consumption.  I'm wondering if there is more going on here than storage of
> the property values.  Would it be simple enough to adapt your test case to
> compare a list of POJOs vs a list of maps and see what the memory footprint
> and difference is that way?
> 
> I personally was thinking the big improvement for using fields directly is
> the speed improvement.  I didn't think the memory consumption difference
> would be that dramatic.
> 
> Thanks,
> 
> mrg
> 
> 
> On Fri, Jun 9, 2017 at 10:55 AM, John Huss <johnthuss@gmail.com> wrote:
> 
>> I did some experimenting recently to see if changes to the way data in
>> stored in Cayenne objects could reduce the amount of memory they consume.
>> 
>> I chose to use separate fields for each property instead of a HashMap
>> (which is what CayenneDataObject uses).  The results were very affirming.
>> For my test of loading 10,000 objects from every table in my database I got
>> it to use about about *half the memory* of the default class (from 921 MB
>> down to 431 MB).
>> 
>> I know there has been some discussion already about addressing this topic
>> for the next major release, so I thought I'd throw in some observations /
>> questions here.
>> 
>> For my implementation I subclassed CayenneDataObject because in previous
>> experience I found implementing a replacement to be much more difficult and
>> subject to more bugs due to the less frequently used code path that
>> PersistentObject and it's descriptors take you down.  My apps rely on
>> things that are sort of specific to CayenneDataObject like Validating.
>> 
>> So one question is how we should be addressing the need that people may
>> have to create their own data classes. Right now I believe the recommended
>> path is to subclass PersistentObject, but I'm not convinced that that is a
>> viable solution without wholesale copying most of CayenneDataObject into
>> your subclass.  I'd rather see a fuller base class (in addition to keeping
>> PersistentObject around) that includes all of CayenneDataObject except the
>> property storage (HashMap).
>> 
>> For my implementation I had to modify CayenneDataObject, but only slightly
>> to avoid creating the HashMap which I wasn't using. However, because class
>> isn't really intended for customization this map is referenced in multiple
>> methods that can't easily be overridden to change the way things are
>> stored.
>> 
>> Another approach might be to ask why anyone should need to customize the
>> way data is stored in the objects if we can just use the best solution
>> possible in the first place?  I can't imagine a more efficient
>> representation that fields.  However, fields present difficulties for the
>> use case where you aren't generating unique classes for your model but just
>> rely on the generic class.  In theory this could be addressed via runtime
>> code generation or something else, but that would be quite a change.
>> 
>> So I'm looking forward to discussing this and toward the future.
>> 
>> John
>> 


Mime
View raw message