ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Rafkind <dave.rafk...@gmail.com>
Subject Re: ibatis 3 memory leak
Date Mon, 18 Jan 2010 16:37:29 GMT
Unfortunately the data model is very thin (it's a database representation of
a very large series of XSD schema documents), with "types" that have
"properties", where each "type" has a "parent", and each "property" has a
"type" as well (among other things).

Big objects and N+1 selects don't bother me too much, but since a given
"type" or "property" may be involved in several relationships I was hoping
that information about them could be pulled from the cache instead of
re-selected, during lazy-load joins.

On Mon, Jan 18, 2010 at 11:10 AM, Clinton Begin <clinton.begin@gmail.com>wrote:

> Hmm... it looks to me like your situation is a perfect storm.
>  * You're loading lots of objects.
>  * You're loading big objects.
>  * You are guaranteeing an N+1 selects problem with that for loop.
> No one of these situations should create a major problem, but all three
> together are a guaranteed disappointment.
> That said, I'm surprised that 2000 rows would be that much of a problem...
> how big are these objects really?
> Also, memory wise, there should be almost no difference between using one
> session or multiple.  And no, I would not open multiple as in your second
> example.
> As for the serialization error, that might be a bug if LoadPair wasn't
> serializable.
> Clinton
> On Mon, Jan 18, 2010 at 8:24 AM, Dave Rafkind <dave.rafkind@gmail.com>wrote:
>> Ok, that's some good information, I understand that you should marshall
>> your objects with care. Unfortunately for me at some point I will have to
>> marshall every single one of my database rows into an object graph. However
>> what I took from your advice was not to be so stingy in re-using the session
>> objects. So if I do something like this:
>> for (id : ids) {
>>   SqlSession session2 = sessionMapper.openSession();
>>   MyObject o = session2.select("getOne", id.getActualId());
>>   session2.close();
>> }
>> Now memory usage is pretty good! However now performance is terrible. In
>> attempting to rectify this I tried to put in a custom cache implementation,
>> logging cache access to see how it was performing. I noticed it was never
>> actually using the cache (never doing cache.putObject).
>> I noticed that according to this discussion:
>> http://markmail.org/search/?q=select+commit+cache+list%3Aorg.apache.ibatis.user-java#query:select%20commit%20cache%20list%3Aorg.apache.ibatis.user-java+page:1+mid:4mmwki3dnp57gweu+state:results
>> ...commits don't occur (and thus cache fill doesn't occur) unless a
>> transaction is committed. So for a pure read-only use case of the db the
>> cache won't be very useful, right?
>> Additionally, putting in a "session2.commit()" before the call to close()
>> causes the following error (which I assume, but don't know is caused by
>> committing when nothing actually needs to be comitted:)
>> ### Error committing transaction.  Cause:
>> org.apache.ibatis.cache.CacheException: Error serializing object.  Cause:
>> java.io.NotSerializableException:
>> org.apache.ibatis.executor.loader.ResultLoaderRegistry$LoadPair
>> On Thu, Jan 14, 2010 at 2:44 PM, Clinton Begin <clinton.begin@gmail.com>wrote:
>>> By nested results, yes, I mean collections and associations.  And by
>>> "flattening" I mean avoiding the use of those.
>>> iBATIS exhibits this behavior, the same way any ORM would, because the
>>> object instances need to be cached to preserve object identity.  So as
>>> you're result set is being read through, each unique object is stored.
>>> iBATIS isn't quite as effcient as something like Hibernate can be with
>>> circular references though, in that depending on how you map it out, you may
>>> end up with multiple instances of the same data (parent/child relationships
>>> mapped with resultMap).
>>> So with iBATIS, the most memory efficient approach is to use nested
>>> select associations/collections.  But for query performance, nested
>>> resultMaps are ideal.  I often find I need to use a combination of both to
>>> get the best optimization.  But if I often don't load lists of complete
>>> objects either.  If I'm loading a large list, I'll use a lighter weight
>>> representation, and then only load the complete object graph for individual
>>> instances.  I've never found a case where this wasn't a good idea anyway.
>>> Even when working with something like Rails, a rich domain ORM, I would
>>> often write optimized lightweight queries for large lists of flatter
>>> objects.
>>> The memory should not be significantly more than will ultimately be
>>> required to store your final result set.   And any additional memory used
>>> should be released upon the closing of the SqlSession.
>>> If the memory isn't being released at the end of the session, that's a
>>> different story... but otherwise, this is normal behavior.
>>> Clinton
>>> On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <dave.rafkind@gmail.com>wrote:
>>>> Thanks for the reply. What do you mean by nested result maps or selects?
>>>> Do you mean collections or associations with their own selects and result
>>>> maps? Why would ibatis exhibit this behavior in that case?
>>>>  And by flattening, you mean the same kind of stuff used to avoid the
>>>> n+1 select problem?
>>>> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <clinton.begin@gmail.com
>>>> > wrote:
>>>>> If it uses nested result maps or nested selects, I'm afraid you're out
>>>>> of luck.   You'll need to reduce the query results, or flatten out the
>>>>> results.
>>>>> Clinton
>>>>> On 2010-01-14, Dave Rafkind <dave.rafkind@gmail.com> wrote:
>>>>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question.
>>>>> I'm
>>>>> > using Ibatis 3 (ibatis-3-core- with a somewhat
>>>>> complicated
>>>>> > schema (plenty of circular links etc).
>>>>> >
>>>>> > I'm doing something like this:
>>>>> >
>>>>> > List<MyIdObject> ids = session.selectList("getAll");
>>>>> >
>>>>> > for (id : ids) {
>>>>> >   MyObject o = session.select("getOne", id.getActualId());
>>>>> > }
>>>>> >
>>>>> > The first query returns a list about 2k big, and the second query
>>>>> the for
>>>>> > loop returns objects that are somewhat large (have several
>>>>> collections in
>>>>> > them, a discriminator, etc).
>>>>> >
>>>>> > The problem I have is that as the for loop marches on it uses an
>>>>> > ever-increasing amount of memory. I would assuming that when the
>>>>> objects in
>>>>> > the body of the for loop go out of scope they can get garbage
>>>>> collected, but
>>>>> > apparently that never happens; is there some weird interaction with
>>>>> the
>>>>> > "first-level cache"? Should I be going about this a different way?
>>>>> >
>>>>> > Thanks!
>>>>> > Dave
>>>>> >
>>>>> --
>>>>> Sent from my mobile device
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>>>>> For additional commands, e-mail: user-java-help@ibatis.apache.org

View raw message