cayenne-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Huss <johnth...@gmail.com>
Subject Re: Watch out for memory leaks with EhCache
Date Tue, 10 Dec 2019 18:39:11 GMT
On Tue, Dec 10, 2019 at 9:51 AM John Huss <johnthuss@gmail.com> wrote:

>
> Thanks for the feedback - I appreciate it.
>
> On Sun, Dec 8, 2019 at 3:17 AM Andrus Adamchik <andrus@objectstyle.org>
> wrote:
>
>> Hi John,
>>
>> Thanks for the analysis. You've put a significant effort in this issue,
>> but I still feel like we have differences in understanding of the causes
>> (and hence the remedies). Let me try to clarify how I see it.
>>
>> > On Dec 7, 2019, at 12:09 AM, John Huss <johnthuss@gmail.com> wrote:
>> >
>> > 1) The lifetime of entries in the Local Query Cache exceeds their
>> > availability, which is the life of their ObjectContext. Any cache that
>> is
>> > not expiring entries (or limiting them) will just leak this memory.
>>
>> A properly configured cache will "overcommit" memory, but not "leak"
>> (i.e. it won't expand indefinitely). Misconfigured cache was the cause of
>> the the issue in the parent message. Once I added upper limits on the
>> number of entries to all cache groups, the leak disappeared.
>>
>> > 2) Cached query results will retain the ObjectContext they were fetched
>> > into, which in turn may retain a much larger number of objects than
>> > intended. For example. If you use a single ObjectContext to fetch 1
>> million
>> > uncached objects along with 1 cached object, you will retain 1 million
>> and
>> > 1 objects in memory rather than just 1.
>>
>> This doesn't look right. Objects are stored via weak references by the
>> OC, so if there are no other references to them, they will be GC'd even if
>> the context is still around. I wrote a self-contained test project [1] that
>> proves this assumption [2]. If you see unexpected extra objects cached,
>> they are likely retained via prefetched / faulted relationships from the
>> explicitly cached objects (see "testWeakReferences_ToOneRelationships" test
>> in [2]).
>>
>> > This is potentially an issue with both the Shared and Local Query
>> Caches.
>>
>> Shared cache stores snapshots that do not have references to the
>> ObjectContext. So it can't be an issue there.
>>
>> > Also, because the cached objects still reference the ObjectContext, it
>> > appears that the context will not be garbage collected.
>>
>> Correct.
>>
>> > Possible Solutions:
>> >
>> > One solution is to null out the ObjectContext on any objects that are
>> > inserted into the Query Cache. This solves both problems above, and it
>> > seems logical since when the objects are retrieved from the cache they
>> will
>> > be placed into a new context anyway. This should work, but the
>> > implementation has been tricky.
>>
>> This will not work at the framework level, as Cayenne doesn't know who
>> else beside the cache references cached objects. So you'd be breaking the
>> state of the object while it is still exposed to the world.
>>
>
> My approach was to:
> 1) make a localObject copy of the object first
> 2) null the ObjectContext on the copy
> 3) store it in the cache
>
> That way only the cached object is affected. Based on some VERY simple
> tests this seems to work, but I haven't tried with any real data or real
> apps yet.
>
>
>> > What Now?
>> >
>> > I've taken a first stab at implementing both of these solutions and have
>> > had some concerns raised about each of them [1] [2]. I'd like to
>> implement
>> > something to fix this problem directly in Cayenne rather than fixing it
>> > only for myself. I'd love to hear any feedback or suggestions on this
>> > before I go further down what might be the wrong road.
>>
>> I happen to agree the the cache model needs improvement to be more
>> transparent and easy to use. But I'd like to establish some common ground
>> in understanding the problems before we can discuss solutions. So to
>> reiterate what I said in reply to items 1 and 2:
>>
>> * Is there a difference in terminology of what a "leak" is? Do you view
>> it as just an overuse of memory but with a fixed upper boundary, or do you
>> see it as a constant expansion that eventually leads to the app running out
>> of memory no matter how much memory it had?
>>
>
> The constant expansion is the real problem. The overuse of memory is
> undesirable, but not a real problem. What makes it overuse instead of
> constant expansion depends on how the cache is configured (as you said). I
> would prefer to leave my cache groups unbounded for groups I'm using Local
> Caching for, just so that I can have consistent (unsurprising) behavior
> (always hits the cache or not for a given code path). Additionally, like
> this original message to the user's list points out, new or unaware users
> can accidentally cause constant expansion without knowing it - and I think
> we can do better there.
>
>
>> * Objects are stored in the context via weak references, so if they are
>> not directly cached, they can be GC'd, even if their context is retained.
>> Do you observe a behavior that contradicts this?
>>
>
> My projects are set up to use soft references using "
> cayenne.server.object_retain_strategy=soft". I wouldn't expect this to
> behave differently than weak under memory pressure, but my unit test with
> soft is not releasing this memory. I'll try running the test with "weak"
> and see if that changes it. And I'll look at your project (thanks for
> that).
>

To clarify for future readers, we're talking about objects being retained
by this reference path:
QueryCache -> cached_object -> ObjectContext ->
other_objects_that_weren't_cached

Sorry, I was wrong about the cause here. The persistent objects ARE
released by the context's objectStore (whether using weak or soft). But the
context is still retaining a lot of extra memory. I'm having a hard time
determining the specific cause. It might be the dataRowCache?

My test fetch 100 objects and just throws them away (not cached). Then it
fetches a single object (from a different entity) and caches it. It repeats
this over and over with a new context each time. Since the 100 objects
aren't used in any way I wouldn't expect them to be retained in memory, but
the unit test ends with OutOfMemory much sooner (around 100x sooner) when
that fetch for 100 objects is present. So the single cached object is
retaining the ObjectContext, which in turn is retaining a bunch of memory
for things I'm no longer using and don't want cached.


> * Shared cache can not be the cause of ObjectContext retention. Again, do
>> you observe a behavior that contradicts this?
>>
>
> No, I haven't had issues with the shared cache and really only been
> looking at the Local Cache. So that was just conjecture, sorry. The Shared
> cache clearly needs expiration or bounding (or both) to be configured by
> the user so it isn't subject to these same considerations anyway.
>
>
>> Andrus
>>
>> [1] https://github.com/andrus/cache-test/
>> [2]
>> https://github.com/andrus/cache-test/blob/master/src/test/java/com/foo/ApplicationTest.java
>>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message