From dev-return-11146-archive-asf-public=cust-asf.ponee.io@cayenne.apache.org Tue Dec 10 18:39:35 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6D83A180630 for ; Tue, 10 Dec 2019 19:39:35 +0100 (CET) Received: (qmail 51141 invoked by uid 500); 10 Dec 2019 18:39:34 -0000 Mailing-List: contact dev-help@cayenne.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cayenne.apache.org Delivered-To: mailing list dev@cayenne.apache.org Received: (qmail 51129 invoked by uid 99); 10 Dec 2019 18:39:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Dec 2019 18:39:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D70BD181377 for ; Tue, 10 Dec 2019 18:39:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id OK-Ea1PmJaBR for ; Tue, 10 Dec 2019 18:39:30 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.173; helo=mail-lj1-f173.google.com; envelope-from=johnthuss@gmail.com; receiver= Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 6F9FDBC509 for ; Tue, 10 Dec 2019 18:39:29 +0000 (UTC) Received: by mail-lj1-f173.google.com with SMTP id z17so21003675ljk.13 for ; Tue, 10 Dec 2019 10:39:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=/XGMw8Mp1+4KdxbXq392isVx09nbGa7YFBT/1QIPL6A=; b=PJ6Buc080Ng6LsF4nv6Hcp0qxNk5alXZZzYgSinSElxsPLlKInpaHqYWLCxMTkTmv0 Der49NJP13fPLx0VB363v6QgKm7oXeJuSnh1x16kj/WvfUZpuiakUF5sWoWKLguVZ2Bh mxXE/ozeNpRyCOVPpGq05L+dw+edlfq3V97myhB5wbldiYW9Y867JmZQoH4VfinTqTgT WoT9jBpWwzv29EaYFNwOhAECmPwSOGn2eCXtqtLjcHya0LxBXLMmnzjuUrvjzK21n0H8 T8eCGg8HzXFi6k817L18vfE3lzbQxZScXIgxaxQm+QVvjz+LQZV3yPPabDPn5VQqpbgA dWYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=/XGMw8Mp1+4KdxbXq392isVx09nbGa7YFBT/1QIPL6A=; b=dezhHfpptFXVh/iijQjMH523NT5iDIVebzn5Od1u4lR4AI1QbrlSqqskOeFOyLvEcm /wFLnHJDMtPF8eWHYUSM2ahQfrs/s58zGCXaqifsNg8hEHuS8vtB+eF3rD86iU4HrSsJ Jhc5yAFVLTqOnrrUGs899fv7Y9j1PfChLG6J+D28hD6L6TukEIZBt5bD3Zllaovu4U15 DgA/7z4Zi1UuHjHgu4pqWRBcOT4EjCEdAinVUcmApghYbTEgE+t+LNQLp68hBTVUBfIW jJYwhvLIwG2KtLeoOi477wcnwfBEjE4gjHcMaaoowUKgP/+pMr2I0X2nVH4jC3S1Sglg 5xUA== X-Gm-Message-State: APjAAAWwuZEbyfHcNPRd1Cny5X0xlKwMmbpNLsnEYICe0Dymn4XN+jrL hiMCL+jFxrBm5IZrpa8BB0DyPyGK/hO9Txs5SVAAH5Qy X-Google-Smtp-Source: APXvYqxCFwV/Axhrus25r0y4DUffUiORLCuICTqopjIQJkRx8puOnWXvRTSvVxMU6n5Ccnfg6UdpH1/RoKvitGX9U+0= X-Received: by 2002:a2e:8916:: with SMTP id d22mr21854345lji.19.1576003167883; Tue, 10 Dec 2019 10:39:27 -0800 (PST) MIME-Version: 1.0 References: <98905A1E-DA4E-4AE3-9A3C-531FF610FABB@objectstyle.org> In-Reply-To: From: John Huss Date: Tue, 10 Dec 2019 12:39:11 -0600 Message-ID: Subject: Re: Watch out for memory leaks with EhCache To: dev@cayenne.apache.org Content-Type: multipart/alternative; boundary="000000000000fc964d05995dd4fb" --000000000000fc964d05995dd4fb Content-Type: text/plain; charset="UTF-8" On Tue, Dec 10, 2019 at 9:51 AM John Huss wrote: > > Thanks for the feedback - I appreciate it. > > On Sun, Dec 8, 2019 at 3:17 AM Andrus Adamchik > wrote: > >> Hi John, >> >> Thanks for the analysis. You've put a significant effort in this issue, >> but I still feel like we have differences in understanding of the causes >> (and hence the remedies). Let me try to clarify how I see it. >> >> > On Dec 7, 2019, at 12:09 AM, John Huss wrote: >> > >> > 1) The lifetime of entries in the Local Query Cache exceeds their >> > availability, which is the life of their ObjectContext. Any cache that >> is >> > not expiring entries (or limiting them) will just leak this memory. >> >> A properly configured cache will "overcommit" memory, but not "leak" >> (i.e. it won't expand indefinitely). Misconfigured cache was the cause of >> the the issue in the parent message. Once I added upper limits on the >> number of entries to all cache groups, the leak disappeared. >> >> > 2) Cached query results will retain the ObjectContext they were fetched >> > into, which in turn may retain a much larger number of objects than >> > intended. For example. If you use a single ObjectContext to fetch 1 >> million >> > uncached objects along with 1 cached object, you will retain 1 million >> and >> > 1 objects in memory rather than just 1. >> >> This doesn't look right. Objects are stored via weak references by the >> OC, so if there are no other references to them, they will be GC'd even if >> the context is still around. I wrote a self-contained test project [1] that >> proves this assumption [2]. If you see unexpected extra objects cached, >> they are likely retained via prefetched / faulted relationships from the >> explicitly cached objects (see "testWeakReferences_ToOneRelationships" test >> in [2]). >> >> > This is potentially an issue with both the Shared and Local Query >> Caches. >> >> Shared cache stores snapshots that do not have references to the >> ObjectContext. So it can't be an issue there. >> >> > Also, because the cached objects still reference the ObjectContext, it >> > appears that the context will not be garbage collected. >> >> Correct. >> >> > Possible Solutions: >> > >> > One solution is to null out the ObjectContext on any objects that are >> > inserted into the Query Cache. This solves both problems above, and it >> > seems logical since when the objects are retrieved from the cache they >> will >> > be placed into a new context anyway. This should work, but the >> > implementation has been tricky. >> >> This will not work at the framework level, as Cayenne doesn't know who >> else beside the cache references cached objects. So you'd be breaking the >> state of the object while it is still exposed to the world. >> > > My approach was to: > 1) make a localObject copy of the object first > 2) null the ObjectContext on the copy > 3) store it in the cache > > That way only the cached object is affected. Based on some VERY simple > tests this seems to work, but I haven't tried with any real data or real > apps yet. > > >> > What Now? >> > >> > I've taken a first stab at implementing both of these solutions and have >> > had some concerns raised about each of them [1] [2]. I'd like to >> implement >> > something to fix this problem directly in Cayenne rather than fixing it >> > only for myself. I'd love to hear any feedback or suggestions on this >> > before I go further down what might be the wrong road. >> >> I happen to agree the the cache model needs improvement to be more >> transparent and easy to use. But I'd like to establish some common ground >> in understanding the problems before we can discuss solutions. So to >> reiterate what I said in reply to items 1 and 2: >> >> * Is there a difference in terminology of what a "leak" is? Do you view >> it as just an overuse of memory but with a fixed upper boundary, or do you >> see it as a constant expansion that eventually leads to the app running out >> of memory no matter how much memory it had? >> > > The constant expansion is the real problem. The overuse of memory is > undesirable, but not a real problem. What makes it overuse instead of > constant expansion depends on how the cache is configured (as you said). I > would prefer to leave my cache groups unbounded for groups I'm using Local > Caching for, just so that I can have consistent (unsurprising) behavior > (always hits the cache or not for a given code path). Additionally, like > this original message to the user's list points out, new or unaware users > can accidentally cause constant expansion without knowing it - and I think > we can do better there. > > >> * Objects are stored in the context via weak references, so if they are >> not directly cached, they can be GC'd, even if their context is retained. >> Do you observe a behavior that contradicts this? >> > > My projects are set up to use soft references using " > cayenne.server.object_retain_strategy=soft". I wouldn't expect this to > behave differently than weak under memory pressure, but my unit test with > soft is not releasing this memory. I'll try running the test with "weak" > and see if that changes it. And I'll look at your project (thanks for > that). > To clarify for future readers, we're talking about objects being retained by this reference path: QueryCache -> cached_object -> ObjectContext -> other_objects_that_weren't_cached Sorry, I was wrong about the cause here. The persistent objects ARE released by the context's objectStore (whether using weak or soft). But the context is still retaining a lot of extra memory. I'm having a hard time determining the specific cause. It might be the dataRowCache? My test fetch 100 objects and just throws them away (not cached). Then it fetches a single object (from a different entity) and caches it. It repeats this over and over with a new context each time. Since the 100 objects aren't used in any way I wouldn't expect them to be retained in memory, but the unit test ends with OutOfMemory much sooner (around 100x sooner) when that fetch for 100 objects is present. So the single cached object is retaining the ObjectContext, which in turn is retaining a bunch of memory for things I'm no longer using and don't want cached. > * Shared cache can not be the cause of ObjectContext retention. Again, do >> you observe a behavior that contradicts this? >> > > No, I haven't had issues with the shared cache and really only been > looking at the Local Cache. So that was just conjecture, sorry. The Shared > cache clearly needs expiration or bounding (or both) to be configured by > the user so it isn't subject to these same considerations anyway. > > >> Andrus >> >> [1] https://github.com/andrus/cache-test/ >> [2] >> https://github.com/andrus/cache-test/blob/master/src/test/java/com/foo/ApplicationTest.java >> >> >> --000000000000fc964d05995dd4fb--