Mailing-List: contact user-help@cayenne.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cayenne.apache.org
Received-SPF: pass (nike.apache.org: domain of mkienenb@gmail.com designates
 209.85.212.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=oYKIwu7QbPSRmhQ4E0/AoxMqjI6EA6nYLraqXo4//05QWMzOBrMUaEtNFpcJapxTM4
         7/wDbTU6fPN4sMubumfSIEvM6esOSQmEihD1TLy3TlTa9V9d3e76d4lsrbRhEcu8C3nO
         VsV44ebZVo/ROkFU3INR7qDVfcy9VqKUtA5Qw=
MIME-Version: 1.0
In-Reply-To: <9031107A-B21C-433F-A1EF-BABCAEEC8E14@earthlink.net>
References: <99572F4D-847C-4E58-A125-597528AA0BDA@earthlink.net>
	<8f985b960909161125p6161b189l15f57fe27a5f776e@mail.gmail.com>
	<19F8240B-FF49-4292-A797-A6FFFE336AC5@earthlink.net>
 <8f985b960909161210k6ab17b7dv417e3864652d4eb@mail.gmail.com>
	<9031107A-B21C-433F-A1EF-BABCAEEC8E14@earthlink.net>
From: Mike Kienenberger <mkienenb@gmail.com>
Date: Wed, 16 Sep 2009 15:37:17 -0400
Message-ID: <8f985b960909161237t1a1db903p63992dab2d826128@mail.gmail.com>
Subject: Re: Memory Management using Tomcat
To: user@cayenne.apache.org
Content-Type: multipart/mixed; boundary=0016e64602a610b4c70473b7083e

--0016e64602a610b4c70473b7083e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think there's a default filter provided by Cayenne you can specify
in your config file, but it really comes down to something as simple
as this to make it per request:

    public void doFilter(ServletRequest servletRequest,
ServletResponse servletResponse, FilterChain chain)
          throws IOException, ServletException
    {
        // set base context on thread

        chain.doFilter(servletRequest, servletResponse);

        // remove base context from thread
    }

Attaching a simple one I wrote a long time ago that perserves the
DataContext across a session (but insures it's in a clean state at the
end of each request).


On Wed, Sep 16, 2009 at 3:29 PM, Joe Baldwin <jfbaldwin@earthlink.net> wrot=
e:
> Mike,
>
> RE BaseContext.getThreadObjectContext()
>
> Of course, this could be my problem. I was using DataContext until 3.0 th=
en
> converted over to BaseContext.
>
>
>> 2.) =A0BaseContext.getThreadObjectContext() just tells how you're
>> getting a context. =A0It doesn't tell how it's managed. =A0 Do you have =
a
>> servlet filter that creates a new ObjectContext at the start of a
>> request and then clears it out at the end? =A0 Or does it do this per
>> session?
>
> I do not know how it is being initialized in the webapp (in my experiment=
al
> non-web apps I explicitly intitialize it, but in the webapp it is already
> initialized).
>
>
>> =A0I've never looked at the method, but it might default to
>> creating one permanent data context per thread if you don't do
>> anything else (like set up a servlet filter). =A0 That could be part of
>> the problem.
>
>
> That sounds plausible. =A0I could not find an example of how to initializ=
e
> this and manage it as you suggest in a web app. =A0I have not created a
> servlet filter but have follow the Cayenne docs for configuration of the
> web.xml.
>
> If this is insufficient then I agree, this could be the problem.
> =A0Unfortunately, I have not found docs on how to accomplish what you are
> recommending.
>
> Joe
>
>
>
>
> On Sep 16, 2009, at 3:10 PM, Mike Kienenberger wrote:
>
>> 1.) =A0128 still seems small to me. =A0 I don't think I run anything at
>> less than 256.
>> On the other hand, We have an app with 1000s of customers that uses
>> 512Mb, I think. =A0 So 1500 seems excessive.
>>
>> 2.) =A0BaseContext.getThreadObjectContext() just tells how you're
>> getting a context. =A0It doesn't tell how it's managed. =A0 Do you have =
a
>> servlet filter that creates a new ObjectContext at the start of a
>> request and then clears it out at the end? =A0 Or does it do this per
>> session? =A0 I've never looked at the method, but it might default to
>> creating one permanent data context per thread if you don't do
>> anything else (like set up a servlet filter). =A0 That could be part of
>> the problem.
>>
>> On Wed, Sep 16, 2009 at 2:49 PM, Joe Baldwin <jfbaldwin@earthlink.net>
>> wrote:
>>>
>>> Caveat: Apparently I am not as well. :)
>>>
>>> 1.) I looked at the 65M issue. =A0On my development box (OSX) I set it =
to
>>> -Xms128m -Xmx128m (basically arbitrary). =A0So when I went to the remot=
e
>>> hosting company, I purchase a similar amount.
>>> =A0 =A0 =A0 a. However, we are doing *very* little work (lots of produc=
t
>>> fetches
>>> and only a few product inserts and updates) and it runs out of memory
>>> *very*
>>> fast which I *assume* means it is my code, but I don't know.
>>> =A0 =A0 =A0 b. I am doing research and here is a "web recommendation" (=
for all
>>> that is worth)
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 Whenever possible, Unidata recommends
>>> =A0 =A0 =A0 -Xmx1500m for 32-bit systems, and -Xmx2048m --Xmx4096m for =
64-bit
>>> systems.
>>> =A0 =A0 =A0 c. Since the host is 64-bit, I am wondering whether my assu=
mptions
>>> may be off for 64-bit systems.
>>>
>>> 2. DataContext: Sorry, but I am getting confused on this one. =A0I am u=
sing
>>> BaseContext.getThreadObjectContext() based on recommendations (I
>>> converted
>>> all the old DataContext refs to BaseContext, but I don't really
>>> understand
>>> it from reading the docs) and am *not* releasing it at the end of
>>> session.
>>> =A0Not quite sure of how to do this properly.
>>>
>>> 3. Don't know how to set the cache to retain N number of objects. I
>>> experimented with
>>> =A0 =A0 =A0 query.setPageSize(RowsPerPage);
>>> =A0 =A0 =A0 query.setCacheStrategy(QueryCacheStrategy.SHARED_CACHE);
>>> =A0 =A0 =A0 query.setCacheGroups("product", "ProductList");
>>>
>>> This seemed to help quite a bit but I still eventually ran out of memor=
y.
>>> =A0I
>>> recently removed *all* the SHARED_CACHE and it ran out of memory very
>>> fast.
>>>
>>> Thanks for your input
>>> Joe
>>>
>>>
>>>
>>> On Sep 16, 2009, at 2:25 PM, Mike Kienenberger wrote:
>>>
>>>> Caveat: I'm not really an expert on Cayenne memory management.
>>>>
>>>> 1) Are you allocating enough heap memory to the app server to start
>>>> with? =A0 I don't know what the default is these days, but in the old
>>>> days, an application by default only gets 64Mb of memory -- that's
>>>> pretty small.
>>>>
>>>> 2) Are you using a new DataContext per request? =A0Or at least per
>>>> session?
>>>>
>>>> 3) I seem to remember that the cache strategy is configurable. =A0Have
>>>> you configured a cache that only retains N number of objects for a
>>>> suitable value of N?
>>>>
>>>> On Wed, Sep 16, 2009 at 2:11 PM, Joe Baldwin <jfbaldwin@earthlink.net>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have asked this question a number of ways but I still have a very
>>>>> serious
>>>>> problem with Cayenne-specific memory management configuration
>>>>> associated
>>>>> with Tomcat.
>>>>>
>>>>> The problem with debugging is that given that I have very little
>>>>> visibility
>>>>> into the Cayenne memory management it is extremely difficult to debug
>>>>> this
>>>>> using conventional strategies. =A0Also I am not requesting anything f=
rom
>>>>> the
>>>>> system that is terribly exceptional so I am attempting to use default
>>>>> settings as much as possible.
>>>>>
>>>>> My strategy is to ask the experts for a Cayenne configuration and
>>>>> standard
>>>>> memory management steps I should take to conform to the new Cayenne
>>>>> memory
>>>>> management design intentions.
>>>>>
>>>>> Problem:
>>>>> 1. I have essentially a webstore, three tier design with Tomcat,
>>>>> Cayenne
>>>>> and
>>>>> MySQL.
>>>>> 2. When after only a few queries of products, tomcat freezes up and
>>>>> reports
>>>>> out of memory errors.
>>>>>
>>>>> I have attempted to configure the caching strategy ask best as I can
>>>>> understand from the docks but this only gets me a few more hours of
>>>>> usage
>>>>> before the out of memory errors. =A0(I tried the SHARED_CACHE). The
>>>>> NO_CACHE
>>>>> strategy is worse.
>>>>>
>>>>> I would appreciate a set of steps (aka a primer) that should handle a
>>>>> website with a lot of fetches of hundreds of data objects (i.e.
>>>>> products)
>>>>> and very few updates.
>>>>>
>>>>> Note: My gut feeling is that I am not properly managing the data obje=
ct
>>>>> array properly and it is leaking memory.
>>>>>
>>>>> I would appreciate any input, but I would first like to know what the
>>>>> minimum require steps are for managing at data object result set
>>>>> ArrayList
>>>>> so as to properly cache and then properly free the memory after it is
>>>>> no
>>>>> longer needed.
>>>>>
>>>>> Context: Tomcat, MySQL, Cayenne 3.0M6
>>>>>
>>>>> Thanks,
>>>>> Joe Baldwin
>>>>>
>>>>>
>>>
>>>
>
>

--0016e64602a610b4c70473b7083e
Content-Type: text/x-java; charset=US-ASCII;
 name="DataContextManagerFilter.java"
Content-Disposition: attachment; filename="DataContextManagerFilter.java"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_fzoh4z2r0

cGFja2FnZSBjb20ueHl6LnNlcnZsZXQuZmlsdGVyOwoKaW1wb3J0IGphdmEuaW8uSU9FeGNlcHRp
b247CmltcG9ydCBqYXZhLnV0aWwuSXRlcmF0b3I7CgppbXBvcnQgamF2YXguc2VydmxldC5GaWx0
ZXI7CmltcG9ydCBqYXZheC5zZXJ2bGV0LkZpbHRlckNoYWluOwppbXBvcnQgamF2YXguc2Vydmxl
dC5GaWx0ZXJDb25maWc7CmltcG9ydCBqYXZheC5zZXJ2bGV0LlNlcnZsZXRFeGNlcHRpb247Cmlt
cG9ydCBqYXZheC5zZXJ2bGV0LlNlcnZsZXRSZXF1ZXN0OwppbXBvcnQgamF2YXguc2VydmxldC5T
ZXJ2bGV0UmVzcG9uc2U7CmltcG9ydCBqYXZheC5zZXJ2bGV0Lmh0dHAuSHR0cFNlcnZsZXRSZXF1
ZXN0OwppbXBvcnQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXNzaW9uOwoKaW1wb3J0IG9yZy5v
YmplY3RzdHlsZS5jYXllbm5lLkNheWVubmVSdW50aW1lRXhjZXB0aW9uOwppbXBvcnQgb3JnLm9i
amVjdHN0eWxlLmNheWVubmUuRGF0YU9iamVjdDsKaW1wb3J0IG9yZy5vYmplY3RzdHlsZS5jYXll
bm5lLmFjY2Vzcy5EYXRhQ29udGV4dDsKaW1wb3J0IG9yZy5vYmplY3RzdHlsZS5jYXllbm5lLmNv
bmYuU2VydmxldFV0aWw7CgpwdWJsaWMgY2xhc3MgRGF0YUNvbnRleHRNYW5hZ2VyRmlsdGVyIGlt
cGxlbWVudHMgRmlsdGVyCnsKICAgIHB1YmxpYyB2b2lkIGluaXQoRmlsdGVyQ29uZmlnIGZpbHRl
ckNvbmZpZykgdGhyb3dzIFNlcnZsZXRFeGNlcHRpb24KICAgIHsKICAgIH0KCiAgICBwdWJsaWMg
dm9pZCBkb0ZpbHRlcihTZXJ2bGV0UmVxdWVzdCBzZXJ2bGV0UmVxdWVzdCwgU2VydmxldFJlc3Bv
bnNlIHNlcnZsZXRSZXNwb25zZSwgRmlsdGVyQ2hhaW4gY2hhaW4pIHRocm93cyBJT0V4Y2VwdGlv
biwgU2VydmxldEV4Y2VwdGlvbgogICAgewogICAgICAgIHJlcXVlc3RJbml0aWFsaXplZChzZXJ2
bGV0UmVxdWVzdCwgc2VydmxldFJlc3BvbnNlKTsKICAgICAgICAKICAgICAgICBjaGFpbi5kb0Zp
bHRlcihzZXJ2bGV0UmVxdWVzdCwgc2VydmxldFJlc3BvbnNlKTsKICAgICAgICAKICAgICAgICBy
ZXF1ZXN0RGVzdHJveWVkKHNlcnZsZXRSZXF1ZXN0LCBzZXJ2bGV0UmVzcG9uc2UpOwogICAgfQoK
ICAgIHB1YmxpYyB2b2lkIGRlc3Ryb3koKQogICAgewogICAgfQoKICAgIHB1YmxpYyB2b2lkIHJl
cXVlc3RJbml0aWFsaXplZChTZXJ2bGV0UmVxdWVzdCBzZXJ2bGV0UmVxdWVzdCwgU2VydmxldFJl
c3BvbnNlIHNlcnZsZXRSZXNwb25zZSkKICAgIHsKICAgICAgICBIdHRwU2Vzc2lvbiBzZXNzaW9u
ID0gKChIdHRwU2VydmxldFJlcXVlc3Qpc2VydmxldFJlcXVlc3QpLmdldFNlc3Npb24odHJ1ZSk7
CiAgICAgICAgRGF0YUNvbnRleHQgZGF0YUNvbnRleHQgPSBTZXJ2bGV0VXRpbC5nZXRTZXNzaW9u
Q29udGV4dChzZXNzaW9uKTsKICAgICAgICBEYXRhQ29udGV4dC5iaW5kVGhyZWFkRGF0YUNvbnRl
eHQoZGF0YUNvbnRleHQpOwogICAgfQoKICAgIHB1YmxpYyB2b2lkIHJlcXVlc3REZXN0cm95ZWQo
U2VydmxldFJlcXVlc3Qgc2VydmxldFJlcXVlc3QsIFNlcnZsZXRSZXNwb25zZSBzZXJ2bGV0UmVz
cG9uc2UpIHRocm93cyBJT0V4Y2VwdGlvbgogICAgewogICAgICAgIERhdGFDb250ZXh0IGRhdGFD
b250ZXh0ID0gRGF0YUNvbnRleHQuZ2V0VGhyZWFkRGF0YUNvbnRleHQoKTsKCiAgICAgICAgZmlu
YWwgU3RyaW5nQnVmZmVyIGxvZ2dlZFN0cmluZ0J1ZmZlciA9IG5ldyBTdHJpbmdCdWZmZXIoKTsK
ICAgICAgICB0cnkgewogICAgICAgICAgICByZXBvcnRVbmNvbW1pdHRlZE9iamVjdHMoZGF0YUNv
bnRleHQsIG5ldyBMb2dnZXIoKSB7CiAgICAgICAgICAgICAgICBwdWJsaWMgdm9pZCBsb2coU3Ry
aW5nIG1zZykgewogICAgICAgICAgICAgICAgICAgIGxvZ2dlZFN0cmluZ0J1ZmZlci5hcHBlbmQo
bXNnKTsKICAgICAgICAgICAgICAgICAgICBsb2dnZWRTdHJpbmdCdWZmZXIuYXBwZW5kKCJcbiIp
OwogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9KTsKICAgICAgICB9CiAgICAgICAgZmlu
YWxseQogICAgICAgIHsKICAgICAgICAgICAgRGF0YUNvbnRleHQuYmluZFRocmVhZERhdGFDb250
ZXh0KG51bGwpOwogICAgICAgICAgICBkYXRhQ29udGV4dC5yb2xsYmFja0NoYW5nZXMoKTsKICAg
ICAgICB9CiAgICB9CiAgICAKICAgIGludGVyZmFjZSBMb2dnZXIKICAgIHsKICAgICAgICBwdWJs
aWMgdm9pZCBsb2coU3RyaW5nIG1zZyk7CiAgICB9CiAgICAKICAgIHByaXZhdGUgdm9pZCByZXBv
cnRVbmNvbW1pdHRlZE9iamVjdHMoRGF0YUNvbnRleHQgZGF0YUNvbnRleHQsIExvZ2dlciBsb2dn
ZXIpCiAgICB7CiAgICAgICAgaWYgKGRhdGFDb250ZXh0Lmhhc0NoYW5nZXMoKSkKICAgICAgICB7
CiAgICAgICAgICAgIGlmICgwICE9IGRhdGFDb250ZXh0Lm5ld09iamVjdHMoKS5zaXplKCkpCiAg
ICAgICAgICAgIHsKICAgICAgICAgICAgICAgIEl0ZXJhdG9yIG9iamVjdEl0ZXJhdG9yID0gZGF0
YUNvbnRleHQubmV3T2JqZWN0cygpLml0ZXJhdG9yKCk7CiAgICAgICAgICAgICAgICB3aGlsZSAo
b2JqZWN0SXRlcmF0b3IuaGFzTmV4dCgpKQogICAgICAgICAgICAgICAgewogICAgICAgICAgICAg
ICAgICAgIERhdGFPYmplY3QgZGF0YU9iamVjdCA9IChEYXRhT2JqZWN0KSBvYmplY3RJdGVyYXRv
ci5uZXh0KCk7CiAgICAgICAgICAgICAgICAgICAgbG9nZ2VyLmxvZygiTGVmdG92ZXIgbmV3IG9i
amVjdDogIiArIGRhdGFPYmplY3QpOwogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9Cgog
ICAgICAgICAgICBpZiAoMCAhPSBkYXRhQ29udGV4dC5tb2RpZmllZE9iamVjdHMoKS5zaXplKCkp
CiAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgIEl0ZXJhdG9yIG9iamVjdEl0ZXJhdG9yID0g
ZGF0YUNvbnRleHQubW9kaWZpZWRPYmplY3RzKCkuaXRlcmF0b3IoKTsKICAgICAgICAgICAgICAg
IHdoaWxlIChvYmplY3RJdGVyYXRvci5oYXNOZXh0KCkpCiAgICAgICAgICAgICAgICB7CiAgICAg
ICAgICAgICAgICAgICAgRGF0YU9iamVjdCBkYXRhT2JqZWN0ID0gKERhdGFPYmplY3QpIG9iamVj
dEl0ZXJhdG9yLm5leHQoKTsKICAgICAgICAgICAgICAgICAgICBsb2dnZXIubG9nKCJMZWZ0b3Zl
ciBtb2RpZmllZCBvYmplY3Q6ICIgKyBkYXRhT2JqZWN0KTsKICAgICAgICAgICAgICAgIH0KICAg
ICAgICAgICAgfQoKICAgICAgICAgICAgaWYgKDAgIT0gZGF0YUNvbnRleHQuZGVsZXRlZE9iamVj
dHMoKS5zaXplKCkpCiAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgIEl0ZXJhdG9yIG9iamVj
dEl0ZXJhdG9yID0gZGF0YUNvbnRleHQuZGVsZXRlZE9iamVjdHMoKS5pdGVyYXRvcigpOwogICAg
ICAgICAgICAgICAgd2hpbGUgKG9iamVjdEl0ZXJhdG9yLmhhc05leHQoKSkKICAgICAgICAgICAg
ICAgIHsKICAgICAgICAgICAgICAgICAgICBEYXRhT2JqZWN0IGRhdGFPYmplY3QgPSAoRGF0YU9i
amVjdCkgb2JqZWN0SXRlcmF0b3IubmV4dCgpOwogICAgICAgICAgICAgICAgICAgIGxvZ2dlci5s
b2coIkxlZnRvdmVyIGRlbGV0ZWQgb2JqZWN0OiAiICsgZGF0YU9iamVjdCk7CiAgICAgICAgICAg
ICAgICB9CiAgICAgICAgICAgIH0KICAgICAgICAgICAgCiAgICAgICAgICAgIC8vIE1heWJlIHJl
bW92ZSBsYXRlcj8KICAgICAgICAgICAgdGhyb3cgbmV3IENheWVubmVSdW50aW1lRXhjZXB0aW9u
KCJEYXRhQ29udGV4dCBpcyBkaXJ0eS4iKTsKICAgICAgICB9CiAgICB9Cgp9Cg==
--0016e64602a610b4c70473b7083e--