Return-Path: Delivered-To: apmail-openjpa-users-archive@minotaur.apache.org Received: (qmail 75012 invoked from network); 14 Jan 2010 19:57:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Jan 2010 19:57:32 -0000 Received: (qmail 22275 invoked by uid 500); 14 Jan 2010 19:57:32 -0000 Delivered-To: apmail-openjpa-users-archive@openjpa.apache.org Received: (qmail 22219 invoked by uid 500); 14 Jan 2010 19:57:32 -0000 Mailing-List: contact users-help@openjpa.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@openjpa.apache.org Delivered-To: mailing list users@openjpa.apache.org Received: (qmail 22209 invoked by uid 99); 14 Jan 2010 19:57:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 19:57:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kwsutter@gmail.com designates 209.85.218.209 as permitted sender) Received: from [209.85.218.209] (HELO mail-bw0-f209.google.com) (209.85.218.209) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 19:57:23 +0000 Received: by bwz1 with SMTP id 1so19797bwz.33 for ; Thu, 14 Jan 2010 11:57:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=4nSYBId+UErdC5He5HgwDvdrjr6QsDMHLBLqS69rDKk=; b=F/4YNi+K/I7Oqghd9+TUF+ze18c8bc2w3A+aMt3Q4srVdd8Vqfj3HOf7DGUw1oVFXt ZsMOynOTij7TosZDFmzSGLgM7Y+9dsxlvERd9J+DvOsXAr1Xcj/ibX/BLlb6s7i0mM6i 65vz4RtALjg6HOJKGSxkt6ySJ6rpY7octDLvY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=gHTb8ERnKNdLo3afifcLxzoZMALBW++bWyn8kQjbJq0aykKqwLZmLXds9RpFEWQBCl 0s7MaGJjRZrsM1R/pxB00oMngRIuWgJxm8Q4Y0N42MbyNlwaq097QEU+KElZ4yYv95Uo M/I852nHTB5XdQ/0nQlxlNlK/EM/m1eQtcukU= MIME-Version: 1.0 Received: by 10.204.14.82 with SMTP id f18mr713391bka.204.1263499022383; Thu, 14 Jan 2010 11:57:02 -0800 (PST) In-Reply-To: <89c0c52c1001141146v9a83d8cl1edcf190716407f8@mail.gmail.com> References: <4B4E0E47.8040005@gmail.com> <89c0c52c1001131247t17370d5chc64cbb1df392940c@mail.gmail.com> <4B4E3CC6.2090007@gmail.com> <4B4E4877.1080504@gmail.com> <89c0c52c1001131525u228b943akfc621e4e580b0d59@mail.gmail.com> <4B4E8F05.1090303@gmail.com> <89c0c52c1001140716t18badcc8n66b3d7dfea4ffe21@mail.gmail.com> <4B4F54B0.2070208@gmail.com> <89c0c52c1001141146v9a83d8cl1edcf190716407f8@mail.gmail.com> Date: Thu, 14 Jan 2010 13:57:02 -0600 Message-ID: <89c0c52c1001141157s525d41je4306f41d961952f@mail.gmail.com> Subject: Re: memory leak? - simple question From: Kevin Sutter To: users@openjpa.apache.org Content-Type: multipart/alternative; boundary=00032555ac02673d20047d254a57 X-Virus-Checked: Checked by ClamAV on apache.org --00032555ac02673d20047d254a57 Content-Type: text/plain; charset=ISO-8859-1 Hey, this wasn't so difficult... :-) Maybe I can still learn new things... I have Tomcat up and running and I can get your app running. On Thu, Jan 14, 2010 at 1:46 PM, Kevin Sutter wrote: > Hi Kurt, > These instructions, of course, assume that I know something about Tomcat. > This would not be a good assumption... :-) If you really want me to see > this in action, I will need more information on how to start Tomcat and how > to load applications. In the mean time, a trace log of OpenJPA would be an > excellent start. > > Thanks! > Kevin > > > On Thu, Jan 14, 2010 at 11:30 AM, Kurt T Stam wrote: > >> Hi Kevin, >> >> I tried adding the javaagent stuff, and it still ends up leaking. I >> extracted the code into a tiny >> war with just one servlet which persist the authToken. The build creates a >> Tomcat distro backed by a >> derby database. You can download it from: >> >> http://people.apache.org/~kstam/leak.tgz >> >> You can build it with: mvn install, you can find the final tomcat in >> tomcat/target/tomcat-1.0.SNAPSHOT.zip >> >> or if you don't want to build it: >> http://people.apache.org/~kstam/tomcat-1.0.SNAPSHOT.zip >> >> After you bring up tomcat it persistent an authToken when you hit >> http://localhost:8080/leak/test >> >> In the profiler I see an org.apache.juddi.AuthToken left behind for each >> request. >> >> Hopefully this little example will pinpoint what's going on. >> >> Thx Kev for sticking with me :) >> >> --Kurt >> >> >> >> Kevin Sutter wrote: >> >>> Hi Kurt, >>> Since you have the environment to reproduce this, could you run with the >>> OpenJPA trace enabled? I'm not even sure what I'm looking for at this >>> point. :-) But, maybe there's some tracing that would point us in the >>> right direction. >>> >>> >>> >>> We have been always been on the watch for memory leaks, so this one is >>> puzzling. You did mention blocked threads as well. At this point, you >>> are >>> probably hitting blocked threads due to the GC and Memory problems. But, >>> we >>> did recently resolve some locking situations [1] in our >>> MetaDataRepository >>> that could affect scaling (under extreme load). This was resolved in the >>> 1.2.x SNAPSHOT drivers (Mike is currently pushing for a 1.2.2 release, so >>> that should be coming soon). >>> >>> Does your WebService version of the problem use EntityManager >>> transactions, >>> or is the transaction service coming from Tomcat? And, does your >>> WebService >>> version also do the em.clear() and em.close()? Basically, are you using >>> application-managed or container-managed persistence while running as a >>> WebService? >>> >>> I'm very interested in helping to resolve this issue. Like I mentioned, >>> we >>> do a lot of memory leakage and scalability testing of the OpenJPA >>> solution >>> and we're not aware of anything. Although there is always the >>> possibility >>> that something was resolved in the 1.2.x branch after the 1.2.1 release >>> that >>> I'm not remembering... >>> >>> Thanks for your help, >>> Kevin >>> >>> [1] https://issues.apache.org/jira/browse/OPENJPA-250 >>> >>> On Wed, Jan 13, 2010 at 9:27 PM, Kurt T Stam >>> wrote: >>> >>> >>> >>>> Thanks Kevin, >>>> >>>> We're enhancing at build time: >>>> >>>> >>>> http://svn.apache.org/repos/asf/webservices/juddi/trunk/juddi-core/pom.xml >>>> >>>> Yeah we've been running load tests and things are nice and stable with >>>> Hibernate but with Openjpa we see increasing memory use, blocking >>>> threads >>>> and then an OOM. http://issues.apache.org/jira/browse/JUDDI-267. Our >>>> preference would be to ship with openjpa by default; but our build >>>> supports >>>> both hibernate and openjpa. >>>> >>>> And yes we use openjpa 1.2.1 (latest stable version). >>>> >>>> --Kurt >>>> >>>> >>>> Kevin Sutter wrote: >>>> >>>> >>>> >>>>> Interesting detective work, Kurt. Thanks. >>>>> >>>>> Why the WebService version of the app would behave differently as far >>>>> as >>>>> GC >>>>> is concerned is a mystery. And, you said that plugging in Hibernate >>>>> into >>>>> this scenario, everything works okay? Very confusing. >>>>> >>>>> How are you performing the Entity enhancement processing? Are you >>>>> pre-enhancing via your build process? Or, are you using the -javaagent >>>>> mechanism? Or, are you falling back to the subclassing support within >>>>> OpenJPA? (See [1] for more information on these questions in case they >>>>> don't make sense.) >>>>> >>>>> This would be one area that is different between Hibernate and OpenJPA >>>>> -- >>>>> enhancement processing. >>>>> >>>>> In the Tomcat environment, you may be falling back to the subclassing >>>>> support (which we do not recommend) and hitting a memory leak with >>>>> that. >>>>> >>>>> You said OpenJPA 1.2.x, right? >>>>> >>>>> Just a couple of thoughts on the subject... >>>>> Kevin >>>>> >>>>> [1] >>>>> >>>>> http://webspherepersistence.blogspot.com/2009/02/openjpa-enhancement.html >>>>> >>>>> >>>>> >>>>> On Wed, Jan 13, 2010 at 4:25 PM, Kurt T Stam >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> The same code executed straight from a java client (inVM) shows no >>>>>> memory >>>>>> leak. >>>>>> >>>>>> So is the fact that it is WebService significant then? What else can >>>>>> be >>>>>> different? I think one thread remains up, and somehow this causes >>>>>> openjpa >>>>>> not being able to clean up after itself. What can I do to debug this >>>>>> more? I >>>>>> can actually see in the profiler that the objects are allocated by the >>>>>> WebService, but why aren't they cleaned up? >>>>>> >>>>>> Thx, >>>>>> >>>>>> >>>>>> --Kurt >>>>>> >>>>>> >>>>>> Kurt T Stam wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Thanks Kevin, thanks for your response. >>>>>>> >>>>>>> I just replaced the static call by: >>>>>>> >>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>> apiAuthToken.setAuthInfo(modelAuthToken.getAuthToken()); >>>>>>> //MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>> apiAuthToken); >>>>>>> >>>>>>> which did not make a difference. >>>>>>> >>>>>>> I'm wondering if the fact that my class is a webservice makes a >>>>>>> difference. I'll try extracting it into >>>>>>> a regular class with a main method and profile that. At least I know >>>>>>> that >>>>>>> I didn't forget something >>>>>>> completely obvious.. >>>>>>> >>>>>>> --Kurt >>>>>>> >>>>>>> Kevin Sutter wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Kurt, >>>>>>>> I agree that this is very common usage of the JPA programming model. >>>>>>>> And, >>>>>>>> we are not aware of any memory leaks. About the only thing that >>>>>>>> jumps >>>>>>>> out >>>>>>>> at me is the following two lines: >>>>>>>> >>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>> apiAuthToken); >>>>>>>> >>>>>>>> What do these do? Can you comment these out and see if the memory >>>>>>>> leak >>>>>>>> still exists? Since you are passing the modelAuthToken into this >>>>>>>> method, >>>>>>>> I >>>>>>>> don't know what it's doing with the reference and could it be >>>>>>>> holding >>>>>>>> onto >>>>>>>> something to prevent the GC from cleaning up? >>>>>>>> >>>>>>>> The rest of your example seems very straight forward with creating >>>>>>>> and >>>>>>>> persisting objects. >>>>>>>> >>>>>>>> Kevin >>>>>>>> >>>>>>>> On Wed, Jan 13, 2010 at 2:09 PM, Rick Curtis >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> If you change the 1000 to something like 1000000... does your >>>>>>>>> application >>>>>>>>> go >>>>>>>>> OOM? Are you running in a JSE environment? What is >>>>>>>>> PersistenceManager? >>>>>>>>> >>>>>>>>> On Wed, Jan 13, 2010 at 2:05 PM, Kurt T Stam >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> BTW I'm running with the cache off >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> (that turns it off right?) >>>>>>>>>> >>>>>>>>>> --Kurt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Kurt T Stam wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi guys, >>>>>>>>>>> >>>>>>>>>>> [DESCRIPTION] The code below inserts a 1000 records in the >>>>>>>>>>> database. >>>>>>>>>>> >>>>>>>>>>> for (int i=1; i<1000; i++) { >>>>>>>>>>> EntityManager em = PersistenceManager.getEntityManager(); >>>>>>>>>>> EntityTransaction tx = em.getTransaction(); >>>>>>>>>>> try { >>>>>>>>>>> tx.begin(); >>>>>>>>>>> // Generate auth token and store it! >>>>>>>>>>> String authInfo = AUTH_TOKEN_PREFIX + >>>>>>>>>>> UUID.randomUUID(); >>>>>>>>>>> org.apache.juddi.model.AuthToken modelAuthToken = new >>>>>>>>>>> org.apache.juddi.model.AuthToken(); >>>>>>>>>>> if (authInfo != null) { >>>>>>>>>>> modelAuthToken.setAuthToken(authInfo); >>>>>>>>>>> modelAuthToken.setCreated(new Date()); >>>>>>>>>>> modelAuthToken.setLastUsed(new Date()); >>>>>>>>>>> modelAuthToken.setAuthorizedName(publisherId); >>>>>>>>>>> modelAuthToken.setNumberOfUses(0); >>>>>>>>>>> modelAuthToken.setTokenState(AUTHTOKEN_ACTIVE); >>>>>>>>>>> em.persist(modelAuthToken); >>>>>>>>>>> } >>>>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>>>> apiAuthToken); >>>>>>>>>>> tx.commit(); >>>>>>>>>>> } finally { >>>>>>>>>>> if (tx.isActive()) { >>>>>>>>>>> tx.rollback(); >>>>>>>>>>> } >>>>>>>>>>> em.clear(); >>>>>>>>>>> em.close(); >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [ISSUE] >>>>>>>>>>> After it leaving this code I end up with a 1000 >>>>>>>>>>> org.apache.juddi.model.AuthToken objects in memory. I've been >>>>>>>>>>> using >>>>>>>>>>> the >>>>>>>>>>> profiler, and these objects cannot be garbage collected. >>>>>>>>>>> >>>>>>>>>>> This seems to be pretty the most common use case of using an >>>>>>>>>>> OR-mapping >>>>>>>>>>> tool, so I find it hard to believe openjpa has a memory leak >>>>>>>>>>> here. >>>>>>>>>>> Does >>>>>>>>>>> anyone see what I'm doing wrong? Or can someone point me to an >>>>>>>>>>> example >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> that >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> does not exhibit this behavior? BTW same code using hibernate does >>>>>>>>>> not >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> accumulate these objects. >>>>>>>>>>> >>>>>>>>>>> We're using openjpa 1.2.1. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thx, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Kurt >>>>>>>>>>> >>>>>>>>>>> Apache jUDDI. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks, >>>>>>>>> Rick >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >>> >> >> > --00032555ac02673d20047d254a57--