openjpa-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Curtis <curti...@gmail.com>
Subject Re: OOM with JDBCConfigurationImpl holding 1+ GB of heap
Date Tue, 28 May 2013 16:33:34 GMT
You are reading too far into my analysis. The leak is NOT
EntityManagerFactoryImpl instances, it is a data structure
(MappingRepository) that gets created as a result of creating an EMF. The
MappingRepository is statically referenced elsewhere in our code and if you
fail to close all EntityManagerFactoryImpl instances, the MappingRepository
instances don't get cleaned up.

Thanks,
Rick


On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tucu00@gmail.com>wrote:

> Rick,
>
> Thanks for following up.
>
> After you email I've looked again at the hprof dump and the number
> of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is 1.
> I've also noticed the heap has
> 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they
> are referenced from a finalizer; thus in queue to be GC. So, it would
> argue, the leak does not come from there.
>
> I'll give it a ry using 2.2.0 to see if that makes a difference.
>
> Thanks again.
>
> Alejandro
>
>
>
> On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <curtisr7@gmail.com> wrote:
>
> > After digging into the heapdump, I'm fairly certain that somewhere in
> your
> > application EntityManagerFactories are being created, yet they are never
> > closed. I know you pointed me at your JPA service source code, but there
> > must be something else going on.  For giggles you could move to version
> > 2.2.0, where we made some changes[1] to better tolerate applications that
> > fail to close EMFs. The JIRA talks about additional diagnostics, but we
> > just changed a bit of our code to use WeakReferences in hopes of avoiding
> > these sorts of OOMs. If moving to 2.2.0 makes your problem go away,
> you'll
> > need to look elsewhere in your code for EMF creation. Hope this helps.
> >
> > Thanks,
> > Rick
> >
> >
> > [1] https://issues.apache.org/jira/browse/OPENJPA-2042
> >
> >
> > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tucu00@gmail.com
> > >wrote:
> >
> > > Hi Rick, a hprof dump is available at
> > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz
> > >
> > > Thank you very much for looking into this.
> > >
> > > Alejandro
> > >
> > >
> > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <curtisr7@gmail.com>
> wrote:
> > >
> > > > Can you post a heapdump somewhere for review?
> > > >
> > > > Thanks,
> > > > Rick
> > > >
> > > >
> > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <
> tucu00@gmail.com
> > > > >wrote:
> > > >
> > > > > Rick,
> > > > >
> > > > > We create a single EMF:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160
> > > > >
> > > > > And we use create/use/dispose all EMs using a command patter:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209
> > > > >
> > > > > Any other idea/hint?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <curtisr7@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Alejandro -
> > > > > >
> > > > > > It appears that your application is creating
> > EntityManagerFactories,
> > > > yet
> > > > > > they are never getting closed.
> > > > > >
> > > > > > Thanks,
> > > > > > Rick
> > > > > >
> > > > > >
> > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur <
> > > tucu00@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Rick, by looking at the other thread memory analyzer image,
the
> > > > issues
> > > > > > seem
> > > > > > > to be different.
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <
> curtisr7@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Alejandro --
> > > > > > > >
> > > > > > > > Is the problem that you're asking about the same as
Hovan? If
> > so,
> > > > > lets
> > > > > > > > consolidate this conversation down to a single thread.
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rick
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter <
> > > kwsutter@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Alejandro,
> > > > > > > > > It looks like something is holding onto references...
 How
> > are
> > > > you
> > > > > > > > > enhancing your classes?  At build time?  Or,
some other
> > > > mechanism?
> > > > > > > > >
> > > > > > > > > Does this memory leak happen with any of your
configured
> > > > databases?
> > > > > > >  Or,
> > > > > > > > > just select ones?
> > > > > > > > >
> > > > > > > > > This shouldn't make a difference, but specifying
your
> Entity
> > > > > classes
> > > > > > > via
> > > > > > > > > both <class> and the openjpa.MetaDataFactory
property is
> > > > overkill.
> > > > > >  The
> > > > > > > > > <class> elements are sufficient and is
the standard, so I'd
> > > > suggest
> > > > > > > > > sticking with that mechanism and removing the
> > > > > openjpa.MetaDataFactory
> > > > > > > > > property.
> > > > > > > > >
> > > > > > > > > We do considerable memory leak testing both as
a standalone
> > JPA
> > > > > > > provider
> > > > > > > > as
> > > > > > > > > well as the JPA provider for WebSphere Application
Server.
> >  We
> > > > > could
> > > > > > > have
> > > > > > > > > missed something due to your configuration requirements,
> but
> > > this
> > > > > > looks
> > > > > > > > to
> > > > > > > > > be a pretty major leak...
> > > > > > > > >
> > > > > > > > > Can you narrow down or simplify your scenario
to the point
> of
> > > > > > sharing a
> > > > > > > > > project that demonstrates the leak?  Or, maybe
there is
> > memory
> > > > data
> > > > > > > dumps
> > > > > > > > > that can be shared to help diagnose the issue?
> > > > > > > > >
> > > > > > > > > Kevin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur
<
> > > > > > tucu00@gmail.com
> > > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > And .... I'm back.....
> > > > > > > > > >
> > > > > > > > > > It seems that removing runtime enhancement
has some
> effect,
> > > but
> > > > > the
> > > > > > > > heap
> > > > > > > > > > keeps building up in the same spot.
> > > > > > > > > >
> > > > > > > > > > The following link is show where is the
memory sink
> > > > > > > > > >
> > > > > > > > > >   http://people.apache.org/~tucu/tmp/OpenJPALeak.png
> > > > > > > > > >
> > > > > > > > > > I'd appreciate any ideas/suggestions.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > Alejandro
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin
Sutter <
> > > > > kwsutter@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > It shoudn't...  But, with that flag
set as "supported",
> > we
> > > > > can't
> > > > > > be
> > > > > > > > > sure
> > > > > > > > > > > that the entities were actually enhanced
at build time.
> >  By
> > > > > > setting
> > > > > > > > > this
> > > > > > > > > > > property to "unsupported" or removing
it altogether,
> then
> > > if
> > > > an
> > > > > > > > > > unenhanced
> > > > > > > > > > > entity class is loaded, we'll log a
message and quit
> > > > > processing.
> > > > > > >  So,
> > > > > > > > > > let's
> > > > > > > > > > > go this route first and then we'll
deal with any
> > potential
> > > > > memory
> > > > > > > > leak.
> > > > > > > > > > >
> > > > > > > > > > > Thanks, Kevin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro
Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > > > >wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I'm in the process of verifying
if Kevin's suggestion
> > > fixes
> > > > > the
> > > > > > > > > > problem.
> > > > > > > > > > > >
> > > > > > > > > > > > While making the necessary changes,
I've notice we
> were
> > > > > already
> > > > > > > > > > enhancing
> > > > > > > > > > > > the classes at build time.
> > > > > > > > > > > >
> > > > > > > > > > > > If the classes were enhanced at
build time, still
> would
> > > be
> > > > we
> > > > > > see
> > > > > > > > the
> > > > > > > > > > > leak
> > > > > > > > > > > > if the runtime enhancement flag
is enabled?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM,
Alejandro Abdelnur <
> > > > > > > > tucu00@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks Kevin, will try that.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59
AM, Kevin Sutter <
> > > > > > > kwsutter@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Don't use this:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses"
> > > > > > > > > > value="supported"/>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> That is probably your
issue.  Reference this page
> > for
> > > > more
> > > > > > > > > > > information:
> > > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Good luck, Kevin
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, May 9, 2013 at
10:17 AM, Alejandro
> Abdelnur
> > <
> > > > > > > > > > tucu00@gmail.com
> > > > > > > > > > > > >> >wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Rick,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks for looking
into this, the
> persistence.xml
> > > file
> > > > > we
> > > > > > > are
> > > > > > > > > > using
> > > > > > > > > > > > is:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > The EMF is instatiated
here:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Thanks again.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Alejandro
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Thu, May 9, 2013
at 6:06 AM, Rick Curtis <
> > > > > > > > curtisr7@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Can we see
the contents of your
> persistence.xml
> > > > file?
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Wed, May
8, 2013 at 6:34 PM, Alejandro
> > > Abdelnur <
> > > > > > > > > > > > tucu00@gmail.com>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > > We are
seeing this using OpenJPA 2.1.0 in
> > Oozie.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > A JDBCConfigurationImpl
seems to be holding
> a
> > > > large
> > > > > > > amount
> > > > > > > > > of
> > > > > > > > > > > > memory
> > > > > > > > > > > > >> > via
> > > > > > > > > > > > >> > > > its dataCacheTimeout
(then listeners)
> instance
> > > > > > variable
> > > > > > > > > > keeping
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Class
                         # Objects
>  Used
> > > > Heap
> > > > > > > > > Retained
> > > > > > > > > > > > Heap
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository
 1,902
> > > 304,320
> > > > > > > > > > > 904,410,184
> > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping
      3,824
> > > > 1,223,680
> > > > > > > > > > > 105,073,624
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > >
> > > > > > >
> > -------------------------------------------------------------------
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > We are
using a single EntityManagerFactory
> and
> > > we
> > > > > are
> > > > > > > > > closing
> > > > > > > > > > > all
> > > > > > > > > > > > >> > > > EntityManager
instances correctly (via a
> > command
> > > > > > > pattern).
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > All caching
settings are default ones.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Any hint
where to look would be appreciated
> > it.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Thanks
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Alejandro
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > --
> > > > > > > > > > > > >> > > *Rick Curtis*
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Rick Curtis*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Rick Curtis*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rick Curtis*
> > > >
> > >
> >
> >
> >
> > --
> > *Rick Curtis*
> >
>



-- 
*Rick Curtis*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message