ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harvey Kim" <cash...@fastmail.fm>
Subject Re: groupby x out of memory
Date Fri, 11 Apr 2008 17:57:23 GMT
Is the app running for days/years or is the query running for
days/years?  If the query returns 24 million rows, I get out of error
memory every single time.  In theory, no query should really take days
to return but we just happen to run into this case because our DBA
forgot to put the index on the tables :-).  After the indexing, the
analyzing, paralleling, ... the query came back much much faster.  But
it also ran out of memory much much faster.  I think it's the function
of how many rows get processed on a single query - not really how long
the apps been running.  And I doubt there are many application out there
that returns 24 millions rows on a single query where each object
returned can be thousands of bytes.  Or maybe there are... If there are,
I'd love to hear if there were memory leak problems on their app.

I'm just guessing here but internally, doesn't groupBy allocate HashMap
to keep track of what it is grouping?

On Fri, 11 Apr 2008 11:42:10 -0600, "Clinton Begin"
<clinton.begin@gmail.com> said:
> I can't imagine why that would happen.  If this was a bug, I would think
> far
> more people would be experiencing it.  I have apps that have run for
> months,
> and in some cases years, without running out of memory.  GroupBy doesn't
> do
> anything special really.  It just skips duplicate rows for a parent
> class.
> 
> Try iBATIS 2.3.1.  If there is a problem, it might have to do with
> session/transaction management.  2.3.1 improves the handling of such.
> 
> Clinton
> 
> On Fri, Apr 11, 2008 at 10:10 AM, Harvey Kim <cashtag@fastmail.fm> wrote:
> 
> > If you use groupBy, you will eventually run out of memory.  Few months
> > ago, I e-mailed a sample program which showed my application doing
> > absolute nothing except issuing a ibatis query.  It was using
> > queryWithRowHandler.  Literally, all it did was call
> > "queryWithRowHandler" and my rowHanlder did absolutely nothing.
> > Literally this:
> >
> > public class IbatisDownloadThread implements Runnable, Serializable,
> > RowHandler
> > {
> >  private SqlMapClient sqlMapClient = null;
> >
> >  public void run()
> >  {
> >      try
> >      {
> >        sqlMapClient = BaseDAO.getSqlMapClient();
> >        sqlMapClient.queryWithRowHandler("getIdList", queryDto, this);
> >     }
> >  }
> >
> >  public void handleRow()
> >  {
> >  }
> > }
> >
> > The above code eventually gets "Out of Memory" error and the query is
> > using a groupBy.  When I took out the "groupBy", it went MUCH MUCH MUCH
> > longer.  But even that ran out of memory after few hours of running.
> > However, once I converted everything to a JDBC loop, "Out of Memory"
> > error went away.  When I examined the available memory at runtime, I
> > could see the available memory decreasing with every call to
> > "handleRow()".  With JDBC, I examined the heap size inside the loop and
> > it held steady for hours and finished without a problem.
> >
> > The moral of the story - you could probably get away with ibatis if you
> > take out the groupBy (that was my quick fix).  I took out the groupBy
> > and simply issued another ibatis command to get the list inside
> > "handleRow()".  That was my quick fix and it is probably sufficient for
> > now.
> >
> > You can still run out of memory if you use "queryWithRowHandler" but
> > highly unlikely unless you have an application like mine which could
> > have a thread running for days.  But to be really safe, you have to go
> > back to JDBC.  Since I was on a tight deadline, I didn't bother going
> > through the ibatis code but I may do that one of these days because it
> > is a wonderful tool.
> >
> >
> > On Tue, 8 Apr 2008 12:27:29 -0700 (PDT), "Carlos de Luna Saenz"
> > <cdelunasaenz@yahoo.com.mx> said:
> > > It seems like there is not enough data in your mail...
> > > what data base are you accessing? how are you dealing with it
> > (DataSource
> > > is JDBC or via the App Server) your class is keeping the data for any
> > > reason (maybe in a session or application bean)? what's the App Server?
> > > etc, etc... if i were you i should look all that kind of variables and
> > > monitor with a profiler to see if there's somehint wrong with the query,
> > > the server, the managed memory, or all of the above.
> > > Greetings
> > > Carlos de Luna
> > >
> > > ----- Mensaje original ----
> > > De: Fernando Simonetti <fernando@cosipa.com.br>
> > > Para: "user-java@ibatis.apache.org" <user-java@ibatis.apache.org>
> > > Enviado: martes, 8 de abril, 2008 7:51:26
> > > Asunto: groupby x out of memory
> > >
> > > Eventually my aplication server is going down by "Out of Memory".
> > > Analizing
> > > de heap dumping, it's point for the same class allocating 600 Mb.
> > >
> > > This event can occur 3 hours or 3 days after I restart my server. This
> > > class
> > > is used thounsands times in the period. Monitoring the memory of the
> > > server,
> > > I could see 300MB to 700MB of memory been allocated in  3 minutes. It
> > > hapens
> > > eventualy but not in a determined period.
> > >
> > > The unique diference for this class for others, is the groupby clause in
> > > Ibatis. First I thouth it could be a cartesian product, but I ran the
> > > query
> > > without parameters and found nothing.
> > >
> > > I going to implement this class without groupby clause, but first I
> > would
> > > like to see if there are other options.
> > >
> > >
> > > thanks,
> > > Fernando
> > >
> > >
> > >
> > ____________________________________________________________________________________
> > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > http://correo.espanol.yahoo.com/
> > >
> >
> > --
> > http://www.fastmail.fm - Accessible with your email software
> >                          or over the web
> >
> >

-- 
http://www.fastmail.fm - mmm... Fastmail...


Mime
View raw message