ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Maves" <nathan.ma...@gmail.com>
Subject Re: groupby x out of memory
Date Fri, 11 Apr 2008 19:32:55 GMT
We have seen a few posts about using RowHandlers in the past few months.
Though I have not tried it with the optional groupBy attribute I can
definitively say that I have processed millions of row through a row handler
and my application never user more the 10mg of memory.


On Fri, Apr 11, 2008 at 11:57 AM, Harvey Kim <cashtag@fastmail.fm> wrote:

> Is the app running for days/years or is the query running for
> days/years?  If the query returns 24 million rows, I get out of error
> memory every single time.  In theory, no query should really take days
> to return but we just happen to run into this case because our DBA
> forgot to put the index on the tables :-).  After the indexing, the
> analyzing, paralleling, ... the query came back much much faster.  But
> it also ran out of memory much much faster.  I think it's the function
> of how many rows get processed on a single query - not really how long
> the apps been running.  And I doubt there are many application out there
> that returns 24 millions rows on a single query where each object
> returned can be thousands of bytes.  Or maybe there are... If there are,
> I'd love to hear if there were memory leak problems on their app.
>
> I'm just guessing here but internally, doesn't groupBy allocate HashMap
> to keep track of what it is grouping?
>
> On Fri, 11 Apr 2008 11:42:10 -0600, "Clinton Begin"
> <clinton.begin@gmail.com> said:
> > I can't imagine why that would happen.  If this was a bug, I would think
> > far
> > more people would be experiencing it.  I have apps that have run for
> > months,
> > and in some cases years, without running out of memory.  GroupBy doesn't
> > do
> > anything special really.  It just skips duplicate rows for a parent
> > class.
> >
> > Try iBATIS 2.3.1.  If there is a problem, it might have to do with
> > session/transaction management.  2.3.1 improves the handling of such.
> >
> > Clinton
> >
> > On Fri, Apr 11, 2008 at 10:10 AM, Harvey Kim <cashtag@fastmail.fm>
> wrote:
> >
> > > If you use groupBy, you will eventually run out of memory.  Few months
> > > ago, I e-mailed a sample program which showed my application doing
> > > absolute nothing except issuing a ibatis query.  It was using
> > > queryWithRowHandler.  Literally, all it did was call
> > > "queryWithRowHandler" and my rowHanlder did absolutely nothing.
> > > Literally this:
> > >
> > > public class IbatisDownloadThread implements Runnable, Serializable,
> > > RowHandler
> > > {
> > >  private SqlMapClient sqlMapClient = null;
> > >
> > >  public void run()
> > >  {
> > >      try
> > >      {
> > >        sqlMapClient = BaseDAO.getSqlMapClient();
> > >        sqlMapClient.queryWithRowHandler("getIdList", queryDto, this);
> > >     }
> > >  }
> > >
> > >  public void handleRow()
> > >  {
> > >  }
> > > }
> > >
> > > The above code eventually gets "Out of Memory" error and the query is
> > > using a groupBy.  When I took out the "groupBy", it went MUCH MUCH
> MUCH
> > > longer.  But even that ran out of memory after few hours of running.
> > > However, once I converted everything to a JDBC loop, "Out of Memory"
> > > error went away.  When I examined the available memory at runtime, I
> > > could see the available memory decreasing with every call to
> > > "handleRow()".  With JDBC, I examined the heap size inside the loop
> and
> > > it held steady for hours and finished without a problem.
> > >
> > > The moral of the story - you could probably get away with ibatis if
> you
> > > take out the groupBy (that was my quick fix).  I took out the groupBy
> > > and simply issued another ibatis command to get the list inside
> > > "handleRow()".  That was my quick fix and it is probably sufficient
> for
> > > now.
> > >
> > > You can still run out of memory if you use "queryWithRowHandler" but
> > > highly unlikely unless you have an application like mine which could
> > > have a thread running for days.  But to be really safe, you have to go
> > > back to JDBC.  Since I was on a tight deadline, I didn't bother going
> > > through the ibatis code but I may do that one of these days because it
> > > is a wonderful tool.
> > >
> > >
> > > On Tue, 8 Apr 2008 12:27:29 -0700 (PDT), "Carlos de Luna Saenz"
> > > <cdelunasaenz@yahoo.com.mx> said:
> > > > It seems like there is not enough data in your mail...
> > > > what data base are you accessing? how are you dealing with it
> > > (DataSource
> > > > is JDBC or via the App Server) your class is keeping the data for
> any
> > > > reason (maybe in a session or application bean)? what's the App
> Server?
> > > > etc, etc... if i were you i should look all that kind of variables
> and
> > > > monitor with a profiler to see if there's somehint wrong with the
> query,
> > > > the server, the managed memory, or all of the above.
> > > > Greetings
> > > > Carlos de Luna
> > > >
> > > > ----- Mensaje original ----
> > > > De: Fernando Simonetti <fernando@cosipa.com.br>
> > > > Para: "user-java@ibatis.apache.org" <user-java@ibatis.apache.org>
> > > > Enviado: martes, 8 de abril, 2008 7:51:26
> > > > Asunto: groupby x out of memory
> > > >
> > > > Eventually my aplication server is going down by "Out of Memory".
> > > > Analizing
> > > > de heap dumping, it's point for the same class allocating 600 Mb.
> > > >
> > > > This event can occur 3 hours or 3 days after I restart my server.
> This
> > > > class
> > > > is used thounsands times in the period. Monitoring the memory of the
> > > > server,
> > > > I could see 300MB to 700MB of memory been allocated in  3 minutes.
> It
> > > > hapens
> > > > eventualy but not in a determined period.
> > > >
> > > > The unique diference for this class for others, is the groupby
> clause in
> > > > Ibatis. First I thouth it could be a cartesian product, but I ran
> the
> > > > query
> > > > without parameters and found nothing.
> > > >
> > > > I going to implement this class without groupby clause, but first I
> > > would
> > > > like to see if there are other options.
> > > >
> > > >
> > > > thanks,
> > > > Fernando
> > > >
> > > >
> > > >
> > >
> ____________________________________________________________________________________
> > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > > http://correo.espanol.yahoo.com/
> > > >
> > >
> > > --
> > > http://www.fastmail.fm - Accessible with your email software
> > >                          or over the web
> > >
> > >
>
> --
> http://www.fastmail.fm - mmm... Fastmail...
>
>
Mime
View raw message