ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harvey Kim" <cash...@fastmail.fm>
Subject Re: groupby x out of memory
Date Sat, 12 Apr 2008 19:49:01 GMT
Yeap, that's what I ended up doing.  I used subsequent selects to get
the list and that seem to have solved the problem.  Although, I did get
out of memory with using RowHandler even after I got rid of the groupBy
but that only happened once (out of thousands of runs as oppose to
happening everytime with groupBy).  So I can't be 100% positive that
just using the RowHandler (without the groupby) will cause memory
problem since I haven't been really able to reproduce it except seeing
the available memory decreasing slowly with every call to the
rowhandler.  And based on what people are saying, I'm starting to feel
little more relieved and confident that it was strictly a groupBy
problem and the RowHandler itself isn't the cause.  Any other reassuring
experience and supporting comments will be greatly appreciated since I
did see it happen once :).


On Fri, 11 Apr 2008 22:50:33 -0600, "Clinton Begin"
<clinton.begin@gmail.com> said:
> OOOOOHHHHHH!!!!..... now this makes sense.....  I misunderstood the
> original
> issue.
> 
> My apologies.  Yes, now it's clear.  The internal tracking of the row IDs
> will cause an out of memory exception to occur (as you know, it is a
> hashmap).
> 
> Unfortunately there's no practical solution to this.  For groupBy to
> work,
> it has to track the IDs, and really, for groupby to be effective, the end
> result set should end up in memory to avoid handling partial data.  I'll
> have to check, but I'm not even sure the rows would be processed in the
> correct order for the rowhandler to work with groupby.  For example, if
> there's 100 records between the parent object and a child element in the
> collection, the parent may have already been passed back to the
> rowhandler
> before all the children are loaded.
> 
> These features are fundamentally incompatible.  There's no magic solution
> for it unfortunately.  Your best bet is to flatten the object hierarchy
> for
> this process to avoid the need for groupby. You'll have the same problem
> if
> you tried to mimick this behavior with JDBC or any other toolset (other
> than
> something that simply used a subsequent select to load children).
> 
> Cheers,
> Clinton
> 
> 
> 
> 
> On Fri, Apr 11, 2008 at 11:57 AM, Harvey Kim <cashtag@fastmail.fm> wrote:
> 
> > Is the app running for days/years or is the query running for
> > days/years?  If the query returns 24 million rows, I get out of error
> > memory every single time.  In theory, no query should really take days
> > to return but we just happen to run into this case because our DBA
> > forgot to put the index on the tables :-).  After the indexing, the
> > analyzing, paralleling, ... the query came back much much faster.  But
> > it also ran out of memory much much faster.  I think it's the function
> > of how many rows get processed on a single query - not really how long
> > the apps been running.  And I doubt there are many application out there
> > that returns 24 millions rows on a single query where each object
> > returned can be thousands of bytes.  Or maybe there are... If there are,
> > I'd love to hear if there were memory leak problems on their app.
> >
> > I'm just guessing here but internally, doesn't groupBy allocate HashMap
> > to keep track of what it is grouping?
> >
> > On Fri, 11 Apr 2008 11:42:10 -0600, "Clinton Begin"
> > <clinton.begin@gmail.com> said:
> > > I can't imagine why that would happen.  If this was a bug, I would think
> > > far
> > > more people would be experiencing it.  I have apps that have run for
> > > months,
> > > and in some cases years, without running out of memory.  GroupBy doesn't
> > > do
> > > anything special really.  It just skips duplicate rows for a parent
> > > class.
> > >
> > > Try iBATIS 2.3.1.  If there is a problem, it might have to do with
> > > session/transaction management.  2.3.1 improves the handling of such.
> > >
> > > Clinton
> > >
> > > On Fri, Apr 11, 2008 at 10:10 AM, Harvey Kim <cashtag@fastmail.fm>
> > wrote:
> > >
> > > > If you use groupBy, you will eventually run out of memory.  Few months
> > > > ago, I e-mailed a sample program which showed my application doing
> > > > absolute nothing except issuing a ibatis query.  It was using
> > > > queryWithRowHandler.  Literally, all it did was call
> > > > "queryWithRowHandler" and my rowHanlder did absolutely nothing.
> > > > Literally this:
> > > >
> > > > public class IbatisDownloadThread implements Runnable, Serializable,
> > > > RowHandler
> > > > {
> > > >  private SqlMapClient sqlMapClient = null;
> > > >
> > > >  public void run()
> > > >  {
> > > >      try
> > > >      {
> > > >        sqlMapClient = BaseDAO.getSqlMapClient();
> > > >        sqlMapClient.queryWithRowHandler("getIdList", queryDto, this);
> > > >     }
> > > >  }
> > > >
> > > >  public void handleRow()
> > > >  {
> > > >  }
> > > > }
> > > >
> > > > The above code eventually gets "Out of Memory" error and the query is
> > > > using a groupBy.  When I took out the "groupBy", it went MUCH MUCH
> > MUCH
> > > > longer.  But even that ran out of memory after few hours of running.
> > > > However, once I converted everything to a JDBC loop, "Out of Memory"
> > > > error went away.  When I examined the available memory at runtime, I
> > > > could see the available memory decreasing with every call to
> > > > "handleRow()".  With JDBC, I examined the heap size inside the loop
> > and
> > > > it held steady for hours and finished without a problem.
> > > >
> > > > The moral of the story - you could probably get away with ibatis if
> > you
> > > > take out the groupBy (that was my quick fix).  I took out the groupBy
> > > > and simply issued another ibatis command to get the list inside
> > > > "handleRow()".  That was my quick fix and it is probably sufficient
> > for
> > > > now.
> > > >
> > > > You can still run out of memory if you use "queryWithRowHandler" but
> > > > highly unlikely unless you have an application like mine which could
> > > > have a thread running for days.  But to be really safe, you have to go
> > > > back to JDBC.  Since I was on a tight deadline, I didn't bother going
> > > > through the ibatis code but I may do that one of these days because it
> > > > is a wonderful tool.
> > > >
> > > >
> > > > On Tue, 8 Apr 2008 12:27:29 -0700 (PDT), "Carlos de Luna Saenz"
> > > > <cdelunasaenz@yahoo.com.mx> said:
> > > > > It seems like there is not enough data in your mail...
> > > > > what data base are you accessing? how are you dealing with it
> > > > (DataSource
> > > > > is JDBC or via the App Server) your class is keeping the data for
> > any
> > > > > reason (maybe in a session or application bean)? what's the App
> > Server?
> > > > > etc, etc... if i were you i should look all that kind of variables
> > and
> > > > > monitor with a profiler to see if there's somehint wrong with the
> > query,
> > > > > the server, the managed memory, or all of the above.
> > > > > Greetings
> > > > > Carlos de Luna
> > > > >
> > > > > ----- Mensaje original ----
> > > > > De: Fernando Simonetti <fernando@cosipa.com.br>
> > > > > Para: "user-java@ibatis.apache.org" <user-java@ibatis.apache.org>
> > > > > Enviado: martes, 8 de abril, 2008 7:51:26
> > > > > Asunto: groupby x out of memory
> > > > >
> > > > > Eventually my aplication server is going down by "Out of Memory".
> > > > > Analizing
> > > > > de heap dumping, it's point for the same class allocating 600 Mb.
> > > > >
> > > > > This event can occur 3 hours or 3 days after I restart my server.
> > This
> > > > > class
> > > > > is used thounsands times in the period. Monitoring the memory of
the
> > > > > server,
> > > > > I could see 300MB to 700MB of memory been allocated in  3 minutes.
> > It
> > > > > hapens
> > > > > eventualy but not in a determined period.
> > > > >
> > > > > The unique diference for this class for others, is the groupby
> > clause in
> > > > > Ibatis. First I thouth it could be a cartesian product, but I ran
> > the
> > > > > query
> > > > > without parameters and found nothing.
> > > > >
> > > > > I going to implement this class without groupby clause, but first
I
> > > > would
> > > > > like to see if there are other options.
> > > > >
> > > > >
> > > > > thanks,
> > > > > Fernando
> > > > >
> > > > >
> > > > >
> > > >
> > ____________________________________________________________________________________
> > > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > > > http://correo.espanol.yahoo.com/
> > > > >
> > > >
> > > > --
> > > > http://www.fastmail.fm - Accessible with your email software
> > > >                          or over the web
> > > >
> > > >
> >
> > --
> > http://www.fastmail.fm - mmm... Fastmail...
> >
> >

-- 
http://www.fastmail.fm - Send your email first class


Mime
View raw message