ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Larry Meadors" <larry.mead...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Sun, 16 Mar 2008 15:19:43 GMT
Yeah, I just noticed that - it looks like it's failing in the jdbc
driver when it tries to read a blob, maybe?

I know that some versions of the postgresql jdbc driver load the
entire contents of a resultset into a massive byte[][] structure. I
wonder if that is what is happening here? I think you can get the
source for the MySQL driver and check that if you want to.

If it is, you may need to structure your app to take smaller bites of
this elephant.

Try "select * from your_wikipedia_dump limit $start$, $size$" instead
- that will limit the size of the results to $size$ rows, starting on
row $start$ (the $start$ value is zero-based). If you experiment to
see how big $size$ can get, you should be able to get pretty decent
performance out of this.

Psuedo-code would be like this...

int start = 0;
int size = 1000; // see how big you can make this to improve performance
while(still_more_data){
  still_more_data = index(start, size);
  start += size;
}

Your index method will return true if the query returned any data. If
the query returns no data, it'll return false.

I can't imagine what Hibernate is doing differently here to make this
work if the JDBC driver is failing when executing the query.

Larry


On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com> wrote:
>
>
> Sure. Please, see attached.
> I don't think the problem is in the RowHandler, though, because the
> OutOfMemoryError occurs before invoking RowHandle#handleRow.
>
> Cheers
>
>
> ----- Original Message ----
> From: Larry Meadors <larry.meadors@gmail.com>
> To: user-java@ibatis.apache.org
>
> Sent: Sunday, March 16, 2008 3:13:27 AM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
>  Can you post the row handler you are using.
>
> Larry
>
>
> On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com> wrote:
> >
> >
> > Hi, Nathan.
> > I did so, but I'm still having the same issue. Perhaps I'm not using it
> > correctly?
> > See my last post to this forum entry:
> >
> > http://forum.compass-project.org/thread.jspa?threadID=215278
> >
> > See the stack trace:
> >
> > MemoryError: Java heap space:
> > java.lang.OutOfMemoryError: Java heap space
> >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> >        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> >        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> >        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> >        at
> > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> >        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> >        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> >        at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> >        at
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> >        at
> > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> >        at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> >        at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> >        at
> >
> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
> >        at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
> >        at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
> >        at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
> >        at
> >
> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
> >        at
> >
> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
> >        at
> >
> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
> >        at
> >
> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
> >        at
> >
> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
> >        at
> > org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> >        at
> >
> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
> >        at
> > org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
> >        at
> >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
> >        at
> >
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
> >        at
> >
> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
> >        at
> >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
> >        at
> >
> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
> >        at
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
> >        at
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
> >
> >
> >
> >
> >
> > ----- Original Message ----
> > From: Nathan Maves <nathan.maves@gmail.com>
> > To: user-java@ibatis.apache.org
> >
> > Sent: Saturday, March 15, 2008 5:43:04 PM
> > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> >  this is only my 2 cents but I would throw that class out.  Write your own
> > implementation which uses a row handler.  This is the type of situation in
> > which a row handler could really help.  almost no memory would be used.
> >
> >
> >
> > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com> wrote:
> > >
> > > Yes, I agree that dividing the query into several queries should do as
> > long as none of them uses more memory than available. Gracias, Carlos.
> > >
> > > Nevertheless, I looked into
> > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> > fundamentally, what it does is a queryForPaginatedList and iterates
> through
> > the resulting PaginatedList with nextPage()
> > >
> > > I guess that the reason why pages are kept in memory after performing
> > paginatedList.nextPage() is because a paginatedList.previousPage() can be
> > issued but, should't it free pages if heap space is running out?
> > >
> > >
> > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx> wrote:
> > >
> > > Since we have a similar trouble around here i must say that you have lot
> > of options when going to compass..
> > > The first one is to index "object by object" by hand, the second one is
> to
> > use the Gps wich will use the complete bunch of data and make subindexes
> > (making parts of your "main index" with diferent "maps" for each subindex
> in
> > your gpsDriver.
> > > if your indexing is as strong as ours i should look forward for numer 1
> or
> > number 3 depending of your needs. (More a Compass than iBatis issue)
> > > Greetings
> > > Carlos de Luna
> > >
> > > ----- Mensaje original ----
> > > De: nch
> > >
> > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
> > >
> > >
> > > Sorry. Yes, I need all the data at one time, because the Compass API
> seems
> > to be meant in that way.
> > >
> > > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i
> need
> > all the data at
> > >
> > >
> > >
> > > one time | no, i do not need all of the data at one time ].
> > >
> > > Larry
> > >
> > >
> > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > >
> > > > Hi!
> > > >
> > > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap
> space
> > in
> > > > both cases. I am using a profiler to look into what's happening during
> > > > execution of the indexing process and I can tell that's the exact
> amount
> > it
> > > > displays for the VM.
> > > >
> > > > As I describe in the Compass user forum (see link bellow), I'm
> indexing
> > a
> > > > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB
> of
> > > > data.
> > > >
> > > > Hibernate can handle all that data, I guess it's using some sort of
> > > > pagination. iBatis seems to divide the select statement into several
> > select
> > > > statements, but it also seems each of such statements surpasses the
> > limit of
> > > > 1024MB when trying to index only 300000 of the database records.
> > > >
> > > > Thank you
> > > >
> > > >
> > > >
> > > > Chris Lamey wrote:
> > > > What is you JVM heap size set to and how much data is returned by
> select
> > *
> > > > from table?
> > > >
> > > > If you're trying to pull back 1G worth of data into a JVM with a heap
> > size
> > > > set to 64M, you will hit the heap limit pretty quick.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > Sent: Thu 3/13/2008 8:03 AM
> > > > To: user-java@ibatis.apache.org
> > > > Subject: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > Hi, everybody.
> > > > I'm testing the Compass search engine and, in particular, how to index
> a
> > big
> > > > set of documents from a table in a MySQL database. In order to do this
> I
> > > > issue a "select * from table_name" using iBatis, but this seems to be
> > > > causing the application to use all heap space available.
> > > > I added a new entry into Compass user forum with details:
> > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > >
> > > > Can you figure out why is this happening?
> > > >
> > > > Many thanks
> > > >
> > > >
> > > > ---------------------------------
> > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> > it
> > > > now.
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> ____________________________________________________________________________________
> > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > http://correo.espanol.yahoo.com/
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ________________________________
> > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >
> >
> >  ________________________________
> > Never miss a thing. Make Yahoo your homepage.
>
>
>  ________________________________
>
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.

Mime
View raw message