ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clinton Begin" <clinton.be...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Mon, 17 Mar 2008 13:22:39 GMT
Wow, this thread is interesting.   I suppose I could read the past emails,
but could someone summarize the problem and progress to date?  I may be able
to shed some light on what is happening.

Clinton

On Sun, Mar 16, 2008 at 11:49 PM, nch <underscore_dot@yahoo.com> wrote:

>
> Another thing I forgot. This works perfectly well if we replace iBatis by
> Hibernate, so I don't think this is being caused by the MySQL driver.
>
> ----- Original Message ----
> From: nch <underscore_dot@yahoo.com>
> To: user-java@ibatis.apache.org
> Sent: Sunday, March 16, 2008 10:18:42 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
>
> Well, I modified the mediumblob into a mediumtext and removed all other
> fields except the id one.
>
> ----- Original Message ----
> From: nch <underscore_dot@yahoo.com>
> To: user-java@ibatis.apache.org
> Sent: Sunday, March 16, 2008 10:06:34 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
>
> You're probably right. I'm using the table called "text" which contains a
> field of type "mediumblob".
>
> CREATE TABLE /*$wgDBprefix*/text (
>   old_id int unsigned NOT NULL auto_increment,
>   old_text mediumblob NOT NULL,
>   old_flags tinyblob NOT NULL,
>   PRIMARY KEY old_id (old_id)
> ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
>
> You can find it's definition here:
>
> http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
>
> I'll try to modify that field, first, and see what happens.
> I've already tried "select * from your_wikipedia_dump limit $start$,
> $size$" and it seems to work fine, but I see a possible issue with this, you
> can not remove any of the records you've already read during the whole
> process. This might not be an option.
>
> Thank you.
>
> ----- Original Message ----
> From: Larry Meadors <larry.meadors@gmail.com>
> To: user-java@ibatis.apache.org
> Sent: Sunday, March 16, 2008 4:19:43 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> Yeah, I just noticed that - it looks like it's failing in the jdbc
> driver when it tries to read a blob, maybe?
>
> I know that some versions of the postgresql jdbc driver load the
> entire contents of a resultset into a massive byte[][] structure. I
> wonder if that is what is happening here? I think you can get the
> source for the MySQL driver and check that if you want to.
>
> If it is, you may need to structure your app to take smaller bites of
> this elephant.
>
> Try "select * from your_wikipedia_dump limit $start$, $size$" instead
> - that will limit the size of the results to $size$ rows, starting on
> row $start$ (the $start$ value is zero-based). If you experiment to
> see how big $size$ can get, you should be able to get pretty decent
> performance out of this.
>
> Psuedo-code would be like this...
>
> int start = 0;
> int size = 1000; // see how big you can make this to improve performance
> while(still_more_data){
>   still_more_data = index(start, size);
>   start += size;
> }
>
> Your index method will return true if the query returned any data. If
> the query returns no data, it'll return false.
>
> I can't imagine what Hibernate is doing differently here to make this
> work if the JDBC driver is failing when executing the query.
>
> Larry
>
>
> On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com> wrote:
> >
> >
> > Sure. Please, see attached.
> > I don't think the problem is in the RowHandler, though, because the
> > OutOfMemoryError occurs before invoking RowHandle#handleRow.
> >
> > Cheers
> >
> >
> > ----- Original Message ----
> > From: Larry Meadors <larry.meadors@gmail.com>
> > To: user-java@ibatis.apache.org
> >
> > Sent: Sunday, March 16, 2008 3:13:27 AM
> > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> >  Can you post the row handler you are using.
> >
> > Larry
> >
> >
> > On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com> wrote:
> > >
> > >
> > > Hi, Nathan.
> > > I did so, but I'm still having the same issue. Perhaps I'm not using
> it
> > > correctly?
> > > See my last post to this forum entry:
> > >
> > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > >
> > > See the stack trace:
> > >
> > > MemoryError: Java heap space:
> > > java.lang.OutOfMemoryError: Java heap space
> > >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > >        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> > >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> > >        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> > >        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> > >        at
> > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> > >        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> > >        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> > >        at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> > >        at
> > >
> > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java
> :1268)
> > >        at
> > > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> > >        at
> > >
> > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> DelegatingPreparedStatement.java:169)
> > >        at
> > >
> > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> DelegatingPreparedStatement.java:169)
> > >        at
> > >
> > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(
> SqlExecutor.java:186)
> > >        at
> > >
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery
> (GeneralStatement.java:205)
> > >        at
> > >
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback
> (GeneralStatement.java:173)
> > >        at
> > >
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler
> (GeneralStatement.java:133)
> > >        at
> > >
> > com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler
> (SqlMapExecutorDelegate.java:649)
> > >        at
> > >
> > com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(
> SqlMapSessionImpl.java:156)
> > >        at
> > >
> > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(
> SqlMapClientGpsDeviceWithRowHandler.java:33)
> > >        at
> > >
> > org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(
> AbstractGpsDevice.java:93)
> > >        at
> > >
> > org.compass.core.CompassCallbackWithoutResult.doInCompass(
> CompassCallbackWithoutResult.java:29)
> > >        at
> > > org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> > >        at
> > >
> > org.compass.gps.impl.SingleCompassGps.executeForIndex(
> SingleCompassGps.java:161)
> > >        at
> > > org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java
> :91)
> > >        at
> > >
> >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult
> (SpringSyncTransactionGpsDeviceWrapper.java:98)
> > >        at
> > >
> >
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction
> (TransactionCallbackWithoutResult.java:33)
> > >        at
> > >
> > org.springframework.transaction.support.TransactionTemplate.execute(
> TransactionTemplate.java:128)
> > >        at
> > >
> > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(
> SpringSyncTransactionGpsDeviceWrapper.java:96)
> > >        at
> > >
> > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(
> SingleCompassGps.java:133)
> > >        at
> > >
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep
> (DefaultLuceneSearchEngineIndexManager.java:233)
> > >        at
> > >
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate
> (DefaultLuceneSearchEngineIndexManager.java:182)
> > >
> > >
> > >
> > >
> > >
> > > ----- Original Message ----
> > > From: Nathan Maves <nathan.maves@gmail.com>
> > > To: user-java@ibatis.apache.org
> > >
> > > Sent: Saturday, March 15, 2008 5:43:04 PM
> > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > >
> > >  this is only my 2 cents but I would throw that class out.  Write your
> own
> > > implementation which uses a row handler.  This is the type of
> situation in
> > > which a row handler could really help.  almost no memory would be
> used.
> > >
> > >
> > >
> > > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com> wrote:
> > > >
> > > > Yes, I agree that dividing the query into several queries should do
> as
> > > long as none of them uses more memory than available. Gracias, Carlos.
> > > >
> > > > Nevertheless, I looked into
> > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> > > fundamentally, what it does is a queryForPaginatedList and iterates
> > through
> > > the resulting PaginatedList with nextPage()
> > > >
> > > > I guess that the reason why pages are kept in memory after
> performing
> > > paginatedList.nextPage() is because a paginatedList.previousPage() can
> be
> > > issued but, should't it free pages if heap space is running out?
> > > >
> > > >
> > > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx> wrote:
> > > >
> > > > Since we have a similar trouble around here i must say that you have
> lot
> > > of options when going to compass..
> > > > The first one is to index "object by object" by hand, the second one
> is
> > to
> > > use the Gps wich will use the complete bunch of data and make
> subindexes
> > > (making parts of your "main index" with diferent "maps" for each
> subindex
> > in
> > > your gpsDriver.
> > > > if your indexing is as strong as ours i should look forward for
> numer 1
> > or
> > > number 3 depending of your needs. (More a Compass than iBatis issue)
> > > > Greetings
> > > > Carlos de Luna
> > > >
> > > > ----- Mensaje original ----
> > > > De: nch
> > > >
> > > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > > Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > Sorry. Yes, I need all the data at one time, because the Compass API
> > seems
> > > to be meant in that way.
> > > >
> > > > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i
> > need
> > > all the data at
> > > >
> > > >
> > > >
> > > > one time | no, i do not need all of the data at one time ].
> > > >
> > > > Larry
> > > >
> > > >
> > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap
> > space
> > > in
> > > > > both cases. I am using a profiler to look into what's happening
> during
> > > > > execution of the indexing process and I can tell that's the exact
> > amount
> > > it
> > > > > displays for the VM.
> > > > >
> > > > > As I describe in the Compass user forum (see link bellow), I'm
> > indexing
> > > a
> > > > > Wikipedia dump of about 650000-675000 records. That's sort of
> 1-1.5GB
> > of
> > > > > data.
> > > > >
> > > > > Hibernate can handle all that data, I guess it's using some sort
> of
> > > > > pagination. iBatis seems to divide the select statement into
> several
> > > select
> > > > > statements, but it also seems each of such statements surpasses
> the
> > > limit of
> > > > > 1024MB when trying to index only 300000 of the database records.
> > > > >
> > > > > Thank you
> > > > >
> > > > >
> > > > >
> > > > > Chris Lamey wrote:
> > > > > What is you JVM heap size set to and how much data is returned by
> > select
> > > *
> > > > > from table?
> > > > >
> > > > > If you're trying to pull back 1G worth of data into a JVM with a
> heap
> > > size
> > > > > set to 64M, you will hit the heap limit pretty quick.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > > Sent: Thu 3/13/2008 8:03 AM
> > > > > To: user-java@ibatis.apache.org
> > > > > Subject: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > >
> > > > > Hi, everybody.
> > > > > I'm testing the Compass search engine and, in particular, how to
> index
> > a
> > > big
> > > > > set of documents from a table in a MySQL database. In order to do
> this
> > I
> > > > > issue a "select * from table_name" using iBatis, but this seems to
> be
> > > > > causing the application to use all heap space available.
> > > > > I added a new entry into Compass user forum with details:
> > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > >
> > > > > Can you figure out why is this happening?
> > > > >
> > > > > Many thanks
> > > > >
> > > > >
> > > > > ---------------------------------
> > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile.
> Try
> > > it
> > > > > now.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > > Never miss a thing. Make Yahoo your homepage.
> > > >
> > > >
> > > > Never miss a thing. Make Yahoo your homepage.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> ____________________________________________________________________________________
> > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > > http://correo.espanol.yahoo.com/
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > >
> > >
> > >  ________________________________
> > > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >  ________________________________
> >
> > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it
> > now.
>
>
> ------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
>
>
> ------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ%20>
>
>
> ------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
>

Mime
View raw message