ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nch <underscore_...@yahoo.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Sat, 15 Mar 2008 19:14:22 GMT

Hi, Nathan.
I did so, but I'm still having the same issue. Perhaps I'm not using it correctly?
See my last post to this forum entry:
http://forum.compass-project.org/thread.jspa?threadID=215278

See the stack trace:

MemoryError: Java heap space:
java.lang.OutOfMemoryError: Java heap space
        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
        at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
        at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
        at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
        at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
        at org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
        at org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
        at org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
        at org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
        at org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
        at org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
        at org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
        at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
        at org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
        at org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
        at org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
        at org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)




----- Original Message ----
From: Nathan Maves <nathan.maves@gmail.com>
To: user-java@ibatis.apache.org
Sent: Saturday, March 15, 2008 5:43:04 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"

this is only my 2 cents but I would throw that class out.  Write your own implementation which
uses a row handler.  This is the type of situation in which a row handler could really help.
 almost no memory would be used.



On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com> wrote:

Yes, I agree that dividing the query into several queries should do as long as none of them
uses more memory than available. Gracias, Carlos.

Nevertheless, I looked into org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
fundamentally, what it does is a queryForPaginatedList and iterates through the resulting
PaginatedList with nextPage()

I guess that the reason why pages are kept in memory after performing paginatedList.nextPage()
is because a paginatedList.previousPage() can be issued but, should't it free pages if heap
space is running out?

Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx> wrote:
 Since we have a similar trouble around here i must say that you have lot of options when
going to compass..
The first one is to index "object by object" by hand, the second one is to use the Gps wich
will use the complete bunch of data and make subindexes (making parts of your "main index"
with diferent "maps" for each subindex in your gpsDriver.
if your indexing is as strong as ours i should look forward for numer 1 or number 3 depending
of your needs. (More a Compass than iBatis issue)
Greetings
Carlos de Luna

----- Mensaje original ----
De: nch 

Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
Enviado: jueves, 13 de marzo, 2008 9:37:04
Asunto: Re: select * causing " OutOfMemoryError: Java heap space"


Sorry. Yes, I need all the data at one time, because the Compass API seems to be meant in
that way.


Larry Meadors  wrote: OK, so the answer to the question is...[ yes, i need all the data at

one time | no, i do not need all of the data at one time ].

Larry


On Thu, Mar 13, 2008 at 9:04 AM, nch  wrote:
>
> Hi!
>
> I'm doing tests both under tomcat and jetty. I set 1024MB of heap space in
> both cases. I am using a profiler to look into what's happening during
> execution of the indexing process and I can tell that's the exact amount it
> displays for the VM.
>
> As I describe in the Compass user forum (see link bellow), I'm indexing a
> Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB  of
> data.
>
> Hibernate can handle all that data, I guess it's using some sort of
> pagination. iBatis seems to divide the select statement into several select
> statements, but it also seems each of such statements surpasses the limit of
> 1024MB when trying to index only 300000 of the database records.
>
> Thank you
>
>
>
> Chris Lamey  wrote:
>  What is you JVM heap size set to and how much data is returned by select *
> from table?
>
> If you're trying to pull back 1G worth of data into a JVM with a heap size
> set to 64M, you will hit the heap limit pretty quick.
>
>
> -----Original Message-----
> From: nch [mailto:underscore_dot@yahoo..com]
> Sent: Thu 3/13/2008 8:03 AM
> To: user-java@ibatis.apache.org
> Subject: select * causing " OutOfMemoryError: Java heap space"
>
>
> Hi,  everybody.
> I'm testing the Compass search engine and, in particular, how to index a big
> set of documents from a table in a MySQL database. In order to do this I
> issue a "select * from table_name" using iBatis, but this seems to be
> causing the application to use all heap space available.
> I added a new entry into Compass user forum with details:
> http://forum.compass-project.org/thread.jspa?threadID=215278
>
> Can you figure out why is this happening?
>
> Many thanks
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.
>
>
>
>
>  ________________________________
> Never miss a thing. Make Yahoo your homepage.

         
Never miss a thing.   Make Yahoo your homepage.  






      ____________________________________________________________________________________
¡Capacidad ilimitada de almacenamiento en tu correo!
No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:                      
http://correo.espanol.yahoo.com/





 



      
Never miss a thing.   Make Yahoo your homepage.









      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ


Mime
View raw message