Mailing-List: contact user-java-help@ibatis.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user-java@ibatis.apache.org
Received-SPF: pass (athena.apache.org: domain of larry.meadors@gmail.com
 designates 66.249.92.173 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=M7MEFcghA+rSNvUCRKu76MJqMlVpuR9x2eDMFUkGwGEV3CbTcYx0wBr39D2FhiCcERGsmyQtq+4bLQ9NWmY7r3gSjMSAyric2aR79hBkz8lPCSQrNdkmQvUVAJffBg8VFfamXdCjc4zPski/3QPxV7FmvjV18DKTuyyD4iCoE7w=
Message-ID: <a10a823c0803151913k79254126wc5c091ac7ff16917@mail.gmail.com>
Date: Sat, 15 Mar 2008 20:13:27 -0600
From: "Larry Meadors" <larry.meadors@gmail.com>
Reply-To: larry.meadors@gmail.com
To: user-java@ibatis.apache.org
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
In-Reply-To: <323514.89556.qm@web32401.mail.mud.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <323514.89556.qm@web32401.mail.mud.yahoo.com>

Can you post the row handler you are using.

Larry


On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com> wrote:
>
>
> Hi, Nathan.
> I did so, but I'm still having the same issue. Perhaps I'm not using it
> correctly?
> See my last post to this forum entry:
>
> http://forum.compass-project.org/thread.jspa?threadID=3D215278
>
> See the stack trace:
>
> MemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
>         at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
>         at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
>         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
>         at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
>         at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
>         at
> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
>         at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
>         at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
>         at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1=
268)
>         at
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
>         at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre=
paredStatement.java:169)
>         at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre=
paredStatement.java:169)
>         at
> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.j=
ava:186)
>         at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQue=
ry(GeneralStatement.java:205)
>         at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW=
ithCallback(GeneralStatement.java:173)
>         at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW=
ithRowHandler(GeneralStatement.java:133)
>         at
> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(=
SqlMapExecutorDelegate.java:649)
>         at
> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMa=
pSessionImpl.java:156)
>         at
> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClien=
tGpsDeviceWithRowHandler.java:33)
>         at
> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(Abstr=
actGpsDevice.java:93)
>         at
> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallback=
WithoutResult.java:29)
>         at
> org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
>         at
> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.ja=
va:161)
>         at
> org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
>         at
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTra=
nsactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
>         at
> org.springframework.transaction.support.TransactionCallbackWithoutResult.=
doInTransaction(TransactionCallbackWithoutResult.java:33)
>         at
> org.springframework.transaction.support.TransactionTemplate.execute(Trans=
actionTemplate.java:128)
>         at
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(Spr=
ingSyncTransactionGpsDeviceWrapper.java:96)
>         at
> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassG=
ps.java:133)
>         at
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana=
ger$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
>         at
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana=
ger.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
>
>
>
>
>
> ----- Original Message ----
> From: Nathan Maves <nathan.maves@gmail.com>
> To: user-java@ibatis.apache.org
>
> Sent: Saturday, March 15, 2008 5:43:04 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
>  this is only my 2 cents but I would throw that class out.  Write your ow=
n
> implementation which uses a row handler.  This is the type of situation i=
n
> which a row handler could really help.  almost no memory would be used.
>
>
>
> On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com> wrote:
> >
> > Yes, I agree that dividing the query into several queries should do as
> long as none of them uses more memory than available. Gracias, Carlos.
> >
> > Nevertheless, I looked into
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> fundamentally, what it does is a queryForPaginatedList and iterates throu=
gh
> the resulting PaginatedList with nextPage()
> >
> > I guess that the reason why pages are kept in memory after performing
> paginatedList.nextPage() is because a paginatedList.previousPage() can be
> issued but, should't it free pages if heap space is running out?
> >
> >
> > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx> wrote:
> >
> > Since we have a similar trouble around here i must say that you have lo=
t
> of options when going to compass..
> > The first one is to index "object by object" by hand, the second one is=
 to
> use the Gps wich will use the complete bunch of data and make subindexes
> (making parts of your "main index" with diferent "maps" for each subindex=
 in
> your gpsDriver.
> > if your indexing is as strong as ours i should look forward for numer 1=
 or
> number 3 depending of your needs. (More a Compass than iBatis issue)
> > Greetings
> > Carlos de Luna
> >
> > ----- Mensaje original ----
> > De: nch
> >
> > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> >
> > Sorry. Yes, I need all the data at one time, because the Compass API se=
ems
> to be meant in that way.
> >
> > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i ne=
ed
> all the data at
> >
> >
> >
> > one time | no, i do not need all of the data at one time ].
> >
> > Larry
> >
> >
> > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > >
> > > Hi!
> > >
> > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap spa=
ce
> in
> > > both cases. I am using a profiler to look into what's happening durin=
g
> > > execution of the indexing process and I can tell that's the exact amo=
unt
> it
> > > displays for the VM.
> > >
> > > As I describe in the Compass user forum (see link bellow), I'm indexi=
ng
> a
> > > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB=
 of
> > > data.
> > >
> > > Hibernate can handle all that data, I guess it's using some sort of
> > > pagination. iBatis seems to divide the select statement into several
> select
> > > statements, but it also seems each of such statements surpasses the
> limit of
> > > 1024MB when trying to index only 300000 of the database records.
> > >
> > > Thank you
> > >
> > >
> > >
> > > Chris Lamey wrote:
> > > What is you JVM heap size set to and how much data is returned by sel=
ect
> *
> > > from table?
> > >
> > > If you're trying to pull back 1G worth of data into a JVM with a heap
> size
> > > set to 64M, you will hit the heap limit pretty quick.
> > >
> > >
> > > -----Original Message-----
> > > From: nch [mailto:underscore_dot@yahoo..com]
> > > Sent: Thu 3/13/2008 8:03 AM
> > > To: user-java@ibatis.apache.org
> > > Subject: select * causing " OutOfMemoryError: Java heap space"
> > >
> > >
> > > Hi, everybody.
> > > I'm testing the Compass search engine and, in particular, how to inde=
x a
> big
> > > set of documents from a table in a MySQL database. In order to do thi=
s I
> > > issue a "select * from table_name" using iBatis, but this seems to be
> > > causing the application to use all heap space available.
> > > I added a new entry into Compass user forum with details:
> > > http://forum.compass-project.org/thread.jspa?threadID=3D215278
> > >
> > > Can you figure out why is this happening?
> > >
> > > Many thanks
> > >
> > >
> > > ---------------------------------
> > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Tr=
y
> it
> > > now.
> > >
> > >
> > >
> > >
> > > ________________________________
> > > Never miss a thing. Make Yahoo your homepage.
> >
> >
> > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >
> >
> >
> >
> >
> _________________________________________________________________________=
___________
> > =A1Capacidad ilimitada de almacenamiento en tu correo!
> > No te preocupes m=E1s por el espacio de tu cuenta con Correo Yahoo!:
> > http://correo.espanol.yahoo.com/
> >
> >
> >
> >
> >
> >
> >
> > ________________________________
> Never miss a thing. Make Yahoo your homepage.
>
>
>
>
>  ________________________________
> Never miss a thing. Make Yahoo your homepage.