ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Maves" <nathan.ma...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Sat, 15 Mar 2008 16:43:04 GMT
this is only my 2 cents but I would throw that class out.  Write your own
implementation which uses a row handler.  This is the type of situation in
which a row handler could really help.  almost no memory would be used.



On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com> wrote:

>
> Yes, I agree that dividing the query into several queries should do as
> long as none of them uses more memory than available. Gracias, Carlos.
>
> Nevertheless, I looked into
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> fundamentally, what it does is a queryForPaginatedList and iterates
> through the resulting PaginatedList with nextPage()
>
> I guess that the reason why pages are kept in memory after performing
> paginatedList.nextPage() is because a paginatedList.previousPage() can be
> issued but, should't it free pages if heap space is running out?
>
> *Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>* wrote:
>
> Since we have a similar trouble around here i must say that you have lot
> of options when going to compass..
> The first one is to index "object by object" by hand, the second one is to
> use the Gps wich will use the complete bunch of data and make subindexes
> (making parts of your "main index" with diferent "maps" for each subindex in
> your gpsDriver.
> if your indexing is as strong as ours i should look forward for numer 1 or
> number 3 depending of your needs. (More a Compass than iBatis issue)
> Greetings
> Carlos de Luna
>
> ----- Mensaje original ----
> De: nch
> Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> Enviado: jueves, 13 de marzo, 2008 9:37:04
> Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
>
>
> Sorry. Yes, I need all the data at one time, because the Compass API seems
> to be meant in that way.
>
> Larry Meadors wrote: OK, so the answer to the question is...[ yes, i need
> all the data at
>
> one time | no, i do not need all of the data at one time ].
>
> Larry
>
>
> On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> >
> > Hi!
> >
> > I'm doing tests both under tomcat and jetty. I set 1024MB of heap space
> in
> > both cases. I am using a profiler to look into what's happening during
> > execution of the indexing process and I can tell that's the exact amount
> it
> > displays for the VM.
> >
> > As I describe in the Compass user forum (see link bellow), I'm indexing
> a
> > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB of
> > data.
> >
> > Hibernate can handle all that data, I guess it's using some sort of
> > pagination. iBatis seems to divide the select statement into several
> select
> > statements, but it also seems each of such statements surpasses the
> limit of
> > 1024MB when trying to index only 300000 of the database records.
> >
> > Thank you
> >
> >
> >
> > Chris Lamey wrote:
> > What is you JVM heap size set to and how much data is returned by select
> *
> > from table?
> >
> > If you're trying to pull back 1G worth of data into a JVM with a heap
> size
> > set to 64M, you will hit the heap limit pretty quick.
> >
> >
> > -----Original Message-----
> > From: nch [mailto:underscore_dot@yahoo..com]
> > Sent: Thu 3/13/2008 8:03 AM
> > To: user-java@ibatis.apache.org
> > Subject: select * causing " OutOfMemoryError: Java heap space"
> >
> >
> > Hi, everybody.
> > I'm testing the Compass search engine and, in particular, how to index a
> big
> > set of documents from a table in a MySQL database. In order to do this I
> > issue a "select * from table_name" using iBatis, but this seems to be
> > causing the application to use all heap space available.
> > I added a new entry into Compass user forum with details:
> > http://forum.compass-project.org/thread.jspa?threadID=215278
> >
> > Can you figure out why is this happening?
> >
> > Many thanks
> >
> >
> > ---------------------------------
> > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it
> > now.
> >
> >
> >
> >
> > ________________________________
> > Never miss a thing. Make Yahoo your homepage.
>
>
> Never miss a thing. Make Yahoo your homepage.
>
>
>
>
>
>
>
> ____________________________________________________________________________________
> ¡Capacidad ilimitada de almacenamiento en tu correo!
> No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> http://correo.espanol.yahoo.com/
>
>
> ------------------------------
> Never miss a thing. Make Yahoo your homepage.<http://us.rd.yahoo.com/evt=51438/*http://www.yahoo.com/r/hs>
>
Mime
View raw message