Return-Path: Delivered-To: apmail-ibatis-user-java-archive@www.apache.org Received: (qmail 26483 invoked from network); 18 Mar 2008 18:11:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 18 Mar 2008 18:11:09 -0000 Received: (qmail 68347 invoked by uid 500); 18 Mar 2008 18:11:04 -0000 Delivered-To: apmail-ibatis-user-java-archive@ibatis.apache.org Received: (qmail 68042 invoked by uid 500); 18 Mar 2008 18:11:03 -0000 Mailing-List: contact user-java-help@ibatis.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-java@ibatis.apache.org Delivered-To: mailing list user-java@ibatis.apache.org Received: (qmail 68031 invoked by uid 99); 18 Mar 2008 18:11:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Mar 2008 11:11:03 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [68.142.207.196] (HELO web32403.mail.mud.yahoo.com) (68.142.207.196) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 18 Mar 2008 18:10:08 +0000 Received: (qmail 33889 invoked by uid 60001); 18 Mar 2008 18:10:26 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=vRNTGSq9QOx8qM950i/nb9S+n8oWY8umHSfjWV4ehxI6EmGqn9OX2/B7dy/Ma6FLs7BNIWLI/ygZCJsEmRBtouwKfKXRQlKvsxZOqiKOycCrKjrghH3kzyoJblFToMhyF/BJhxVoFx29NojIE3EOyoiW2fopSqAMiiYI+tze5Uw=; X-YMail-OSG: NsMtp5oVM1lYo0xhfXUIAkdfayRCfHPKiI0oGntPJcTMLlAdBm7twLvYyYAOaQl4fr_GNa.jCuDQRn_fCMtZCQiA8qzPOu6835jzYvoibLssI_GaH5h93FOH0iwsnBOJyvESePDwEY6V Received: from [85.57.72.125] by web32403.mail.mud.yahoo.com via HTTP; Tue, 18 Mar 2008 11:10:26 PDT X-Mailer: YahooMailRC/902.35 YahooMailWebService/0.7.162 Date: Tue, 18 Mar 2008 11:10:26 -0700 (PDT) From: nch Subject: Re: select * causing " OutOfMemoryError: Java heap space" To: user-java@ibatis.apache.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-870482443-1205863826=:33788" Message-ID: <758263.33788.qm@web32403.mail.mud.yahoo.com> X-Virus-Checked: Checked by ClamAV on apache.org --0-870482443-1205863826=:33788 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable =0AHello, Leon. I tried this with no success.=0ABy the way, I'm using JDK 1= .5.0_13.=0A=0A----- Original Message ----=0AFrom: Leon Liu =0ATo: user-java@ibatis.apache.org=0ASent: Tuesday, March 18, 2008= 6:02:02 PM=0ASubject: Re: select * causing " OutOfMemoryError: Java heap s= pace"=0A=0A pls set enhancementEnabled=3Dfalse and test again.=0ASome cgl= ib + some jdk version cause a big memory problem. I met it inmy project t= hat take me a month to fix it.=0A=0ALeon Liu=0A=0A=0ANathan Maves =E5=86=99= =E9=81=93:did you try defaultAutoCommit=3Dfalse for the connectionpool?=0A = =0A On Tue, Mar 18, 2008 at 9:13 AM, nch wrote:= =0A =0AHi!=0A =0AClinton, answering to your last mail:=0A- = I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currentlyusing) = with MySQL 5.0.45 (Debian). Tried also on Windows with the samedrivers, but= can not remember the MySQL version.=0A- Bellow are, I think I don't miss a= ny, the different relevantconfigurations (some through Spring):=0A =0A- = jdbc.properties: Database connection config=0A =0Ajdbc.driverClassName= =3Dcom.mysql.jdbc.Driver=0Ajdbc.url=3Djdbc:mysql://localhost/wikipedia_arti= cles?createDatabaseIfNotExist=3Dtrue&useUnicode=3Dtrue&characterEnc= oding=3Dutf-8=0Ajdbc.username=3Dx=0Ajdbc.password=3Dy=0A =0A =0A- sql= -map-config.xml: iBatis config file=0A =0A=0A=0A =0A=0A =0A =0A = =0A =0A= =0A =0A =0A- TestingPojoSQL.xml: The operations needed for indexing:= =0A =0A=0A=0A =0A=0A =0A =0A =0A =0A =0A =0A =0A = =0A =0A=0A =0A =0A- applicationContext-resources.xml: Her= e a DBCP data source is defined(I tried also C3P0. Made no difference):=0A = =0A=0A =0A xmlns:= jee=3D"http://www.springframework.org/schema/jee"=0A xsi:schemaLo= cation=3D"http://www.springframework.org/schema/beans http://www.springf= ramework.org/schema/beans/spring-beans-2.0.xsd=0A http://www= .springframework.org/schema/jee http://www.springframework.org/schema/je= e/spring-jee-2.0.xsd">=0A =0A = =0A =0A =0A = classpath:jdbc.properties=0A =0A = =0A =0A =0A =0A = =0A = =0A =0A =0A =0A =0A = =0A <= property name=3D"defaultAutoCommit" value=3D"true"/>=0A =0A = =0A=0A =0A =0A- In applicationContext-dao.xml: the sqlMapClie= nt=0A =0A=0A =0A = xsi:schemaLocation=3D"http://www.springframework.org/schema/beans http:/= /www.springframework.org/schema/beans/spring-beans-2.0.xsd"=0A defaul= t-lazy-init=3D"true">=0A =0A =0A = =0A =0A = =0A =0A =0A = classpath:/sql-map-config.xml=0A = =0A =0A = =0A=0A =0A =0A- In applicationContext-service.xml: This is wh= ere I tell Spring toinstanciate an iBatis based GPS device. =0A =0A=0A =0A =0A org.myorg= .TestingPojo=0A =0A =0A =0A =0A file://compass=0A org.compass.spring.transaction.SpringSyncTransactionF= actory=0A snowball=0A English=0A =0A = 16=0A <= propkey=3D"compass.engine.maxBufferedDocs">10=0A -1=0A 10=0A 10000=0A 5000=0A 60=0A =0A = =0A =0A=0A =0A=0A =0A =0A =0A = =0A getAllTestingPojos=0A =0A =0A =0A=0A = =0A=0A =0A =0A =0A =0A =0A =0A = =0A =0A =0A =0A=0A =0A =0ARegarding y= our previous mail:=0A- I emptied the handleRow method and ran the test - sa= me result. Theexception occurs before handleRow is invoked.=0A- I'm not con= figuring any caches. Is there a default one? Should Iconfig one?=0A- Settin= g fetchSize, different resultSetTypes, use of . You could al= so try a scrollable SesultSet with of SCROL= L_INSENSITIVE or SCROLL_SENSITIVE. =0A =0AFinally, please know that I'= ve used iBATIS to load tens of millions ofrows for ETL and even analysis (N= etFlix Prize) and was able to do sowith a RowHandler or even simply multipl= e calls toqueryForList(String,Object,int offset,int limit). I don't recall= having to even think about the configuration, although I did manage tooptim= ize it to achieve a read/insert performance of 15,000 records persecond bet= ween two databases (on a single machine too). =0A =0AI'm sure there's= something about the configuration here that is causingthe problem, and it = can be solved.=0A =0AI hope one of these helps.=0A =0AClinton = =0A =0A On Mon, Mar 17, 2008 at 1:53 PM, nch wrote:=0A =0AHi, there. :-)=0A = =0AThe thing is, I'm testing a search engine called Compass. One of t= hetests consists of indexing a Wikipedia dump. In this case only the textof= the articles, so I downloaded it and imported it into a MySQLdatabase (abo= ut 650000 records - 1.5GB).=0A =0AYou can configure Compass to acces= s a database and index it's contentsby configuring what they call a GPS Dev= ice. You can do that through anORM such as iBatis, Hibernate or JPA.=0A = =0ACompass provides an implementation of such a GPS Device calledSqlMap= ClientGpsDevice which uses iBatis queryForPaginatedList to getthe results o= f a query and, so, index them. The query is just a "select* from articles_t= able".=0A =0ASo I wired everything up and ran several tests with dif= ferent amountsof data to be indexed and JVM stack space sizes. The result w= as an"OutOfMemoryError: java heap space" error message whenever the size of= the selected rows was bigger than the amount of available heap space (Ican = not index the 650000 records having 1GB of stack).=0A =0AAfter doing= some profiling I concluded iBatis was loading the wholeamount of results i= nto memory. So I decided to try Hibernate, whichworked just fine.=0A = =0AFinally I found queryForPaginatedList was deprecated and that I shouldu= se queryWithRowHandler instead, so I made my own implementation of theSqlMa= pClientGpsDevice based on queryWithRowHandler and tested it, but Ihad the s= ame result (I profiled the application stack usage and I couldsee a line gr= owing steadily and finally crash against the stack ceilingwhile performing = the select statement).=0A =0AShay Banon, Compass project leader, is = planning to patchSqlMapClientGpsDevice by replacing queryForPaginatedList w= ithqueryWithRowHandler, but I'm doubtful it's gonna work.=0A =0AFina= lly you can, of course, devide the select stament into severalselects (and = it works) but, in my opinion, that shouldn't be necessary.=0A You ca= n read more on http://forum.compass-project.org/thread.jspa?threadID=3D2152= 78=0A =0AThanks!=0A =0A =0A ----- Origi= nal Message ----=0AFrom: Clinton Begin =0ATo: user= -java@ibatis.apache.org=0A =0A Sent: Monday, March 17= , 2008 2:22:39 PM=0ASubject: Re: select * causing " OutOfMemoryError: Java = heap space"=0A =0AWow, this thread is interesting. I suppose I cou= ld read the pastemails, but could someone summarize the problem and progres= s to date? I may be able to shed some light on what is happening.=0A = =0AClinton=0A =0A On Sun, Mar 16, 2008 at 11:49 PM, nchwrote:=0A =0AAnot= her thing I forgot. This works perfectly well if we replace iBatisby Hibern= ate, so I don't think this is being caused by the MySQL driver.=0A = =0A ----- Original Message ----=0AFrom: nch =0ATo: user-java@ibatis.apache.org=0A =0A = Sent: Sunday, March 16, 2008 10:18:42 PM=0ASubject: Re: select * = causing " OutOfMemoryError: Java heap space"=0A =0A = =0AWell, I modified the mediumblob into a mediumtext and removed all o= therfields except the id one.=0A =0A -----Original Messag= e ----=0AFrom: nch =0ATo: user-java@ibatis.apache= .org=0ASent: Sunday, March 16, 2008 10:06:34 PM=0ASubject: Re: select * cau= sing " OutOfMemoryError: Java heap space"=0A =0A = =0AYou're probably right. I'm using the table called "text" which contain= sa field of type "mediumblob".=0A CREATE TABLE /*$wgDBprefix*/text= (=0A old_id int unsigned NOT NULL auto_increment,=0A old_text mediumblob= NOT NULL,=0A old_flags tinyblob NOT NULL,=0A PRIMARY KEY old_id (old_id)= =0A) /*$wgDBTableOptions*/ MAX_ROWS=3D10000000 AVG_ROW_LENGTH=3D10240;=0A= =0A=0A=0A=0A You can find it's definition here:=0A http:/= /svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?vie= w=3Dmarkup=0A =0AI'll try to modify that field, first, and see wha= t happens.=0AI've already tried "select * from your_wikipedia_dump limit $s= tart$,$size$" and it seems to work fine, but I see a possible issue withthi= s, you can not remove any of the records you've already read duringthe whol= e process. This might not be an option.=0A =0AThank you.=0A = =0A -----Original Message ----=0AFrom: Larry Meadors =0ATo: user-java@ibatis.apache.org=0ASent: Sunday, March 16,= 2008 4:19:43 PM=0ASubject: Re: select * causing " OutOfMemoryError: Java h= eap space"=0A =0AYeah, I just noticed that - it looks like it's fa= iling in the jdbc=0Adriver when it tries to read a blob, maybe?=0A = =0AI know that some versions of the postgresql jdbc driver load the=0Aenti= re contents of a resultset into a massive byte[][] structure. I=0Awonder if= that is what is happening here? I think you can get the=0Asource for the M= ySQL driver and check that if you want to.=0A =0AIf it is, you may= need to structure your app to take smaller bites of=0Athis elephant.=0A = =0ATry "select * from your_wikipedia_dump limit $start$, $size$" ins= tead=0A- that will limit the size of the results to $size$ rows, starting o= n=0Arow $start$ (the $start$ value is zero-based). If you experiment to=0As= ee how big $size$ can get, you should be able to get pretty decent=0Aperfor= mance out of this.=0A =0APsuedo-code would be like this...=0A = =0Aint start =3D 0;=0Aint size =3D 1000; // see how big you can make t= his to improve performance=0Awhile(still_more_data){=0A still_more_data = =3D index(start, size);=0A start +=3D size;=0A}=0A =0AYour index = method will return true if the query returned any data. If=0Athe query retu= rns no data, it'll return false.=0A =0AI can't imagine what Hibern= ate is doing differently here to make this=0Awork if the JDBC driver is fai= ling when executing the query.=0A =0ALarry=0A =0A = =0AOn Sun, Mar 16, 2008 at 4:12 AM, nch wrote:= =0A>=0A>=0A> Sure. Please, see attached.=0A> I don't think the problem is i= n the RowHandler, though, because the=0A> OutOfMemoryError occurs before in= voking RowHandle#handleRow.=0A>=0A> Cheers=0A>=0A>=0A> ----- Original Messa= ge ----=0A> From: Larry Meadors =0A> To: user-java= @ibatis.apache.org=0A>=0A> Sent: Sunday, March 16, 2008 3:13:27 AM=0A> Subj= ect: Re: select * causing " OutOfMemoryError: Java heap space"=0A>=0A> Can= you post the row handler you are using.=0A>=0A> Larry=0A>=0A>=0A> On Sat, = Mar 15, 2008 at 1:14 PM, nch wrote:=0A> >=0A> >= =0A> > Hi, Nathan.=0A> > I did so, but I'm still having the same issue. Per= haps I'mnot using it=0A> > correctly?=0A> > See my last post to this forum = entry:=0A> >=0A> > http://forum.compass-project.org/thread.jspa?threadID=3D= 215278=0A> >=0A> > See the stack trace:=0A> >=0A> > MemoryError: Java heap = space:=0A> > java.lang.OutOfMemoryError: Java heap space=0A> > at co= m.mysql.jdbc.Buffer.getBytes(Buffer.java:198)=0A> > atcom.mysql.jdbc= .Buffer.readLenByteArray(Buffer.java:318)=0A> > at com.mysql.jdbc.My= sqlIO.nextRow(MysqlIO.java:1366)=0A> > atcom.mysql.jdbc.MysqlIO.read= SingleRowSet(MysqlIO.java:2333)=0A> > atcom.mysql.jdbc.MysqlIO.getRe= sultSet(MysqlIO.java:435)=0A> > at=0A> >com.mysql.jdbc.MysqlIO.readR= esultsForQueryOrUpdate(MysqlIO.java:2040)=0A> > atcom.mysql.jdbc.Mys= qlIO.readAllResults(MysqlIO.java:1443)=0A> > atcom.mysql.jdbc.MysqlI= O.sqlQueryDirect(MysqlIO.java:1777)=0A> > atcom.mysql.jdbc.Connectio= n.execSQL(Connection.java:3249)=0A> > at=0A> >=0A>com.mysql.jdbc.Pre= paredStatement.executeInternal(PreparedStatement.java:1268)=0A> > at= =0A> >com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)= =0A> > at=0A> >=0A>org.apache.commons.dbcp.DelegatingPreparedStateme= nt.execute(DelegatingPreparedStatement.java:169)=0A> > at=0A> >=0A>o= rg.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPrepar= edStatement.java:169)=0A> > at=0A> >=0A>com.ibatis.sqlmap.engine.exe= cution.SqlExecutor.executeQuery(SqlExecutor.java:186)=0A> > at=0A> >= =0A>com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQ= uery(GeneralStatement.java:205)=0A> > at=0A> >=0A>com.ibatis.sqlmap.= engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralS= tatement.java:173)=0A> > at=0A> >=0A>com.ibatis.sqlmap.engine.mappin= g.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.ja= va:133)=0A> > at=0A> >=0A>com.ibatis.sqlmap.engine.impl.SqlMapExecut= orDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)=0A> > = at=0A> >=0A>com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowH= andler(SqlMapSessionImpl.java:156)=0A> > at=0A> >=0A>org.myorg.compa= ss.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRow= Handler.java:33)=0A> > at=0A> >=0A>org.compass.gps.device.AbstractGp= sDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)=0A> > = at=0A> >=0A>org.compass.core.CompassCallbackWithoutResult.doInCompass(Compa= ssCallbackWithoutResult.java:29)=0A> > at=0A> >org.compass.core.Comp= assTemplate.execute(CompassTemplate.java:132)=0A> > at=0A> >=0A>org.= compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161= )=0A> > at=0A> >org.compass.gps.device.AbstractGpsDevice.index(Abstr= actGpsDevice.java:91)=0A> > at=0A> >=0A>org.compass.spring.device.Sp= ringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSy= ncTransactionGpsDeviceWrapper.java:98)=0A> > at=0A> >=0A>org.springf= ramework.transaction.support.TransactionCallbackWithoutResult.doInTransacti= on(TransactionCallbackWithoutResult.java:33)=0A> > at=0A> >=0A>org.s= pringframework.transaction.support.TransactionTemplate.execute(TransactionT= emplate.java:128)=0A> > at=0A> >=0A>org.compass.spring.device.Spring= SyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper= .java:96)=0A> > at=0A> >=0A>org.compass.gps.impl.SingleCompassGps$1.= buildIndexIfNeeded(SingleCompassGps.java:133)=0A> > at=0A> >=0A>org.= compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.= firstStep(DefaultLuceneSearchEngineIndexManager.java:233)=0A> > at= =0A> >=0A>org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineI= ndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)=0A> >= =0A> >=0A> >=0A> >=0A> >=0A> > ----- Original Message ----=0A> > From: Nath= an Maves =0A> > To: user-java@ibatis.apache.org=0A>= >=0A> > Sent: Saturday, March 15, 2008 5:43:04 PM=0A> > Subject: Re: selec= t * causing " OutOfMemoryError: Java heapspace"=0A> >=0A> > this is only m= y 2 cents but I would throw that class out. Write your own=0A> > implementa= tion which uses a row handler. This is the type ofsituation in=0A> > which= a row handler could really help. almost no memorywould be used.=0A> >=0A>= >=0A> >=0A> > On Thu, Mar 13, 2008 at 4:57 PM, nch wrote:=0A> > >=0A> > > Yes, I agree that dividing the query into several= queries should do as=0A> > long as none of them uses more memory than avail= able.Gracias, Carlos.=0A> > >=0A> > > Nevertheless, I looked into=0A> > org= .compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndexand,=0A> > fundamen= tally, what it does is a queryForPaginatedList anditerates=0A> through=0A> = > the resulting PaginatedList with nextPage()=0A> > >=0A> > > I guess that = the reason why pages are kept in memoryafter performing=0A> > paginatedList= .nextPage() is because apaginatedList.previousPage() can be=0A> > issued bu= t, should't it free pages if heap space is runningout?=0A> > >=0A> > >=0A> = > > Carlos de Luna Saenz wrote:=0A> > >=0A> > > = Since we have a similar trouble around here i must saythat you have lot=0A>= > of options when going to compass..=0A> > > The first one is to index "ob= ject by object" by hand,the second one is=0A> to=0A> > use the Gps wich wil= l use the complete bunch of data and makesubindexes=0A> > (making parts of = your "main index" with diferent "maps" foreach subindex=0A> in=0A> > your g= psDriver.=0A> > > if your indexing is as strong as ours i should lookforwar= d for numer 1=0A> or=0A> > number 3 depending of your needs. (More a Compas= s than iBatisissue)=0A> > > Greetings=0A> > > Carlos de Luna=0A> > >=0A> > = > ----- Mensaje original ----=0A> > > De: nch=0A> > >=0A> > > Para: user-ja= va@ibatis.apache.org; larry.meadors@gmail.com=0A> > > Enviado: jue= ves, 13 de marzo, 2008 9:37:04=0A> > > Asunto: Re: select * causing " OutOf= MemoryError: Javaheap space"=0A> > >=0A> > >=0A> > > Sorry. Yes, I need all= the data at one time, because theCompass API=0A> seems=0A> > to be meant i= n that way.=0A> > >=0A> > > Larry Meadors wrote: OK, so the answer to the q= uestionis...[ yes, i=0A> need=0A> > all the data at=0A> > >=0A> > >=0A> > >= =0A> > > one time | no, i do not need all of the data at one time].=0A> > >= =0A> > > Larry=0A> > >=0A> > >=0A> > > On Thu, Mar 13, 2008 at 9:04 AM, nch= wrote:=0A> > > >=0A> > > > Hi!=0A> > > >=0A> > > > I'm doing tests both un= der tomcat and jetty. I set1024MB of heap=0A> space=0A> > in=0A> > > > both= cases. I am using a profiler to look intowhat's happening during=0A> > > >= execution of the indexing process and I can tellthat's the exact=0A> amoun= t=0A> > it=0A> > > > displays for the VM.=0A> > > >=0A> > > > As I describe= in the Compass user forum (see linkbellow), I'm=0A> indexing=0A> > a=0A> >= > > Wikipedia dump of about 650000-675000 records.That's sort of 1-1.5GB= =0A> of=0A> > > > data.=0A> > > >=0A> > > > Hibernate can handle all that d= ata, I guess it'susing some sort of=0A> > > > pagination. iBatis seems to d= ivide the selectstatement into several=0A> > select=0A> > > > statements, b= ut it also seems each of suchstatements surpasses the=0A> > limit of=0A> > = > > 1024MB when trying to index only 300000 of thedatabase records.=0A> > >= >=0A> > > > Thank you=0A> > > >=0A> > > >=0A> > > >=0A> > > > Chris Lamey = wrote:=0A> > > > What is you JVM heap size set to and how much datais retur= ned by=0A> select=0A> > *=0A> > > > from table?=0A> > > >=0A> > > > If you'= re trying to pull back 1G worth of data intoa JVM with a heap=0A> > size=0A= > > > > set to 64M, you will hit the heap limit prettyquick.=0A> > > >=0A> = > > >=0A> > > > -----Original Message-----=0A> > > > From: nch [mailto:unde= rscore_dot@yahoo..com]=0A> > > > Sent: Thu 3/13/2008 8:03 AM=0A> > > > To: = user-java@ibatis.apache.org=0A> > > > Subject: select * causing " OutOfMemo= ryError: Javaheap space"=0A> > > >=0A> > > >=0A> > > > Hi, everybody.=0A> >= > > I'm testing the Compass search engine and, inparticular, how to index= =0A> a=0A> > big=0A> > > > set of documents from a table in a MySQL databas= e.In order to do this=0A> I=0A> > > > issue a "select * from table_name" us= ing iBatis,but this seems to be=0A> > > > causing the application to use al= l heap spaceavailable.=0A> > > > I added a new entry into Compass user foru= m withdetails:=0A> > > > http://forum.compass-project.org/thread.jspa?threa= dID=3D215278=0A> > > >=0A> > > > Can you figure out why is this happening?= =0A> > > >=0A> > > > Many thanks=0A> > > >=0A> > > >=0A> > > > ------------= ---------------------=0A> > > > Be a better friend, newshound, and know-it-= all withYahoo! Mobile. Try=0A> > it=0A> > > > now.=0A> > > >=0A> > > >=0A> = > > >=0A> > > >=0A> > > > ________________________________=0A> > > > Never = miss a thing. Make Yahoo your homepage.=0A> > >=0A> > >=0A> > > Never miss = a thing. Make Yahoo your homepage.=0A> > >=0A> > >=0A> > >=0A> > >=0A> > >= =0A> > >=0A> > >=0A> >=0A>_________________________________________________= ___________________________________=0A> > > =C2=A1Capacidad ilimitada de al= macenamiento en tu correo!=0A> > > No te preocupes m=C3=A1s por el espacio = de tu cuenta conCorreo Yahoo!:=0A> > > http://correo.espanol.yahoo.com/=0A>= > >=0A> > >=0A> > >=0A> > >=0A> > >=0A> > >=0A> > >=0A> > > ______________= __________________=0A> > Never miss a thing. Make Yahoo your homepage.=0A> = >=0A> >=0A> >=0A> >=0A> > ________________________________=0A> > Never mis= s a thing. Make Yahoo your homepage.=0A>=0A>=0A> _________________________= _______=0A>=0A> Be a better friend, newshound, and know-it-all with Yahoo! = Mobile.Try it=0A> now.=0A =0A =0A =0A = =0A =0A Looking for last minute shopping deals? Find the= m fast with Yahoo! Search.=0A =0A =0A =0A = =0A Be a better friend, newshound, and know-it-allwith Yahoo! M= obile. Try it now.=0A =0A =0A =0A =0A = =0A =0A Looking for last minute shoppi= ng deals? Find them fast with Yahoo! Search.=0A =0A =0A = =0A =0A =0A =0A =0A =0A = =0A =0A =0A Never miss a thing. Make Ya= hoo your homepage. =0A =0A =0A =0A =0A= =0A =0A =0A =0A =0A =0A =0A =0A =0A = =0A =0A Be a better friend, newshound, and know-it-allwith Yahoo!= Mobile. Try it now.=0A =0A =0A =0A=0A=0A=0A=0A=0A=0A=0A _____= ___________________________________________________________________________= ____=0ABe a better friend, newshound, and =0Aknow-it-all with Yahoo! Mobile= . Try it now. http://mobile.yahoo.com/;_ylt=3DAhu06i62sR8HDtDypao8Wcj9tAc= J =0A --0-870482443-1205863826=:33788 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Hello, Leon. I tried this with no success.
B= y the way, I'm using JDK 1.5.0_13.

----- Original Message --= --
From: Leon Liu <ljw2083@alibaba-inc.com>
To: user-java@ibati= s.apache.org
Sent: Tuesday, March 18, 2008 6:02:02 PM
Subject: Re: se= lect * causing " OutOfMemoryError: Java heap space"

=0A=0A=0A =0A= =0Apls set enhancementEnabled=3Dfalse  and test again.
=0ASome cgli= b + some jdk version cause a big memory problem.   I met it in=0A= my project that take me a month to fix it.
=0A
=0ALeon Liu
=0A
= =0A
=0ANathan Maves =E5=86=99=E9=81=93:=0A
did = you try defaultAutoCommit=3Dfalse for the connection=0Apool?
=0A
= =0A
On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com&= gt;=0Awrote:
=0A
=0A
=0A
=0A

=0AHi!
=0A
=0AC= linton, answering to your last mail:
=0A- I tried MySQL driver versions = 5.0.5 and 5.0.8 (the one I'm currently=0Ausing) with MySQL 5.0.45 (Debian).= Tried also on Windows with the same=0Adrivers, but can not remember the My= SQL version.
=0A- Bellow are, I think I don't miss any, the different re= levant=0Aconfigurations (some through Spring):
=0A
=0A- jdbc.prop= erties: Database connection config
=0A
=0Ajdbc.driverClassName=3D= com.mysql.jdbc.Driver
=0Ajdbc.url=3Djdbc:mysql://localhost/wikipedia_art= icles?createDatabaseIfNotExist=3Dtrue&amp;useUnicode=3Dtrue&amp;cha= racterEncoding=3Dutf-8
=0Ajdbc.username=3Dx
=0Ajdbc.password=3Dy
= =0A
=0A
=0A- sql-map-config.xml: iBatis config file
=0A =
=0A<?xml version=3D"1.0" encoding=3D"UTF-8"?>
=0A<!DOCTYPE= sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map=0AConfig 2.0//EN"=0A     "http://ibatis.apac= he.org/dtd/sql-map-config-2.dtd">
=0A
=0A<sqlMap= Config>
=0A    <settings enhancementEnabled=3D"true= " maxTransactions=3D"20"=0AmaxRequests=3D"32" maxSessions=3D"10"/>
= =0A
=0A    <sqlMap resource=3D"sqlmaps/TestingPojo= SQL.xml"/>
=0A
=0A</sqlMapConfig>
=0A
=0A <= br>=0A- TestingPojoSQL.xml: The operations needed for indexing:
=0A <= br>=0A<?xml version=3D"1.0" encoding=3D"UTF-8"?>
=0A<!DOCTYPE s= qlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "htt= p://ibatis.apache.org/dtd/sql-map-2.dtd">
=0A
=0A&l= t;sqlMap namespace=3D"TestingPojoSQL">
=0A    <type= Alias alias=3D"article"=0Atype=3D"org.myorg.model.TestingPojo"/>
=0A =
=0A    <resultMap id=3D"articleResult" class=3D"ar= ticle">
=0A        <result prop= erty=3D"id" column=3D"id"/>
=0A      &n= bsp; <result property=3D"contents" column=3D"contents"/>
=0A =    </resultMap>
=0A
=0A    <sel= ect id=3D"getAllTestingPojos" resultMap=3D"articleResult"=0AfetchSize=3D"10= 0" resultSetType=3D"FORWARD_ONLY">
=0A    <![CDATA[=
=0A        select * from articles=0A    ]]>
=0A    </select>
= =0A   
=0A</sqlMap>
=0A
=0A
=0A- = applicationContext-resources.xml: Here a DBCP data source is defined=0A(I t= ried also C3P0. Made no difference):
=0A
=0A<?xml version=3D"1= .0" encoding=3D"UTF-8"?>
=0A
=0A  &= nbsp;     xmlns:jee=3D"http://www.springf= ramework.org/schema/jee"
=0A     = ;   xsi:schemaLocation=3D"http://www.springframew= ork.org/schema/beans=0A http:= //www.springframework.org/schema/beans/spring-beans-2.0.xsd
= =0A           &n= bsp; http://www.springframework.org/schema/jee=0A http://www.springframework.org/schema/jee/spri= ng-jee-2.0.xsd">
=0A
=0A    <bean= id=3D"propertyConfigurer"=0Aclass=3D"org.springframework.beans.factory.con= fig.PropertyPlaceholderConfigurer">
=0A     =    <property name=3D"locations">
=0A   &n= bsp;        <list>
=0A &nb= sp;            =   <value>classpath:jdbc.properties</value>
=0A &nb= sp;          </list>
= =0A        </property>
=0A = ;   </bean>
=0A
=0A    <bean id= =3D"dataSource"=0Aclass=3D"org.apache.commons.dbcp.BasicDataSource"=0Adestr= oy-method=3D"close">
=0A        &l= t;property name=3D"driverClassName"=0Avalue=3D"${jdbc.driverClassName}"/>= ;
=0A        <property name=3D"url= " value=3D"${jdbc.url}"/>
=0A      &nbs= p; <property name=3D"username" value=3D"${jdbc.username}"/>
=0A&nb= sp;       <property name=3D"password" valu= e=3D"${jdbc.password}"/>
=0A       = ; <property name=3D"maxActive" value=3D"100"/>
=0A  &nbs= p;     <property name=3D"maxWait" value=3D"1000"/>= ;
=0A        <property name=3D"poo= lPreparedStatements" value=3D"true"/>
=0A    &nbs= p;   <property name=3D"defaultAutoCommit" value=3D"true"/><= br>=0A    </bean>
=0A
=0A</beans>
= =0A
=0A
=0A- In applicationContext-dao.xml: the sqlMapClient<= br>=0A
=0A<?xml version=3D"1.0" encoding=3D"UTF-8"?>
=0A =
=0A        xsi:sch= emaLocation=3D"http://www.springframework.org/schema/beans<= /a>=0A http://www.springframework.o= rg/schema/beans/spring-beans-2.0.xsd"
=0A   &n= bsp;   default-lazy-init=3D"true">
=0A    =0A    <bean id=3D"transactionManager"=0Aclass=3D"org.s= pringframework.jdbc.datasource.DataSourceTransactionManager">
=0A&nbs= p;       <property name=3D"dataSource" ref= =3D"dataSource"/>
=0A    </bean>
=0A
= =0A    <bean id=3D"sqlMapClient"=0Aclass=3D"org.springfra= mework.orm.ibatis.SqlMapClientFactoryBean">
=0A   &nbs= p;    <property name=3D"configLocation">
=0A &= nbsp;          <value>cl= asspath:/sql-map-config.xml</value>
=0A    &nb= sp;   </property>
=0A      = ;  <property name=3D"dataSource" ref=3D"dataSource"/>
=0A&nbs= p;   </bean>
=0A</beans>
=0A
=0A
= =0A- In applicationContext-service.xml: This is where I tell Spring to=0Ain= stanciate an iBatis based GPS device.
=0A
=0A<bean id=3D"comp= ass" class=3D"org.compass.spring.LocalCompassBean">
=0A  &n= bsp; <property name=3D"classMappings">
=0A    =     <list>
=0A      &= nbsp;     <value>org.myorg.TestingPojo</value&= gt;
=0A        </list>
=0A&n= bsp;   </property>
=0A    <property na= me=3D"compassSettings">
=0A       = <props>
=0A         =    <prop=0Akey=3D"compass.engine.connection">file://compass</prop>
= =0A            <p= rop=0Akey=3D"compass.transaction.factory">org.compass.spring.transaction= .SpringSyncT=0AransactionFactory</prop>
=0A    = ;        <prop=0Akey=3D"compass.engin= e.analyzer.default.type">snowball</prop>
=0A   &= nbsp;        <prop key=3D"compass.engine.analyzer.default.name">English</prop>
= =0A            <!= -- Mainly default values -->
=0A      &= nbsp;     <prop=0Akey=3D"compass.engine.ramBufferSiz= e">16</prop>
=0A        = ;    <prop=0Akey=3D"compass.engine.maxBufferedDocs">10= </prop>
=0A         &= nbsp;  <prop=0Akey=3D"compass.engine.maxBufferedDeletedTerms">-1= </prop>
=0A         &= nbsp;  <prop key=3D"compass.engine.mergeFactor">10</prop><= br>=0A            &l= t;prop=0Akey=3D"compass.engine.maxFieldLength">10000</prop>
=0A=             <prop= =0Akey=3D"compass.engine.cacheIntervalInvalidation">5000</prop>=0A            <= prop=0Akey=3D"compass.engine.indexManagerScheduleInterval">60</prop&g= t;
=0A        </props>
=0A&n= bsp;   </property>
=0A    <property na= me=3D"transactionManager" ref=3D"transactionManager"=0A/>
=0A</bea= n>
=0A
=0A<bean id=3D"iBatisGpsDevice"=0Aclass=3D"org.compa= ss.gps.device.ibatis.SqlMapClientGpsDevice">
=0A    &l= t;property name=3D"name" value=3D"iBatisDevice" />
=0A  &nb= sp; <property name=3D"sqlMapClient" ref=3D"sqlMapClient" />
=0A&nb= sp;   <property name=3D"selectStatementsIds">
=0A &= nbsp;      <list>
=0A   &n= bsp;        <value>getAllTestingPo= jos</value>
=0A        </lis= t>
=0A    </property>
=0A    &= lt;property name=3D"pageSize" value=3D"50" />
=0A</bean>
=0A=
=0A<bean id=3D"compassGps" class=3D"org.compass.gps.impl.SingleC= ompassGps"=0Ainit-method=3D"start" destroy-method=3D"stop">
=0A =    <property name=3D"compass">
=0A   &nbs= p;    <ref bean=3D"compass" />
=0A   = ; </property>
=0A    <property name=3D"gpsDevice= s">
=0A        <list>
=0A=             <bean= =0Aclass=3D"org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper= ">
=0A          &nb= sp;     <property name=3D"gpsDevice" ref=3D"iBatisGp= sDevice"=0A/>
=0A        &nbs= p;   </bean>
=0A      &nbs= p; </list>
=0A    </property>
=0A</bean= >
=0A
=0A
=0ARegarding your previous mail:
=0A- I em= ptied the handleRow method and ran the test - same result. The=0Aexception = occurs before handleRow is invoked.
=0A- I'm not configuring any caches.= Is there a default one? Should I=0Aconfig one?
=0A- Setting fetchSize, = different resultSetTypes, use of <select> or=0A<statement> elem= ents didn't make any differences.
=0A- Additionally, I tried truncating = the contents of each Wikipedia=0Aarticle to 500 characters, and I was able = to index all the 650000 rows,=0Aso the number of records being selected is = not the only factor here (I=0Ajust wanted to make sure).
=0A- I can imag= ine others (including you) have successfully made it with=0Alarge amounts o= f data (I'm new to iBatis and Compass/Lucene) so I'm=0Asure there's a solut= ion.
=0A
=0ALots of thanks.
=0A
=0A
=0A
= =0A
=0A
=0A
----- Original Messa= ge ----
=0AFrom: Clinton Begin <clinton.begin@gmail.com>
=0ATo: user-java@ibatis.apache.org
=0A =0A
=0A
Sent: Monday, March 17, 2008 11= :18:59 PM
=0ASubject: Re: select * causing " OutOfMemoryError: Java heap= space"
=0A
=0AWow, I just read that other thread on the Compass = site.... This is not=0Adirectly an iBATIS problem at all...
=0A
= =0Aat=0Acom.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.= java:1268)=0Aat com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.= java:843)
=0A
=0AThis line tells me that the execution is never r= eturning from the=0Aprepared statement.  iBATIS never has a chance to = even see the=0AResultSet, let alone get to the RowHandler. So swapping out = the=0AGpsDevice here won't make any difference at all. This is odd behavior= =0Afor a JDBC driver and tells me that something is very wrong with the=0Ac= onfiguration. 
=0A
=0AWhich version of the JDBC driver are = you using?  What are the rest of=0Athe iBATIS convfiguration details?<= br>=0A
=0AClinton
=0A
=0A
=0A
On Mon, Mar 17, 2008 at 4:12 PM, Clinton=0ABegin <clinton.begin@gmail.com>=0Awrote:=0A
Thanks= =0Afor the summary.  A few things:
=0A
=0A* PaginatedList = was definitely the wrong solution for that, so I'm glad=0Ait's gone.
=0A=
=0A* A RowHandler is possibly the right solution, but will only s= tore as=0Amany records in memory as you tell it to.  iBATIS does not j= ust=0Aarbitrarily keep records around.  Quite the opposite actually.&n= bsp; Unless=0Ayou keep a reference or configure a cache, iBATIS will not ke= ep the=0Aobject around at all. 
=0A
=0ALooking at your im= plementation, there are a couple of things going on...
=0A
=0As= ession.create(o);
=0Aif(currentItem=3D=3D pageSize){
=0A  if (lo= g.isDebugEnabled()) {
=0A    log.debug("Indexing page num= ber ["+ pageCount++ + "]");
=0A  }
=0Asession.evictAll();
=0A=
=0AThings to Try:
=0A
=0A1)  These lines of code= must be treated as guilty until proven innocent=0A(especially the logging)= .  The easiest way to do that is to delete=0Athem.  Create an emp= ty handler/GpsDevice and run the test to see what=0Ahappens.
=0A =0A2) Look at the iBATIS configuration files.  Is a cache configured= for=0Athis query?  What kind of cache?
=0A
=0A3) You coul= d try multiple queries of fixed sizes with=0AqueryForList(String,Object,int= offset,int limit) ... but I can't see=0Ahow that would be better than a Ro= wHandler.
=0A
=0A4) Next, just in case the driver's default fet= ch size is ridiculously=0Ahigh (or unlimited), try setting <select ... f= etchSize=3D"100"/>. =0AYou could also try a scrollable SesultSet wi= th <statement ...=0AresultSetType /> of  SCROLL_INSENSITIVE or S= CROLL_SENSITIVE.
=0A
=0AFinally, please know that I've used iB= ATIS to load tens of millions of=0Arows for ETL and even analysis (NetFlix = Prize) and was able to do so=0Awith a RowHandler or even simply multiple ca= lls to=0AqueryForList(String,Object,int offset,int limit).  I don't re= call=0Ahaving to even think about the configuration, although I did manage = to=0Aoptimize it to achieve a read/insert performance of 15,000 records per= =0Asecond between two databases (on a single machine too). 
=0A =
=0AI'm sure there's something about the configuration here that is c= ausing=0Athe problem, and it can be solved.
=0A
=0AI hope one o= f these helps.
=0A
=0AClinton=0A=
=0A

=0A
=0A
On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com>=0Awrote:
=0A =0A
=0A =
=0A

=0AHi, there. :-)
=0A
= =0AThe thing is, I'm testing a search engine called Compass. One of the=0At= ests consists of indexing a Wikipedia dump. In this case only the text=0Aof= the articles, so I downloaded it and imported it into a MySQL=0Adatabase (= about 650000 records - 1.5GB).
=0A
=0AYou can configure Compa= ss to access a database and index it's contents=0Aby configuring what they = call a GPS Device. You can do that through an=0AORM such as iBatis, Hiberna= te or JPA.
=0A
=0ACompass provides an implementation of such = a GPS Device called=0ASqlMapClientGpsDevice which uses iBatis queryForPagin= atedList to get=0Athe results of a query and, so, index them. The query is = just a "select=0A* from articles_table".
=0A
=0ASo I wired ev= erything up and ran several tests with different amounts=0Aof data to be in= dexed and JVM stack space sizes. The result was an=0A"OutOfMemoryError: jav= a heap space" error message whenever the size of=0Athe selected rows was bi= gger than the amount of available heap space (I=0Acan not index the 650000 = records having 1GB of stack).
=0A
=0AAfter doing some profili= ng I concluded iBatis was loading the whole=0Aamount of results into memory= . So I decided to try Hibernate, which=0Aworked just fine.
=0A =0AFinally I found queryForPaginatedList was deprecated and that I should= =0Ause queryWithRowHandler instead, so I made my own implementation of the= =0ASqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I= =0Ahad the same result (I profiled the application stack usage and I could= =0Asee a line growing steadily and finally crash against the stack ceiling= =0Awhile performing the select statement).
=0A
= =0AShay Banon, Compass project leader, is planning to patch=0ASqlMapClientG= psDevice by replacing queryForPaginatedList with=0AqueryWithRowHandler, but= I'm doubtful it's gonna work.
=0A
=0AFinally you can, of cou= rse, devide the select stament into several=0Aselects (and it works) but, i= n my opinion, that shouldn't be necessary.
=0A You can read= more on http://forum.compass-project.or= g/thread.jspa?threadID=3D215278
=0A
=0AThanks!
= =0A
=0A
=0A
=0A
----- Orig= inal Message ----
=0AFrom: Clinton Begin <clinton.begin@gmail.com>
=0ATo: user-java@ibatis.apache.org
= =0A
=0A
=0A
Sent: Monday, March 17, 20= 08 2:22:39 PM
=0ASubject: Re: select * causing " OutOfMemoryError: Java = heap space"
=0A
=0AWow, this thread is interesting. &nbs= p; I suppose I could read the past=0Aemails, but could someone summarize th= e problem and progress to date? =0AI may be able to shed some light on= what is happening.
=0A
=0AClinton
=0A
=0A =
On Sun, Mar 16, 2008 at 11:49 PM, nch=0A<<= a rel=3D"nofollow" ymailto=3D"mailto:underscore_dot@yahoo.com" target=3D"_b= lank" href=3D"mailto:underscore_dot@yahoo.com">underscore_dot@yahoo.com= >=0Awrote:
=0A
=0A
=0A
=0A
=0AAnother thing I forgot. This works perfectly well if we replace iBatis= =0Aby Hibernate, so I don't think this is being caused by the MySQL driver.=
=0A
=0A
=0A
----- Original = Message ----
=0AFrom: nch <underscore_dot@yahoo.com>
=0ATo: user-java@ibatis.apache.org
=0A =
=0A
=0A
Sent: Sunday, March 16, 2008 10:= 18:42 PM
=0ASubject: Re: select * causing " OutOfMemoryError: Java heap = space"
=0A
=0A
=0A

= =0AWell, I modified the mediumblob into a mediumtext and removed all other= =0Afields except the id one.
=0A
=0A
-----= =0AOriginal Message ----
=0AFrom: nch <underscore_dot@yahoo.com>
=0ATo: user-java@ibatis.apache.org
= =0ASent: Sunday, March 16, 2008 10:06:34 PM
=0ASubject: Re: select * cau= sing " OutOfMemoryError: Java heap space"
=0A
=0A =
=0A

=0AYou're probably right. I'm using the ta= ble called "text" which contains=0Aa field of type "mediumblob".
=0A = 
CREATE TABLE /*$wgDBprefix*/text (
  old_id int unsigned NOT =
NULL auto_increment,
  old_text mediumblob NOT NULL,
  old_flags tiny=
blob NOT NULL,
  PRIMARY KEY old_id (old_id)
) /*$wgDBTableOptions*/ =
MAX_ROWS=3D10000000 AVG_ROW_LENGTH=3D10240;




          =0AYou can find it's definition here:
=0A          http://svn.wikimedia.o=
rg/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=3Dmarkup=0A          
=0AI'll try to modify that field, first, and see =
what happens.
=0AI've already tried "select * from your_wikipedia_dump l=
imit $start$,=0A$size$" and it seems to work fine, but I see a possible iss=
ue with=0Athis, you can not remove any of the records you've already read d=
uring=0Athe whole process. This might not be an option.
=0A          =0AThank you.
=0A          
=0A          
-----=0AOriginal Me=
ssage ----
=0AFrom: Larry Meadors <larry.meadors@gmail.com>
=0ATo: user-java@ibatis.apache.org
=0ASent=
: Sunday, March 16, 2008 4:19:43 PM
=0ASubject: Re: select * causing " O=
utOfMemoryError: Java heap space"
=0A          
=0AYeah, I just notic=
ed that - it looks like it's failing in the jdbc
=0Adriver when it tries=
 to read a blob, maybe?
=0A          
=0AI know that some versions of=
 the postgresql jdbc driver load the
=0Aentire contents of a resultset i=
nto a massive byte[][] structure. I
=0Awonder if that is what is happeni=
ng here? I think you can get the
=0Asource for the MySQL driver and chec=
k that if you want to.
=0A          
=0AIf it is, you may need to str=
ucture your app to take smaller bites of
=0Athis elephant.
=0A       =
   
=0ATry "select * from your_wikipedia_dump limit $start$, $size$" ins=
tead
=0A- that will limit the size of the results to $size$ rows, starti=
ng on
=0Arow $start$ (the $start$ value is zero-based). If you experimen=
t to
=0Asee how big $size$ can get, you should be able to get pretty dec=
ent
=0Aperformance out of this.
=0A          
=0APsuedo-code would=
 be like this...
=0A          
=0Aint start =3D 0;
=0Aint size =3D=
 1000; // see how big you can make this to improve performance
=0Awhile(=
still_more_data){
=0A  still_more_data =3D index(start, size);
=
=0A  start +=3D size;
=0A}
=0A          
=0AYour index method=
 will return true if the query returned any data. If
=0Athe query return=
s no data, it'll return false.
=0A          
=0AI can't imagine what =
Hibernate is doing differently here to make this
=0Awork if the JDBC dri=
ver is failing when executing the query.
=0A          
=0ALarry
=
=0A          
=0A          
=0AOn Sun, Mar 16, 2008 at 4:12 AM, nch &=
lt;underscore_dot@yahoo.c=
om>=0Awrote:
=0A>
=0A>
=0A> Sure. Please, see atta=
ched.
=0A> I don't think the problem is in the RowHandler, though, be=
cause the
=0A> OutOfMemoryError occurs before invoking RowHandle#hand=
leRow.
=0A>
=0A> Cheers
=0A>
=0A>
=0A> ----- =
Original Message ----
=0A> From: Larry Meadors <larry.meadors@gmail.com>
=0A> To: =
user-java@ibatis.ap=
ache.org
=0A>
=0A> Sent: Sunday, March 16, 2008 3:13:27 AM<=
br>=0A> Subject: Re: select * causing " OutOfMemoryError: Java heap spac=
e"
=0A>
=0A>  Can you post the row handler you are using.<=
br>=0A>
=0A> Larry
=0A>
=0A>
=0A> On Sat, Mar 15=
, 2008 at 1:14 PM, nch <=
underscore_dot@yahoo.com>=0Awrote:
=0A> >
=0A> >=0A> > Hi, Nathan.
=0A> > I did so, but I'm still having t=
he same issue. Perhaps I'm=0Anot using it
=0A> > correctly?
=0A=
> > See my last post to this forum entry:
=0A> >
=0A> =
> http://forum.compass-project.org/th=
read.jspa?threadID=3D215278
=0A> >
=0A> > See the sta=
ck trace:
=0A> >
=0A> > MemoryError: Java heap space:
=
=0A> > java.lang.OutOfMemoryError: Java heap space
=0A> >&nb=
sp;       at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)=

=0A> >        at=0Acom.mysql.jdbc.Buffer.read=
LenByteArray(Buffer.java:318)
=0A> >        at=
 com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
=0A> >  &n=
bsp;     at=0Acom.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.jav=
a:2333)
=0A> >        at=0Acom.mysql.jdbc.Mysq=
lIO.getResultSet(MysqlIO.java:435)
=0A> >      &nbs=
p; at
=0A> >=0Acom.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(=
MysqlIO.java:2040)
=0A> >        at=0Acom.mysq=
l.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
=0A> >  &nbs=
p;     at=0Acom.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:17=
77)
=0A> >        at=0Acom.mysql.jdbc.Connecti=
on.execSQL(Connection.java:3249)
=0A> >       =
 at
=0A> >
=0A>=0Acom.mysql.jdbc.PreparedStatement.executeIn=
ternal(PreparedStatement.java:1268)
=0A> >      &nb=
sp; at
=0A> >=0Acom.mysql.jdbc.PreparedStatement.execute(PreparedS=
tatement.java:843)
=0A> >        at
=0A>=
 >
=0A>=0Aorg.apache.commons.dbcp.DelegatingPreparedStatement.exec=
ute(DelegatingPreparedStatement.java:169)
=0A> >    &nbs=
p;   at
=0A> >
=0A>=0Aorg.apache.commons.dbcp.Delegatin=
gPreparedStatement.execute(DelegatingPreparedStatement.java:169)
=0A>=
 >        at
=0A> >
=0A>=0Acom.ibatis=
.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
=
=0A> >        at
=0A> >
=0A>=0Acom=
.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(Ge=
neralStatement.java:205)
=0A> >        at
=
=0A> >
=0A>=0Acom.ibatis.sqlmap.engine.mapping.statement.Genera=
lStatement.executeQueryWithCallback(GeneralStatement.java:173)
=0A> &=
gt;        at
=0A> >
=0A>=0Acom.ibatis.s=
qlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(=
GeneralStatement.java:133)
=0A> >        at=0A> >
=0A>=0Acom.ibatis.sqlmap.engine.impl.SqlMapExecutorDele=
gate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
=0A> >&n=
bsp;       at
=0A> >
=0A>=0Acom.ibatis.sqlmap=
.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:1=
56)
=0A> >        at
=0A> >
=0A>=
;=0Aorg.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapCli=
entGpsDeviceWithRowHandler.java:33)
=0A> >      &nb=
sp; at
=0A> >
=0A>=0Aorg.compass.gps.device.AbstractGpsDevic=
e$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
=0A> >&nbs=
p;       at
=0A> >
=0A>=0Aorg.compass.core.Co=
mpassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29=
)
=0A> >        at
=0A> >=0Aorg.compa=
ss.core.CompassTemplate.execute(CompassTemplate.java:132)
=0A> >&n=
bsp;       at
=0A> >
=0A>=0Aorg.compass.gps.i=
mpl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
=0A> =
>        at
=0A> >=0Aorg.compass.gps.device=
.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
=0A> >  &=
nbsp;     at
=0A> >
=0A>=0Aorg.compass.spring.devi=
ce.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(Spr=
ingSyncTransactionGpsDeviceWrapper.java:98)
=0A> >    &n=
bsp;   at
=0A> >
=0A>=0Aorg.springframework.transaction=
.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallba=
ckWithoutResult.java:33)
=0A> >        at
=
=0A> >
=0A>=0Aorg.springframework.transaction.support.Transacti=
onTemplate.execute(TransactionTemplate.java:128)
=0A> >  &nbs=
p;     at
=0A> >
=0A>=0Aorg.compass.spring.device.=
SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceW=
rapper.java:96)
=0A> >        at
=0A> &g=
t;
=0A>=0Aorg.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(=
SingleCompassGps.java:133)
=0A> >        at=0A> >
=0A>=0Aorg.compass.core.lucene.engine.manager.DefaultLu=
ceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexMana=
ger.java:233)
=0A> >        at
=0A> >=

=0A>=0Aorg.compass.core.lucene.engine.manager.DefaultLuceneSearchEng=
ineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)=0A> >
=0A> >
=0A> >
=0A> >
=0A> &=
gt;
=0A> > ----- Original Message ----
=0A> > From: Natha=
n Maves <nathan.maves@gmail.=
com>
=0A> > To: user-java@ibatis.apache.org
=0A> >
=0A> > =
Sent: Saturday, March 15, 2008 5:43:04 PM
=0A> > Subject: Re: sele=
ct * causing " OutOfMemoryError: Java heap=0Aspace"
=0A> >
=0A&=
gt; >  this is only my 2 cents but I would throw that class out.&nb=
sp;=0AWrite your own
=0A> > implementation which uses a row handle=
r.  This is the type of=0Asituation in
=0A> > which a row han=
dler could really help.  almost no memory=0Awould be used.
=0A> =
>
=0A> >
=0A> >
=0A> > On Thu, Mar 13, 2008 a=
t 4:57 PM, nch <undersco=
re_dot@yahoo.com>=0Awrote:
=0A> > >
=0A> > >=
 Yes, I agree that dividing the query into several=0Aqueries should do as=0A> > long as none of them uses more memory than available.=0AGrac=
ias, Carlos.
=0A> > >
=0A> > > Nevertheless, I look=
ed into
=0A> > org.compass.gps.device.ibatis.SqlMapClientGpsDevice=
#doIndex=0Aand,
=0A> > fundamentally, what it does is a queryForPa=
ginatedList and=0Aiterates
=0A> through
=0A> > the resulting=
 PaginatedList with nextPage()
=0A> > >
=0A> > > I =
guess that the reason why pages are kept in memory=0Aafter performing
=
=0A> > paginatedList.nextPage() is because a=0ApaginatedList.previous=
Page() can be
=0A> > issued but, should't it free pages if heap sp=
ace is running=0Aout?
=0A> > >
=0A> > >
=0A> =
> > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>=0Awrote:
=0A> > >=0A> > > Since we have a similar trouble around here i must say=
=0Athat you have lot
=0A> > of options when going to compass..
=
=0A> > > The first one is to index "object by object" by hand,=0At=
he second one is
=0A> to
=0A> > use the Gps wich will use th=
e complete bunch of data and make=0Asubindexes
=0A> > (making part=
s of your "main index" with diferent "maps" for=0Aeach subindex
=0A> =
in
=0A> > your gpsDriver.
=0A> > > if your indexing is=
 as strong as ours i should look=0Aforward for numer 1
=0A> or
=0A=
> > number 3 depending of your needs. (More a Compass than iBatis=0Ai=
ssue)
=0A> > > Greetings
=0A> > > Carlos de Luna=0A> > >
=0A> > > ----- Mensaje original ----
=0A&=
gt; > > De: nch
=0A> > >
=0A> > > Para: user-java@ibatis.apache.or=
g;=0A          larry.mead=
ors@gmail.com
=0A> > > Enviado: jueves, 13 de marzo, 2008 9=
:37:04
=0A> > > Asunto: Re: select * causing " OutOfMemoryError=
: Java=0Aheap space"
=0A> > >
=0A> > >
=0A> &=
gt; > Sorry. Yes, I need all the data at one time, because the=0ACompass=
 API
=0A> seems
=0A> > to be meant in that way.
=0A> &=
gt; >
=0A> > > Larry Meadors wrote: OK, so the answer to the=
 question=0Ais...[ yes, i
=0A> need
=0A> > all the data at=0A> > >
=0A> > >
=0A> > >
=0A> &g=
t; > one time | no, i do not need all of the data at one time=0A].
=
=0A> > >
=0A> > > Larry
=0A> > >
=0A>=
; > >
=0A> > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote=
:
=0A> > > >
=0A> > > > Hi!
=0A> > &=
gt; >
=0A> > > > I'm doing tests both under tomcat and je=
tty. I set=0A1024MB of heap
=0A> space
=0A> > in
=0A> =
> > > both cases. I am using a profiler to look into=0Awhat's happ=
ening during
=0A> > > > execution of the indexing process an=
d I can tell=0Athat's the exact
=0A> amount
=0A> > it
=0A=
> > > > displays for the VM.
=0A> > > >
=0A&g=
t; > > > As I describe in the Compass user forum (see link=0Abello=
w), I'm
=0A> indexing
=0A> > a
=0A> > > > Wik=
ipedia dump of about 650000-675000 records.=0AThat's sort of 1-1.5GB
=0A=
> of
=0A> > > > data.
=0A> > > >
=0A>=
; > > > Hibernate can handle all that data, I guess it's=0Ausing s=
ome sort of
=0A> > > > pagination. iBatis seems to divide th=
e select=0Astatement into several
=0A> > select
=0A> > &g=
t; > statements, but it also seems each of such=0Astatements surpasses t=
he
=0A> > limit of
=0A> > > > 1024MB when trying to=
 index only 300000 of the=0Adatabase records.
=0A> > > >
=
=0A> > > > Thank you
=0A> > > >
=0A> > =
> >
=0A> > > >
=0A> > > > Chris Lamey w=
rote:
=0A> > > > What is you JVM heap size set to and how mu=
ch data=0Ais returned by
=0A> select
=0A> > *
=0A> >=
; > > from table?
=0A> > > >
=0A> > > >=
 If you're trying to pull back 1G worth of data into=0Aa JVM with a heap=0A> > size
=0A> > > > set to 64M, you will hit the h=
eap limit pretty=0Aquick.
=0A> > > >
=0A> > > &g=
t;
=0A> > > > -----Original Message-----
=0A> > >=
; > From: nch [mailto:=
underscore_dot@yahoo..com]
=0A> > > > Sent: Thu 3/13/200=
8 8:03 AM
=0A> > > > To: user-java@ibatis.apache.org
=0A> > > >=
; Subject: select * causing " OutOfMemoryError: Java=0Aheap space"
=0A&g=
t; > > >
=0A> > > >
=0A> > > > Hi, e=
verybody.
=0A> > > > I'm testing the Compass search engine a=
nd, in=0Aparticular, how to index
=0A> a
=0A> > big
=0A&g=
t; > > > set of documents from a table in a MySQL database.=0AIn o=
rder to do this
=0A> I
=0A> > > > issue a "select * fr=
om table_name" using iBatis,=0Abut this seems to be
=0A> > > &g=
t; causing the application to use all heap space=0Aavailable.
=0A> &g=
t; > > I added a new entry into Compass user forum with=0Adetails:=0A> > > > http://forum.com=
pass-project.org/thread.jspa?threadID=3D215278
=0A> > > >=
;
=0A> > > > Can you figure out why is this happening?
=
=0A> > > >
=0A> > > > Many thanks
=0A> >=
; > >
=0A> > > >
=0A> > > > -----------=
----------------------
=0A> > > > Be a better friend, newsho=
und, and know-it-all with=0AYahoo! Mobile. Try
=0A> > it
=0A>=
; > > > now.
=0A> > > >
=0A> > > >=0A> > > >
=0A> > > >
=0A> > > >=
; ________________________________
=0A> > > > Never miss a t=
hing. Make Yahoo your homepage.
=0A> > >
=0A> > >=0A> > > Never miss a thing. Make Yahoo your homepage.
=0A>=
; > >
=0A> > >
=0A> > >
=0A> > ><=
br>=0A> > >
=0A> > >
=0A> > >
=0A> &=
gt;
=0A>=0A__________________________________________________________=
__________________________
=0A> > > =C2=A1Capacidad ilimitada d=
e almacenamiento en tu correo!
=0A> > > No te preocupes m=C3=A1=
s por el espacio de tu cuenta con=0ACorreo Yahoo!:
=0A> > > http://correo.espanol.yahoo.com/
=0A> > >
=0A> >=
 >
=0A> > >
=0A> > >
=0A> > >
=0A=
> > >
=0A> > >
=0A> > > __________________=
______________
=0A> > Never miss a thing. Make Yahoo your homepage=
.
=0A> >
=0A> >
=0A> >
=0A> >
=0A>=
; >  ________________________________
=0A> > Never miss a =
thing. Make Yahoo your homepage.
=0A>
=0A>
=0A>  ___=
_____________________________
=0A>
=0A> Be a better friend, new=
shound, and know-it-all with Yahoo! Mobile.=0ATry it
=0A> now.
=0A=
          
=0A          
=0A
=0A
=0A =
=0A
Looking for last minute shopping d= eals? Find them fast with Yahoo! Search.
=0A
= =0A
=0A
=0A
=0A
Be a better friend, newshound, and know-it-all=0Awith Yahoo! Mobile= . = Try it now.
=0A
=0A
=0A =0A
=0A
=0A
=0A
=0A
Looking for last minute shopping deals? Find them fast with Yahoo! Search.
=0A
=0A =
=0A
=0A
=0A
=0A =
=0A
=0A
=0A
=0A =0A
=0A
=0A

=0A
Never miss a thing. Make Yahoo yo= ur homepage.=0A
=0A
=0A
=0A =
=0A
=0A
=0A
=0A
=0A= =0A
=0A
=0A =0A =0A <= /div>=0A
=0A =0A =0A
=0A
= =0A
Be a better friend, newshound, and know-it-all= =0Awith Yahoo! Mobile. Try it now.
=0A =0A =0A = =0A
=0A=0A
=0A

=0A =
Looking for last minute shopping deals? =0AFind them fast with Yahoo! Search.<= /html> --0-870482443-1205863826=:33788--