ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Duffey <tduf...@utilivisor.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 19:08:35 GMT
Not looking to be a MySQL hater but this screams Daily WTF :)

Thanks for the interesting thread.

Tom

On Mar 18, 2008, at 1:53 PM, Clinton Begin wrote:

>
> That's why I get paid the big bucks.  i.e. $0   -- LOL.
>
> I'm surprised nobody has run into it before, but I'm very glad that  
> worked.  :-)
>
> Thanks to the list for putting up with possibly our longest thread  
> in history. Thank you to MySQL for using magic numbers.
>
> Cheers,
> Clinton
>
>
> On Tue, Mar 18, 2008 at 12:24 PM, nch <underscore_dot@yahoo.com>  
> wrote:
>
> So-lu-tion.
> But, yeah, WOW!!
>
> Clinton, you've been really helpful.
> Lots of thanks to you and all people in this mailing list.
>
> ----- Original Message ----
> From: Clinton Begin <clinton.begin@gmail.com>
> To: user-java@ibatis.apache.org
> Sent: Tuesday, March 18, 2008 6:45:28 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> SOLUTION?
>
> A quick Google search yielded this thread from the MySQL boards which
> details the exact problem your having.
>
> http://lists.mysql.com/java/9137
>
> It's actually quite silly and definitely a MySQL driver issue.  I
> imagine that Hibernate's MySQL dialect configuration is aware of this
> problem, which is why it works.  Recall what I said earlier regarding
> the fact that the driver throws the exception from the
> PreparedStatement.execute() method... thus it's not even getting to
> the point of returning a result set.  This is why:
>
> // MySQL Driver code....
> protected boolean createStreamingResultSet() {
>   return ((this.resultSetType == java.sql.ResultSet.TYPE_FORWARD_ONLY)
>   && (this.resultSetConcurrency ==   
> java.sql.ResultSet.CONCUR_READ_ONLY)
>   && (this.fetchSize ==  Integer.MIN_VALUE)); // NOTICE THIS.... WTF!?
> }
>
> Turns out that for some reason the MySQL driver will always load all
> of the results in advance unless the fetchSize is set to
> Integer.MIN_VALUE.... wow.
>
> Try setting fetchSize="-2147483648"
>
> Silly, but wow...
>
> Clinton
>
> On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com> wrote:
> >
> >
> >
> >
> > Hi!
> >
> > Clinton, answering to your last mail:
> > - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm  
> currently using) with MySQL 5.0.45 (Debian). Tried also on Windows  
> with the same drivers, but can not remember the MySQL version.
> > - Bellow are, I think I don't miss any, the different relevant  
> configurations (some through Spring):
> >
> > - jdbc.properties: Database connection config
> >
> > jdbc.driverClassName=com.mysql.jdbc.Driver
> > jdbc.url=jdbc:mysql://localhost/wikipedia_articles? 
> createDatabaseIfNotExist 
> =true&amp;useUnicode=true&amp;characterEncoding=utf-8
> > jdbc.username=x
> > jdbc.password=y
> >
> >
> > - sql-map-config.xml: iBatis config file
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map  
> Config 2.0//EN"
> >    "http://ibatis.apache.org/dtd/sql-map-config-2.dtd">
> >
> > <sqlMapConfig>
> >    <settings enhancementEnabled="true" maxTransactions="20"  
> maxRequests="32" maxSessions="10"/>
> >
> >    <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/>
> >
> > </sqlMapConfig>
> >
> >
> > - TestingPojoSQL.xml: The operations needed for indexing:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "http://ibatis.apache.org/dtd/sql-map-2.dtd

> ">
> >
> > <sqlMap namespace="TestingPojoSQL">
> >    <typeAlias alias="article" type="org.myorg.model.TestingPojo"/>
> >
> >    <resultMap id="articleResult" class="article">
> >        <result property="id" column="id"/>
> >        <result property="contents" column="contents"/>
> >    </resultMap>
> >
> >    <select id="getAllTestingPojos" resultMap="articleResult"  
> fetchSize="100" resultSetType="FORWARD_ONLY">
> >    <![CDATA[
> >        select * from articles
> >    ]]>
> >    </select>
> >
> > </sqlMap>
> >
> >
> > - applicationContext-resources.xml: Here a DBCP data source is  
> defined (I tried also C3P0. Made no difference):
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> >        xmlns:jee="http://www.springframework.org/schema/jee"
> >        xsi:schemaLocation="http://www.springframework.org/schema/beans 
>  http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >            http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.0.xsd

> ">
> >
> >    <bean id="propertyConfigurer"  
> class 
> = 
> "org 
> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >        <property name="locations">
> >            <list>
> >                <value>classpath:jdbc.properties</value>
> >            </list>
> >        </property>
> >    </bean>
> >
> >    <bean id="dataSource"  
> class="org.apache.commons.dbcp.BasicDataSource" destroy- 
> method="close">
> >        <property name="driverClassName" value="$ 
> {jdbc.driverClassName}"/>
> >        <property name="url" value="${jdbc.url}"/>
> >        <property name="username" value="${jdbc.username}"/>
> >        <property name="password" value="${jdbc.password}"/>
> >        <property name="maxActive" value="100"/>
> >        <property name="maxWait" value="1000"/>
> >        <property name="poolPreparedStatements" value="true"/>
> >        <property name="defaultAutoCommit" value="true"/>
> >    </bean>
> >
> > </beans>
> >
> >
> > - In applicationContext-dao.xml: the sqlMapClient
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> >        xsi:schemaLocation="http://www.springframework.org/schema/beans 
>  http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"
> >        default-lazy-init="true">
> >
> >    <bean id="transactionManager"  
> class 
> ="org.springframework.jdbc.datasource.DataSourceTransactionManager">
> >        <property name="dataSource" ref="dataSource"/>
> >    </bean>
> >
> >    <bean id="sqlMapClient"  
> class="org.springframework.orm.ibatis.SqlMapClientFactoryBean">
> >        <property name="configLocation">
> >            <value>classpath:/sql-map-config.xml</value>
> >        </property>
> >        <property name="dataSource" ref="dataSource"/>
> >    </bean>
> > </beans>
> >
> >
> > - In applicationContext-service.xml: This is where I tell Spring  
> to instanciate an iBatis based GPS device.
> >
> > <bean id="compass" class="org.compass.spring.LocalCompassBean">
> >    <property name="classMappings">
> >        <list>
> >            <value>org.myorg.TestingPojo</value>
> >        </list>
> >    </property>
> >    <property name="compassSettings">
> >        <props>
> >            <prop key="compass.engine.connection">file://compass</ 
> prop>
> >            <prop  
> key 
> = 
> "compass 
> .transaction.factory">org.compass.spring.transaction.SpringSyncT  
> ransactionFactory</prop>
> >            <prop  
> key="compass.engine.analyzer.default.type">snowball</prop>
> >            <prop  
> key="compass.engine.analyzer.default.name">English</prop>
> >            <!-- Mainly default values -->
> >            <prop key="compass.engine.ramBufferSize">16</prop>
> >            <prop key="compass.engine.maxBufferedDocs">10</prop>
> >            <prop key="compass.engine.maxBufferedDeletedTerms">-1</ 
> prop>
> >            <prop key="compass.engine.mergeFactor">10</prop>
> >            <prop key="compass.engine.maxFieldLength">10000</prop>
> >            <prop  
> key="compass.engine.cacheIntervalInvalidation">5000</prop>
> >            <prop  
> key="compass.engine.indexManagerScheduleInterval">60</prop>
> >        </props>
> >    </property>
> >    <property name="transactionManager" ref="transactionManager" />
> > </bean>
> >
> > <bean id="iBatisGpsDevice"  
> class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
> >    <property name="name" value="iBatisDevice" />
> >    <property name="sqlMapClient" ref="sqlMapClient" />
> >    <property name="selectStatementsIds">
> >        <list>
> >            <value>getAllTestingPojos</value>
> >        </list>
> >    </property>
> >    <property name="pageSize" value="50" />
> > </bean>
> >
> > <bean id="compassGps"  
> class="org.compass.gps.impl.SingleCompassGps" init-method="start"  
> destroy-method="stop">
> >    <property name="compass">
> >        <ref bean="compass" />
> >    </property>
> >    <property name="gpsDevices">
> >        <list>
> >            <bean  
> class 
> ="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
> >                <property name="gpsDevice" ref="iBatisGpsDevice" />
> >            </bean>
> >        </list>
> >    </property>
> > </bean>
> >
> >
> > Regarding your previous mail:
> > - I emptied the handleRow method and ran the test - same result.  
> The exception occurs before handleRow is invoked.
> > - I'm not configuring any caches. Is there a default one? Should I  
> config one?
> > - Setting fetchSize, different resultSetTypes, use of <select> or  
> <statement> elements didn't make any differences.
> > - Additionally, I tried truncating the contents of each Wikipedia  
> article to 500 characters, and I was able to index all the 650000  
> rows, so the number of records being selected is not the only factor  
> here (I just wanted to make sure).
> > - I can imagine others (including you) have successfully made it  
> with large amounts of data (I'm new to iBatis and Compass/Lucene) so  
> I'm sure there's a solution.
> >
> > Lots of thanks.
> >
> >
> >
> >
> >
> >
> > ----- Original Message ----
> > From: Clinton Begin <clinton.begin@gmail.com>
> > To: user-java@ibatis.apache.org
> >
> >
> >
> > Sent: Monday, March 17, 2008 11:18:59 PM
> > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> > Wow, I just read that other thread on the Compass site.... This is  
> not directly an iBATIS problem at all...
> >
> > at  
> com 
> .mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java: 
> 1268) at  
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> >
> > This line tells me that the execution is never returning from the  
> prepared statement.  iBATIS never has a chance to even see the  
> ResultSet, let alone get to the RowHandler. So swapping out the  
> GpsDevice here won't make any difference at all. This is odd  
> behavior for a JDBC driver and tells me that something is very wrong  
> with the configuration.
> >
> > Which version of the JDBC driver are you using?  What are the rest  
> of the iBATIS convfiguration details?
> >
> > Clinton
> >
> >
> >
> > On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <clinton.begin@gmail.com 
> > wrote:
> >
> > > Thanks for the summary.  A few things:
> > >
> > > * PaginatedList was definitely the wrong solution for that, so  
> I'm glad it's gone.
> > >
> > > * A RowHandler is possibly the right solution, but will only  
> store as many records in memory as you tell it to.  iBATIS does not  
> just arbitrarily keep records around.  Quite the opposite actually.   
> Unless you keep a reference or configure a cache, iBATIS will not  
> keep the object around at all.
> > >
> > > Looking at your implementation, there are a couple of things  
> going on...
> > >
> > > session.create(o);
> > > if(currentItem== pageSize){
> > >  if (log.isDebugEnabled()) {
> > >    log.debug("Indexing page number ["+ pageCount++ + "]");
> > >  }
> > > session.evictAll();
> > >
> > > Things to Try:
> > >
> > > 1)  These lines of code must be treated as guilty until proven  
> innocent (especially the logging).  The easiest way to do that is to  
> delete them.  Create an empty handler/GpsDevice and run the test to  
> see what happens.
> > >
> > > 2) Look at the iBATIS configuration files.  Is a cache  
> configured for this query?  What kind of cache?
> > >
> > > 3) You could try multiple queries of fixed sizes with  
> queryForList(String,Object,int offset,int limit) ... but I can't see  
> how that would be better than a RowHandler.
> > >
> > > 4) Next, just in case the driver's default fetch size is  
> ridiculously high (or unlimited), try setting <select ...  
> fetchSize="100"/>.  You could also try a scrollable SesultSet with  
> <statement ... resultSetType /> of  SCROLL_INSENSITIVE or  
> SCROLL_SENSITIVE.
> > >
> > > Finally, please know that I've used iBATIS to load tens of  
> millions of rows for ETL and even analysis (NetFlix Prize) and was  
> able to do so with a RowHandler or even simply multiple calls to  
> queryForList(String,Object,int offset,int limit).  I don't recall  
> having to even think about the configuration, although I did manage  
> to optimize it to achieve a read/insert performance of 15,000  
> records per second between two databases (on a single machine too).
> > >
> > > I'm sure there's something about the configuration here that is  
> causing the problem, and it can be solved.
> > >
> > > I hope one of these helps.
> > >
> > > Clinton
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com>  
> wrote:
> > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi, there. :-)
> > > >
> > > > The thing is, I'm testing a search engine called Compass. One  
> of the tests consists of indexing a Wikipedia dump. In this case  
> only the text of the articles, so I downloaded it and imported it  
> into a MySQL database (about 650000 records - 1.5GB).
> > > >
> > > > You can configure Compass to access a database and index it's  
> contents by configuring what they call a GPS Device. You can do that  
> through an ORM such as iBatis, Hibernate or JPA.
> > > >
> > > > Compass provides an implementation of such a GPS Device called  
> SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get  
> the results of a query and, so, index them. The query is just a  
> "select * from articles_table".
> > > >
> > > > So I wired everything up and ran several tests with different  
> amounts of data to be indexed and JVM stack space sizes. The result  
> was an "OutOfMemoryError: java heap space" error message whenever  
> the size of the selected rows was bigger than the amount of  
> available heap space (I can not index the 650000 records having 1GB  
> of stack).
> > > >
> > > > After doing some profiling I concluded iBatis was loading the  
> whole amount of results into memory. So I decided to try Hibernate,  
> which worked just fine.
> > > >
> > > > Finally I found queryForPaginatedList was deprecated and that  
> I should use queryWithRowHandler instead, so I made my own  
> implementation of the SqlMapClientGpsDevice based on  
> queryWithRowHandler and tested it, but I had the same result (I  
> profiled the application stack usage and I could see a line growing  
> steadily and finally crash against the stack ceiling while  
> performing the select statement).
> > > >
> > > > Shay Banon, Compass project leader, is planning to patch  
> SqlMapClientGpsDevice by replacing queryForPaginatedList with  
> queryWithRowHandler, but I'm doubtful it's gonna work.
> > > >
> > > > Finally you can, of course, devide the select stament into  
> several selects (and it works) but, in my opinion, that shouldn't be  
> necessary.
> > > > You can read more on http://forum.compass-project.org/thread.jspa?threadID=215278
> > > >
> > > > Thanks!
> > > >
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > > From: Clinton Begin <clinton.begin@gmail.com>
> > > > To: user-java@ibatis.apache.org
> > > >
> > > >
> > > >
> > > > Sent: Monday, March 17, 2008 2:22:39 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap  
> space"
> > > >
> > > > Wow, this thread is interesting.  I suppose I could read the  
> past emails, but could someone summarize the problem and progress to  
> date?  I may be able to shed some light on what is happening.
> > > >
> > > > Clinton
> > > >
> > > >
> > > > On Sun, Mar 16, 2008 at 11:49 PM, nch  
> <underscore_dot@yahoo.com> wrote:
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Another thing I forgot. This works perfectly well if we  
> replace iBatis by Hibernate, so I don't think this is being caused  
> by the MySQL driver.
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: nch <underscore_dot@yahoo.com>
> > > > > To: user-java@ibatis.apache.org
> > > > >
> > > > >
> > > > >
> > > > > Sent: Sunday, March 16, 2008 10:18:42 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap  
> space"
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Well, I modified the mediumblob into a mediumtext and  
> removed all other fields except the id one.
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: nch <underscore_dot@yahoo.com>
> > > > > To: user-java@ibatis.apache.org
> > > > > Sent: Sunday, March 16, 2008 10:06:34 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap  
> space"
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > You're probably right. I'm using the table called "text"  
> which contains a field of type "mediumblob".
> > > > > CREATE TABLE /*$wgDBprefix*/text (
> > > > > old_id int unsigned NOT NULL auto_increment,
> > > > > old_text mediumblob NOT NULL,
> > > > > old_flags tinyblob NOT NULL,
> > > > > PRIMARY KEY old_id (old_id)
> > > > > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000  
> AVG_ROW_LENGTH=10240;
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > You can find it's definition here:
> > > > > http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
> > > > >
> > > > > I'll try to modify that field, first, and see what happens.
> > > > > I've already tried "select * from your_wikipedia_dump limit  
> $start$, $size$" and it seems to work fine, but I see a possible  
> issue with this, you can not remove any of the records you've  
> already read during the whole process. This might not be an option.
> > > > >
> > > > > Thank you.
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > To: user-java@ibatis.apache.org
> > > > > Sent: Sunday, March 16, 2008 4:19:43 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap  
> space"
> > > > >
> > > > > Yeah, I just noticed that - it looks like it's failing in  
> the jdbc
> > > > > driver when it tries to read a blob, maybe?
> > > > >
> > > > > I know that some versions of the postgresql jdbc driver load  
> the
> > > > > entire contents of a resultset into a massive byte[][]  
> structure. I
> > > > > wonder if that is what is happening here? I think you can  
> get the
> > > > > source for the MySQL driver and check that if you want to.
> > > > >
> > > > > If it is, you may need to structure your app to take smaller  
> bites of
> > > > > this elephant.
> > > > >
> > > > > Try "select * from your_wikipedia_dump limit $start$, $size 
> $" instead
> > > > > - that will limit the size of the results to $size$ rows,  
> starting on
> > > > > row $start$ (the $start$ value is zero-based). If you  
> experiment to
> > > > > see how big $size$ can get, you should be able to get pretty  
> decent
> > > > > performance out of this.
> > > > >
> > > > > Psuedo-code would be like this...
> > > > >
> > > > > int start = 0;
> > > > > int size = 1000; // see how big you can make this to improve  
> performance
> > > > > while(still_more_data){
> > > > >  still_more_data = index(start, size);
> > > > >  start += size;
> > > > > }
> > > > >
> > > > > Your index method will return true if the query returned any  
> data. If
> > > > > the query returns no data, it'll return false.
> > > > >
> > > > > I can't imagine what Hibernate is doing differently here to  
> make this
> > > > > work if the JDBC driver is failing when executing the query.
> > > > >
> > > > > Larry
> > > > >
> > > > >
> > > > > On Sun, Mar 16, 2008 at 4:12 AM, nch  
> <underscore_dot@yahoo.com> wrote:
> > > > > >
> > > > > >
> > > > > > Sure. Please, see attached.
> > > > > > I don't think the problem is in the RowHandler, though,  
> because the
> > > > > > OutOfMemoryError occurs before invoking RowHandle#handleRow.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > > ----- Original Message ----
> > > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > > To: user-java@ibatis.apache.org
> > > > > >
> > > > > > Sent: Sunday, March 16, 2008 3:13:27 AM
> > > > > > Subject: Re: select * causing " OutOfMemoryError: Java  
> heap space"
> > > > > >
> > > > > >  Can you post the row handler you are using.
> > > > > >
> > > > > > Larry
> > > > > >
> > > > > >
> > > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com

> > wrote:
> > > > > > >
> > > > > > >
> > > > > > > Hi, Nathan.
> > > > > > > I did so, but I'm still having the same issue. Perhaps
 
> I'm not using it
> > > > > > > correctly?
> > > > > > > See my last post to this forum entry:
> > > > > > >
> > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > >
> > > > > > > See the stack trace:
> > > > > > >
> > > > > > > MemoryError: Java heap space:
> > > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > > >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > > > > > >        at  
> com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> > > > > > >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:

> 1366)
> > > > > > >        at  
> com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> > > > > > >        at  
> com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> > > > > > >        at
> > > > > > >  
> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> > > > > > >        at  
> com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> > > > > > >        at  
> com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> > > > > > >        at  
> com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java: 
> 1268)
> > > > > > >        at
> > > > > > >  
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .apache 
> .commons 
> .dbcp 
> .DelegatingPreparedStatement 
> .execute(DelegatingPreparedStatement.java:169)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .apache 
> .commons 
> .dbcp 
> .DelegatingPreparedStatement 
> .execute(DelegatingPreparedStatement.java:169)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java: 
> 186)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap 
> .engine 
> .mapping 
> .statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap 
> .engine 
> .mapping 
> .statement 
> .GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap 
> .engine 
> .mapping 
> .statement 
> .GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java: 
> 133)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap 
> .engine 
> .impl 
> .SqlMapExecutorDelegate 
> .queryWithRowHandler(SqlMapExecutorDelegate.java:649)
> > > > > > >        at
> > > > > > >
> > > > > >  
> com 
> .ibatis 
> .sqlmap 
> .engine 
> .impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java: 
> 156)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .myorg 
> .compass 
> .SqlMapClientGpsDeviceWithRowHandler 
> .doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
> > > > > > >        at
> > > > > > >
> > > > > > org.compass.gps.device.AbstractGpsDevice 
> $1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .compass 
> .core 
> .CompassCallbackWithoutResult 
> .doInCompass(CompassCallbackWithoutResult.java:29)
> > > > > > >        at
> > > > > > >  
> org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .compass 
> .gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
> > > > > > >        at
> > > > > > >  
> org 
> .compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper 
> $ 
> 1 
> .doInTransactionWithoutResult 
> (SpringSyncTransactionGpsDeviceWrapper.java:98)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .springframework 
> .transaction 
> .support 
> .TransactionCallbackWithoutResult 
> .doInTransaction(TransactionCallbackWithoutResult.java:33)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .springframework 
> .transaction 
> .support.TransactionTemplate.execute(TransactionTemplate.java:128)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .compass 
> .spring 
> .device 
> .SpringSyncTransactionGpsDeviceWrapper 
> .index(SpringSyncTransactionGpsDeviceWrapper.java:96)
> > > > > > >        at
> > > > > > >
> > > > > > org.compass.gps.impl.SingleCompassGps 
> $1.buildIndexIfNeeded(SingleCompassGps.java:133)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .compass 
> .core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager 
> $8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
> > > > > > >        at
> > > > > > >
> > > > > >  
> org 
> .compass 
> .core 
> .lucene 
> .engine 
> .manager 
> .DefaultLuceneSearchEngineIndexManager 
> .doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ----- Original Message ----
> > > > > > > From: Nathan Maves <nathan.maves@gmail.com>
> > > > > > > To: user-java@ibatis.apache.org
> > > > > > >
> > > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM
> > > > > > > Subject: Re: select * causing " OutOfMemoryError: Java
 
> heap space"
> > > > > > >
> > > > > > >  this is only my 2 cents but I would throw that class 

> out.  Write your own
> > > > > > > implementation which uses a row handler.  This is the 

> type of situation in
> > > > > > > which a row handler could really help.  almost no memory
 
> would be used.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com

> > wrote:
> > > > > > > >
> > > > > > > > Yes, I agree that dividing the query into several
 
> queries should do as
> > > > > > > long as none of them uses more memory than available. 

> Gracias, Carlos.
> > > > > > > >
> > > > > > > > Nevertheless, I looked into
> > > > > > >  
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> > > > > > > fundamentally, what it does is a queryForPaginatedList
 
> and iterates
> > > > > > through
> > > > > > > the resulting PaginatedList with nextPage()
> > > > > > > >
> > > > > > > > I guess that the reason why pages are kept in memory
 
> after performing
> > > > > > > paginatedList.nextPage() is because a  
> paginatedList.previousPage() can be
> > > > > > > issued but, should't it free pages if heap space is  
> running out?
> > > > > > > >
> > > > > > > >
> > > > > > > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>
wrote:
> > > > > > > >
> > > > > > > > Since we have a similar trouble around here i must
say  
> that you have lot
> > > > > > > of options when going to compass..
> > > > > > > > The first one is to index "object by object" by hand,
 
> the second one is
> > > > > > to
> > > > > > > use the Gps wich will use the complete bunch of data and
 
> make subindexes
> > > > > > > (making parts of your "main index" with diferent "maps"
 
> for each subindex
> > > > > > in
> > > > > > > your gpsDriver.
> > > > > > > > if your indexing is as strong as ours i should look
 
> forward for numer 1
> > > > > > or
> > > > > > > number 3 depending of your needs. (More a Compass than
 
> iBatis issue)
> > > > > > > > Greetings
> > > > > > > > Carlos de Luna
> > > > > > > >
> > > > > > > > ----- Mensaje original ----
> > > > > > > > De: nch
> > > > > > > >
> > > > > > > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > > > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > > > > > > Asunto: Re: select * causing " OutOfMemoryError: Java
 
> heap space"
> > > > > > > >
> > > > > > > >
> > > > > > > > Sorry. Yes, I need all the data at one time, because
 
> the Compass API
> > > > > > seems
> > > > > > > to be meant in that way.
> > > > > > > >
> > > > > > > > Larry Meadors wrote: OK, so the answer to the question
 
> is...[ yes, i
> > > > > > need
> > > > > > > all the data at
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > one time | no, i do not need all of the data at one
 
> time ].
> > > > > > > >
> > > > > > > > Larry
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > > > > > > >
> > > > > > > > > Hi!
> > > > > > > > >
> > > > > > > > > I'm doing tests both under tomcat and jetty.
I set  
> 1024MB of heap
> > > > > > space
> > > > > > > in
> > > > > > > > > both cases. I am using a profiler to look into
 
> what's happening during
> > > > > > > > > execution of the indexing process and I can tell
 
> that's the exact
> > > > > > amount
> > > > > > > it
> > > > > > > > > displays for the VM.
> > > > > > > > >
> > > > > > > > > As I describe in the Compass user forum (see
link  
> bellow), I'm
> > > > > > indexing
> > > > > > > a
> > > > > > > > > Wikipedia dump of about 650000-675000 records.
 
> That's sort of 1-1.5GB
> > > > > > of
> > > > > > > > > data.
> > > > > > > > >
> > > > > > > > > Hibernate can handle all that data, I guess it's
 
> using some sort of
> > > > > > > > > pagination. iBatis seems to divide the select
 
> statement into several
> > > > > > > select
> > > > > > > > > statements, but it also seems each of such  
> statements surpasses the
> > > > > > > limit of
> > > > > > > > > 1024MB when trying to index only 300000 of the
 
> database records.
> > > > > > > > >
> > > > > > > > > Thank you
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chris Lamey wrote:
> > > > > > > > > What is you JVM heap size set to and how much
data  
> is returned by
> > > > > > select
> > > > > > > *
> > > > > > > > > from table?
> > > > > > > > >
> > > > > > > > > If you're trying to pull back 1G worth of data
into  
> a JVM with a heap
> > > > > > > size
> > > > > > > > > set to 64M, you will hit the heap limit pretty
quick.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > > > > > > Sent: Thu 3/13/2008 8:03 AM
> > > > > > > > > To: user-java@ibatis.apache.org
> > > > > > > > > Subject: select * causing " OutOfMemoryError:
Java  
> heap space"
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi, everybody.
> > > > > > > > > I'm testing the Compass search engine and, in
 
> particular, how to index
> > > > > > a
> > > > > > > big
> > > > > > > > > set of documents from a table in a MySQL database.
 
> In order to do this
> > > > > > I
> > > > > > > > > issue a "select * from table_name" using iBatis,
but  
> this seems to be
> > > > > > > > > causing the application to use all heap space
 
> available.
> > > > > > > > > I added a new entry into Compass user forum with
 
> details:
> > > > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > > > >
> > > > > > > > > Can you figure out why is this happening?
> > > > > > > > >
> > > > > > > > > Many thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ---------------------------------
> > > > > > > > > Be a better friend, newshound, and know-it-all
with  
> Yahoo! Mobile. Try
> > > > > > > it
> > > > > > > > > now.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > > >
> > > > > > > >
> > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >  
> ____________________________________________________________________________________
> > > > > > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > > > > > No te preocupes más por el espacio de tu cuenta con
 
> Correo Yahoo!:
> > > > > > > > http://correo.espanol.yahoo.com/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >  ________________________________
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > >
> > > > > >
> > > > > >  ________________________________
> > > > > >
> > > > > > Be a better friend, newshound, and know-it-all with Yahoo! 

> Mobile. Try it
> > > > > > now.
> > > > >
> > > > >
> > > > > ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo!  
> Search.
> > > > >
> > > > > ________________________________
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  
> Try it now.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo!  
> Search.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> >
> >
> >
> > ________________________________
>
> > Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  
> Try it now.
>
>
> Looking for last minute shopping deals? Find them fast with Yahoo!  
> Search.
>


Mime
View raw message