ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Lamey <cla...@localmatters.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 19:43:13 GMT
Looks like you can use the "userCursorFetch=true" param on the JDBC URL to
get it to adhere to the fetch size:

http://bugs.mysql.com/bug.php?id=18148

Cheers,
topher

On 3/18/08 12:24 PM, "nch" <underscore_dot@yahoo.com> wrote:

> 
> So-lu-tion.
> But, yeah, WOW!!
> 
> Clinton, you've been really helpful.
> Lots of thanks to you and all people in this mailing list.
> 
> ----- Original Message ----
> From: Clinton Begin <clinton.begin@gmail.com>
> To: user-java@ibatis.apache.org
> Sent: Tuesday, March 18, 2008 6:45:28 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> 
> SOLUTION?
> 
> A quick Google search yielded this thread from the MySQL boards which
> details the exact problem your having.
> 
> http://lists.mysql.com/java/9137
> 
> It's actually quite silly and definitely a MySQL driver issue.  I
> imagine that Hibernate's MySQL dialect configuration is aware of this
> problem, which is why it works.  Recall what I said earlier regarding
> the fact that the driver throws the exception from the
> PreparedStatement.execute() method... thus it's not even getting to
> the point of returning a result set.  This is why:
> 
> // MySQL Driver code....
> protected boolean createStreamingResultSet() {
>    return ((this.resultSetType == java.sql.ResultSet.TYPE_FORWARD_ONLY)
>    && (this.resultSetConcurrency ==  java.sql.ResultSet.CONCUR_READ_ONLY)
>    && (this.fetchSize ==  Integer.MIN_VALUE)); // NOTICE THIS.... WTF!?
> }
> 
> Turns out that for some reason the MySQL driver will always load all
> of the results in advance unless the fetchSize is set to
> Integer.MIN_VALUE.... wow.
> 
> Try setting fetchSize="-2147483648"
> 
> Silly, but wow...
> 
> Clinton
> 
> On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com> wrote:
>> 
>> 
>> 
>> 
>> Hi!
>> 
>> Clinton, answering to your last mail:
>> - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currently using)
>> with MySQL 5.0.45 (Debian). Tried also on Windows with the same drivers, but
>> can not remember the MySQL version.
>> - Bellow are, I think I don't miss any, the different relevant configurations
>> (some through Spring):
>> 
>> - jdbc.properties: Database connection config
>> 
>> jdbc.driverClassName=com.mysql.jdbc.Driver
>> jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=t
>> rue&amp;useUnicode=true&amp;characterEncoding=utf-8
>> jdbc.username=x
>> jdbc.password=y
>> 
>> 
>> - sql-map-config.xml: iBatis config file
>> 
>> <?xml version="1.0" encoding="UTF-8"?>
>> <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map Config
>> 2.0//EN"
>>     "http://ibatis.apache.org/dtd/sql-map-config-2.dtd">
>> 
>> <sqlMapConfig>
>>     <settings enhancementEnabled="true" maxTransactions="20" maxRequests="32"
>> maxSessions="10"/>
>> 
>>     <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/>
>> 
>> </sqlMapConfig>
>> 
>> 
>> - TestingPojoSQL.xml: The operations needed for indexing:
>> 
>> <?xml version="1.0" encoding="UTF-8"?>
>> <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN"
>> "http://ibatis.apache.org/dtd/sql-map-2.dtd">
>> 
>> <sqlMap namespace="TestingPojoSQL">
>>     <typeAlias alias="article" type="org.myorg.model.TestingPojo"/>
>> 
>>     <resultMap id="articleResult" class="article">
>>         <result property="id" column="id"/>
>>         <result property="contents" column="contents"/>
>>     </resultMap>
>> 
>>     <select id="getAllTestingPojos" resultMap="articleResult" fetchSize="100"
>> resultSetType="FORWARD_ONLY">
>>     <![CDATA[
>>         select * from articles
>>     ]]>
>>     </select>
>> 
>> </sqlMap>
>> 
>> 
>> - applicationContext-resources.xml: Here a DBCP data source is defined (I
>> tried also C3P0. Made no difference):
>> 
>> <?xml version="1.0" encoding="UTF-8"?>
>> 
>>        xmlns:jee="http://www.springframework.org/schema/jee"
>>        xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>             http://www.springframework.org/schema/jee
>> http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
>> 
>>     <bean id="propertyConfigurer"
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer
>> ">
>>         <property name="locations">
>>             <list>
>>                 <value>classpath:jdbc.properties</value>
>>             </list>
>>         </property>
>>     </bean>
>> 
>>     <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
>> destroy-method="close">
>>         <property name="driverClassName" value="${jdbc.driverClassName}"/>
>>         <property name="url" value="${jdbc.url}"/>
>>         <property name="username" value="${jdbc.username}"/>
>>         <property name="password" value="${jdbc.password}"/>
>>         <property name="maxActive" value="100"/>
>>         <property name="maxWait" value="1000"/>
>>         <property name="poolPreparedStatements" value="true"/>
>>         <property name="defaultAutoCommit" value="true"/>
>>     </bean>
>> 
>> </beans>
>> 
>> 
>> - In applicationContext-dao.xml: the sqlMapClient
>> 
>> <?xml version="1.0" encoding="UTF-8"?>
>> 
>>        xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"
>>        default-lazy-init="true">
>> 
>>     <bean id="transactionManager"
>> class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
>>         <property name="dataSource" ref="dataSource"/>
>>     </bean>
>> 
>>     <bean id="sqlMapClient"
>> class="org.springframework.orm.ibatis.SqlMapClientFactoryBean">
>>         <property name="configLocation">
>>             <value>classpath:/sql-map-config.xml</value>
>>         </property>
>>         <property name="dataSource" ref="dataSource"/>
>>     </bean>
>> </beans>
>> 
>> 
>> - In applicationContext-service.xml: This is where I tell Spring to
>> instanciate an iBatis based GPS device.
>> 
>> <bean id="compass" class="org.compass.spring.LocalCompassBean">
>>     <property name="classMappings">
>>         <list>
>>             <value>org.myorg.TestingPojo</value>
>>         </list>
>>     </property>
>>     <property name="compassSettings">
>>         <props>
>>             <prop key="compass.engine.connection">file://compass</prop>
>>             <prop
>> key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncT
>> ransactionFactory</prop>
>>             <prop key="compass.engine.analyzer.default.type">snowball</prop>
>>             <prop key="compass.engine.analyzer.default.name">English</prop>
>>             <!-- Mainly default values -->
>>             <prop key="compass.engine.ramBufferSize">16</prop>
>>             <prop key="compass.engine.maxBufferedDocs">10</prop>
>>             <prop key="compass.engine.maxBufferedDeletedTerms">-1</prop>
>>             <prop key="compass.engine.mergeFactor">10</prop>
>>             <prop key="compass.engine.maxFieldLength">10000</prop>
>>             <prop key="compass.engine.cacheIntervalInvalidation">5000</prop>
>>             <prop key="compass.engine.indexManagerScheduleInterval">60</prop>
>>         </props>
>>     </property>
>>     <property name="transactionManager" ref="transactionManager" />
>> </bean>
>> 
>> <bean id="iBatisGpsDevice"
>> class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
>>     <property name="name" value="iBatisDevice" />
>>     <property name="sqlMapClient" ref="sqlMapClient" />
>>     <property name="selectStatementsIds">
>>         <list>
>>             <value>getAllTestingPojos</value>
>>         </list>
>>     </property>
>>     <property name="pageSize" value="50" />
>> </bean>
>> 
>> <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps"
>> init-method="start" destroy-method="stop">
>>     <property name="compass">
>>         <ref bean="compass" />
>>     </property>
>>     <property name="gpsDevices">
>>         <list>
>>             <bean
>> class="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
>>                 <property name="gpsDevice" ref="iBatisGpsDevice" />
>>             </bean>
>>         </list>
>>     </property>
>> </bean>
>> 
>> 
>> Regarding your previous mail:
>> - I emptied the handleRow method and ran the test - same result. The
>> exception occurs before handleRow is invoked.
>> - I'm not configuring any caches. Is there a default one? Should I config
>> one?
>> - Setting fetchSize, different resultSetTypes, use of <select> or <statement>
>> elements didn't make any differences.
>> - Additionally, I tried truncating the contents of each Wikipedia article to
>> 500 characters, and I was able to index all the 650000 rows, so the number of
>> records being selected is not the only factor here (I just wanted to make
>> sure).
>> - I can imagine others (including you) have successfully made it with large
>> amounts of data (I'm new to iBatis and Compass/Lucene) so I'm sure there's a
>> solution.
>> 
>> Lots of thanks.
>> 
>> 
>> 
>> 
>> 
>> 
>> ----- Original Message ----
>> From: Clinton Begin <clinton.begin@gmail.com>
>> To: user-java@ibatis.apache.org
>> 
>> 
>> 
>> Sent: Monday, March 17, 2008 11:18:59 PM
>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>> 
>> Wow, I just read that other thread on the Compass site.... This is not
>> directly an iBATIS problem at all...
>> 
>> at 
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
>> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
>> 
>> This line tells me that the execution is never returning from the prepared
>> statement.  iBATIS never has a chance to even see the ResultSet, let alone
>> get to the RowHandler. So swapping out the GpsDevice here won't make any
>> difference at all. This is odd behavior for a JDBC driver and tells me that
>> something is very wrong with the configuration.
>> 
>> Which version of the JDBC driver are you using?  What are the rest of the
>> iBATIS convfiguration details?
>> 
>> Clinton
>> 
>> 
>> 
>> On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <clinton.begin@gmail.com>
>> wrote:
>> 
>>> Thanks for the summary.  A few things:
>>> 
>>> * PaginatedList was definitely the wrong solution for that, so I'm glad it's
>>> gone.
>>> 
>>> * A RowHandler is possibly the right solution, but will only store as many
>>> records in memory as you tell it to.  iBATIS does not just arbitrarily keep
>>> records around.  Quite the opposite actually.  Unless you keep a reference
>>> or configure a cache, iBATIS will not keep the object around at all.
>>> 
>>> Looking at your implementation, there are a couple of things going on...
>>> 
>>> session.create(o);
>>> if(currentItem== pageSize){
>>>   if (log.isDebugEnabled()) {
>>>     log.debug("Indexing page number ["+ pageCount++ + "]");
>>>   }
>>> session.evictAll();
>>> 
>>> Things to Try:
>>> 
>>> 1)  These lines of code must be treated as guilty until proven innocent
>>> (especially the logging).  The easiest way to do that is to delete them.
>>> Create an empty handler/GpsDevice and run the test to see what happens.
>>> 
>>> 2) Look at the iBATIS configuration files.  Is a cache configured for this
>>> query?  What kind of cache?
>>> 
>>> 3) You could try multiple queries of fixed sizes with
>>> queryForList(String,Object,int offset,int limit) ... but I can't see how
>>> that would be better than a RowHandler.
>>> 
>>> 4) Next, just in case the driver's default fetch size is ridiculously high
>>> (or unlimited), try setting <select ... fetchSize="100"/>.  You could also
>>> try a scrollable SesultSet with <statement ... resultSetType /> of
>>> SCROLL_INSENSITIVE or SCROLL_SENSITIVE.
>>> 
>>> Finally, please know that I've used iBATIS to load tens of millions of rows
>>> for ETL and even analysis (NetFlix Prize) and was able to do so with a
>>> RowHandler or even simply multiple calls to queryForList(String,Object,int
>>> offset,int limit).  I don't recall having to even think about the
>>> configuration, although I did manage to optimize it to achieve a read/insert
>>> performance of 15,000 records per second between two databases (on a single
>>> machine too).
>>> 
>>> I'm sure there's something about the configuration here that is causing the
>>> problem, and it can be solved.
>>> 
>>> I hope one of these helps.
>>> 
>>> Clinton
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com> wrote:
>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Hi, there. :-)
>>>> 
>>>> The thing is, I'm testing a search engine called Compass. One of the tests
>>>> consists of indexing a Wikipedia dump. In this case only the text of the
>>>> articles, so I downloaded it and imported it into a MySQL database (about
>>>> 650000 records - 1.5GB).
>>>> 
>>>> You can configure Compass to access a database and index it's contents by
>>>> configuring what they call a GPS Device. You can do that through an ORM
>>>> such as iBatis, Hibernate or JPA.
>>>> 
>>>> Compass provides an implementation of such a GPS Device called
>>>> SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the
>>>> results of a query and, so, index them. The query is just a "select * from
>>>> articles_table".
>>>> 
>>>> So I wired everything up and ran several tests with different amounts of
>>>> data to be indexed and JVM stack space sizes. The result was an
>>>> "OutOfMemoryError: java heap space" error message whenever the size of the
>>>> selected rows was bigger than the amount of available heap space (I can not
>>>> index the 650000 records having 1GB of stack).
>>>> 
>>>> After doing some profiling I concluded iBatis was loading the whole amount
>>>> of results into memory. So I decided to try Hibernate, which worked just
>>>> fine.
>>>> 
>>>> Finally I found queryForPaginatedList was deprecated and that I should use
>>>> queryWithRowHandler instead, so I made my own implementation of the
>>>> SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I had
>>>> the same result (I profiled the application stack usage and I could see a
>>>> line growing steadily and finally crash against the stack ceiling while
>>>> performing the select statement).
>>>> 
>>>> Shay Banon, Compass project leader, is planning to patch
>>>> SqlMapClientGpsDevice by replacing queryForPaginatedList with
>>>> queryWithRowHandler, but I'm doubtful it's gonna work.
>>>> 
>>>> Finally you can, of course, devide the select stament into several selects
>>>> (and it works) but, in my opinion, that shouldn't be necessary.
>>>> You can read more on
>>>> http://forum.compass-project.org/thread.jspa?threadID=215278
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ----- Original Message ----
>>>> From: Clinton Begin <clinton.begin@gmail.com>
>>>> To: user-java@ibatis.apache.org
>>>> 
>>>> 
>>>> 
>>>> Sent: Monday, March 17, 2008 2:22:39 PM
>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>> 
>>>> Wow, this thread is interesting.   I suppose I could read the past emails,
>>>> but could someone summarize the problem and progress to date?  I may be
>>>> able to shed some light on what is happening.
>>>> 
>>>> Clinton
>>>> 
>>>> 
>>>> On Sun, Mar 16, 2008 at 11:49 PM, nch <underscore_dot@yahoo.com> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Another thing I forgot. This works perfectly well if we replace iBatis
by
>>>>> Hibernate, so I don't think this is being caused by the MySQL driver.
>>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----
>>>>> From: nch <underscore_dot@yahoo.com>
>>>>> To: user-java@ibatis.apache.org
>>>>> 
>>>>> 
>>>>> 
>>>>> Sent: Sunday, March 16, 2008 10:18:42 PM
>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Well, I modified the mediumblob into a mediumtext and removed all other
>>>>> fields except the id one.
>>>>> 
>>>>> 
>>>>> ----- Original Message ----
>>>>> From: nch <underscore_dot@yahoo.com>
>>>>> To: user-java@ibatis.apache.org
>>>>> Sent: Sunday, March 16, 2008 10:06:34 PM
>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> You're probably right. I'm using the table called "text" which contains
a
>>>>> field of type "mediumblob".
>>>>> CREATE TABLE /*$wgDBprefix*/text (
>>>>> old_id int unsigned NOT NULL auto_increment,
>>>>> old_text mediumblob NOT NULL,
>>>>> old_flags tinyblob NOT NULL,
>>>>> PRIMARY KEY old_id (old_id)
>>>>> ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> You can find it's definition here:
>>>>> http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.
>>>>> sql?view=markup
>>>>> 
>>>>> I'll try to modify that field, first, and see what happens.
>>>>> I've already tried "select * from your_wikipedia_dump limit $start$,
>>>>> $size$" and it seems to work fine, but I see a possible issue with this,
>>>>> you can not remove any of the records you've already read during the
whole
>>>>> process. This might not be an option.
>>>>> 
>>>>> Thank you.
>>>>> 
>>>>> 
>>>>> ----- Original Message ----
>>>>> From: Larry Meadors <larry.meadors@gmail.com>
>>>>> To: user-java@ibatis.apache.org
>>>>> Sent: Sunday, March 16, 2008 4:19:43 PM
>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>>> 
>>>>> Yeah, I just noticed that - it looks like it's failing in the jdbc
>>>>> driver when it tries to read a blob, maybe?
>>>>> 
>>>>> I know that some versions of the postgresql jdbc driver load the
>>>>> entire contents of a resultset into a massive byte[][] structure. I
>>>>> wonder if that is what is happening here? I think you can get the
>>>>> source for the MySQL driver and check that if you want to.
>>>>> 
>>>>> If it is, you may need to structure your app to take smaller bites of
>>>>> this elephant.
>>>>> 
>>>>> Try "select * from your_wikipedia_dump limit $start$, $size$" instead
>>>>> - that will limit the size of the results to $size$ rows, starting on
>>>>> row $start$ (the $start$ value is zero-based). If you experiment to
>>>>> see how big $size$ can get, you should be able to get pretty decent
>>>>> performance out of this.
>>>>> 
>>>>> Psuedo-code would be like this...
>>>>> 
>>>>> int start = 0;
>>>>> int size = 1000; // see how big you can make this to improve performance
>>>>> while(still_more_data){
>>>>>   still_more_data = index(start, size);
>>>>>   start += size;
>>>>> }
>>>>> 
>>>>> Your index method will return true if the query returned any data. If
>>>>> the query returns no data, it'll return false.
>>>>> 
>>>>> I can't imagine what Hibernate is doing differently here to make this
>>>>> work if the JDBC driver is failing when executing the query.
>>>>> 
>>>>> Larry
>>>>> 
>>>>> 
>>>>> On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com>
wrote:
>>>>>> 
>>>>>> 
>>>>>> Sure. Please, see attached.
>>>>>> I don't think the problem is in the RowHandler, though, because the
>>>>>> OutOfMemoryError occurs before invoking RowHandle#handleRow.
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> 
>>>>>> ----- Original Message ----
>>>>>> From: Larry Meadors <larry.meadors@gmail.com>
>>>>>> To: user-java@ibatis.apache.org
>>>>>> 
>>>>>> Sent: Sunday, March 16, 2008 3:13:27 AM
>>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>>>> 
>>>>>>  Can you post the row handler you are using.
>>>>>> 
>>>>>> Larry
>>>>>> 
>>>>>> 
>>>>>> On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com>
wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Hi, Nathan.
>>>>>>> I did so, but I'm still having the same issue. Perhaps I'm not
using it
>>>>>>> correctly?
>>>>>>> See my last post to this forum entry:
>>>>>>> 
>>>>>>> http://forum.compass-project.org/thread.jspa?threadID=215278
>>>>>>> 
>>>>>>> See the stack trace:
>>>>>>> 
>>>>>>> MemoryError: Java heap space:
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
>>>>>>>        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
>>>>>>>        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
>>>>>>>        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
>>>>>>>        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
>>>>>>>        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
>>>>>>>        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
>>>>>>>        at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
>>>>>>>        at
>>>>>>> 
>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1
>>>>>> 268)
>>>>>>>        at
>>>>>>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
>>>>>>>        at
>>>>>>> 
>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre
>>>>>> paredStatement.java:169)
>>>>>>>        at
>>>>>>> 
>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre
>>>>>> paredStatement.java:169)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.j
>>>>>> ava:186)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQue
>>>>>> ry(GeneralStatement.java:205)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW
>>>>>> ithCallback(GeneralStatement.java:173)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW
>>>>>> ithRowHandler(GeneralStatement.java:133)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(
>>>>>> SqlMapExecutorDelegate.java:649)
>>>>>>>        at
>>>>>>> 
>>>>>> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMa
>>>>>> pSessionImpl.java:156)
>>>>>>>        at
>>>>>>> 
>>>>>> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClien
>>>>>> tGpsDeviceWithRowHandler.java:33)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(Abstr
>>>>>> actGpsDevice.java:93)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallback
>>>>>> WithoutResult.java:29)
>>>>>>>        at
>>>>>>> org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.ja
>>>>>> va:161)
>>>>>>>        at
>>>>>>> 
org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91>>>>>>>
)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTra
>>>>>> nsactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
>>>>>>>        at
>>>>>>> 
>>>>>> org.springframework.transaction.support.TransactionCallbackWithoutResult.
>>>>>> doInTransaction(TransactionCallbackWithoutResult.java:33)
>>>>>>>        at
>>>>>>> 
>>>>>> org.springframework.transaction.support.TransactionTemplate.execute(Trans
>>>>>> actionTemplate.java:128)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(Spr
>>>>>> ingSyncTransactionGpsDeviceWrapper.java:96)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassG
>>>>>> ps.java:133)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana
>>>>>> ger$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
>>>>>>>        at
>>>>>>> 
>>>>>> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana
>>>>>> ger.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ----- Original Message ----
>>>>>>> From: Nathan Maves <nathan.maves@gmail.com>
>>>>>>> To: user-java@ibatis.apache.org
>>>>>>> 
>>>>>>> Sent: Saturday, March 15, 2008 5:43:04 PM
>>>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>>>>>>> 
>>>>>>>  this is only my 2 cents but I would throw that class out.  Write
your
>>>>>>> own
>>>>>>> implementation which uses a row handler.  This is the type of
situation
>>>>>>> in
>>>>>>> which a row handler could really help.  almost no memory would
be used.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com>
wrote:
>>>>>>>> 
>>>>>>>> Yes, I agree that dividing the query into several queries
should do as
>>>>>>> long as none of them uses more memory than available. Gracias,
Carlos.
>>>>>>>> 
>>>>>>>> Nevertheless, I looked into
>>>>>>> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
>>>>>>> fundamentally, what it does is a queryForPaginatedList and iterates
>>>>>> through
>>>>>>> the resulting PaginatedList with nextPage()
>>>>>>>> 
>>>>>>>> I guess that the reason why pages are kept in memory after
performing
>>>>>>> paginatedList.nextPage() is because a paginatedList.previousPage()
can
>>>>>>> be
>>>>>>> issued but, should't it free pages if heap space is running out?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx> wrote:
>>>>>>>> 
>>>>>>>> Since we have a similar trouble around here i must say that
you have
>>>>>>>> lot
>>>>>>> of options when going to compass..
>>>>>>>> The first one is to index "object by object" by hand, the
second one is
>>>>>> to
>>>>>>> use the Gps wich will use the complete bunch of data and make
subindexes
>>>>>>> (making parts of your "main index" with diferent "maps" for each
>>>>>>> subindex
>>>>>> in
>>>>>>> your gpsDriver.
>>>>>>>> if your indexing is as strong as ours i should look forward
for numer 1
>>>>>> or
>>>>>>> number 3 depending of your needs. (More a Compass than iBatis
issue)
>>>>>>>> Greetings
>>>>>>>> Carlos de Luna
>>>>>>>> 
>>>>>>>> ----- Mensaje original ----
>>>>>>>> De: nch
>>>>>>>> 
>>>>>>>> Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
>>>>>>>> Enviado: jueves, 13 de marzo, 2008 9:37:04
>>>>>>>> Asunto: Re: select * causing " OutOfMemoryError: Java heap
space"
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Sorry. Yes, I need all the data at one time, because the
Compass API
>>>>>> seems
>>>>>>> to be meant in that way.
>>>>>>>> 
>>>>>>>> Larry Meadors wrote: OK, so the answer to the question is...[
yes, i
>>>>>> need
>>>>>>> all the data at
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> one time | no, i do not need all of the data at one time
].
>>>>>>>> 
>>>>>>>> Larry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
>>>>>>>>> 
>>>>>>>>> Hi!
>>>>>>>>> 
>>>>>>>>> I'm doing tests both under tomcat and jetty. I set 1024MB
of heap
>>>>>> space
>>>>>>> in
>>>>>>>>> both cases. I am using a profiler to look into what's
happening during
>>>>>>>>> execution of the indexing process and I can tell that's
the exact
>>>>>> amount
>>>>>>> it
>>>>>>>>> displays for the VM.
>>>>>>>>> 
>>>>>>>>> As I describe in the Compass user forum (see link bellow),
I'm
>>>>>> indexing
>>>>>>> a
>>>>>>>>> Wikipedia dump of about 650000-675000 records. That's
sort of 1-1.5GB
>>>>>> of
>>>>>>>>> data.
>>>>>>>>> 
>>>>>>>>> Hibernate can handle all that data, I guess it's using
some sort of
>>>>>>>>> pagination. iBatis seems to divide the select statement
into several
>>>>>>> select
>>>>>>>>> statements, but it also seems each of such statements
surpasses the
>>>>>>> limit of
>>>>>>>>> 1024MB when trying to index only 300000 of the database
records.
>>>>>>>>> 
>>>>>>>>> Thank you
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Chris Lamey wrote:
>>>>>>>>> What is you JVM heap size set to and how much data is
returned by
>>>>>> select
>>>>>>> *
>>>>>>>>> from table?
>>>>>>>>> 
>>>>>>>>> If you're trying to pull back 1G worth of data into a
JVM with a heap
>>>>>>> size
>>>>>>>>> set to 64M, you will hit the heap limit pretty quick.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: nch [mailto:underscore_dot@yahoo..com]
>>>>>>>>> Sent: Thu 3/13/2008 8:03 AM
>>>>>>>>> To: user-java@ibatis.apache.org
>>>>>>>>> Subject: select * causing " OutOfMemoryError: Java heap
space"
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi, everybody.
>>>>>>>>> I'm testing the Compass search engine and, in particular,
how to index
>>>>>> a
>>>>>>> big
>>>>>>>>> set of documents from a table in a MySQL database. In
order to do this
>>>>>> I
>>>>>>>>> issue a "select * from table_name" using iBatis, but
this seems to be
>>>>>>>>> causing the application to use all heap space available.
>>>>>>>>> I added a new entry into Compass user forum with details:
>>>>>>>>> http://forum.compass-project.org/thread.jspa?threadID=215278
>>>>>>>>> 
>>>>>>>>> Can you figure out why is this happening?
>>>>>>>>> 
>>>>>>>>> Many thanks
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ---------------------------------
>>>>>>>>> Be a better friend, newshound, and know-it-all with Yahoo!
Mobile. Try
>>>>>>> it
>>>>>>>>> now.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ________________________________
>>>>>>>>> Never miss a thing. Make Yahoo your homepage.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Never miss a thing. Make Yahoo your homepage.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> _________________________________________________________________________
>>>>>> ___________
>>>>>>>> ¡Capacidad ilimitada de almacenamiento en tu correo!
>>>>>>>> No te preocupes más por el espacio de tu cuenta con Correo
Yahoo!:
>>>>>>>> http://correo.espanol.yahoo.com/
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ________________________________
>>>>>>> Never miss a thing. Make Yahoo your homepage.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>  ________________________________
>>>>>>> Never miss a thing. Make Yahoo your homepage.
>>>>>> 
>>>>>> 
>>>>>>  ________________________________
>>>>>> 
>>>>>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile.
Try it
>>>>>> now.
>>>>> 
>>>>> 
>>>>> ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>>>>> 
>>>>> ________________________________
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ________________________________
> Never miss a thing. Make Yahoo your homepage.
>>> 
>>> 
>> 
>> 
>> 
>> ________________________________
> 
>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
>> now.
> 
> 
> 
> 
> 
> 
>       
> ______________________________________________________________________________
> ______
> Never miss a thing.  Make Yahoo your home page.
> http://www.yahoo.com/r/hs


Mime
View raw message