ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Maves" <nathan.ma...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 16:49:30 GMT
did you try defaultAutoCommit=false for the connection pool?

On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com> wrote:

>
> Hi!
>
> Clinton, answering to your last mail:
> - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currently
> using) with MySQL 5.0.45 (Debian). Tried also on Windows with the same
> drivers, but can not remember the MySQL version.
> - Bellow are, I think I don't miss any, the different relevant
> configurations (some through Spring):
>
> - jdbc.properties: Database connection config
>
> jdbc.driverClassName=com.mysql.jdbc.Driver
>
> jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=true&amp
> ;useUnicode=true&amp;characterEncoding=utf-8
> jdbc.username=x
> jdbc.password=y
>
>
> - sql-map-config.xml: iBatis config file
>
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map Config
> 2.0//EN"
>     "http://ibatis.apache.org/dtd/sql-map-config-2.dtd">
>
> <sqlMapConfig>
>     <settings enhancementEnabled="true" maxTransactions="20"
> maxRequests="32" maxSessions="10"/>
>
>     <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/>
>
> </sqlMapConfig>
>
>
> - TestingPojoSQL.xml: The operations needed for indexing:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "
> http://ibatis.apache.org/dtd/sql-map-2.dtd">
>
> <sqlMap namespace="TestingPojoSQL">
>     <typeAlias alias="article" type="org.myorg.model.TestingPojo"/>
>
>     <resultMap id="articleResult" class="article">
>         <result property="id" column="id"/>
>         <result property="contents" column="contents"/>
>     </resultMap>
>
>     <select id="getAllTestingPojos" resultMap="articleResult"
> fetchSize="100" resultSetType="FORWARD_ONLY">
>     <![CDATA[
>         select * from articles
>     ]]>
>     </select>
>
> </sqlMap>
>
>
> - applicationContext-resources.xml: Here a DBCP data source is defined (I
> tried also C3P0. Made no difference):
>
> <?xml version="1.0" encoding="UTF-8"?>
>
>        xmlns:jee="http://www.springframework.org/schema/jee"
>        xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>             http://www.springframework.org/schema/jee
> http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
>
>     <bean id="propertyConfigurer" class="
> org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <list>
>                 <value>classpath:jdbc.properties</value>
>             </list>
>         </property>
>     </bean>
>
>     <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
>         <property name="driverClassName" value="${jdbc.driverClassName}"/>
>         <property name="url" value="${jdbc.url}"/>
>         <property name="username" value="${jdbc.username}"/>
>         <property name="password" value="${jdbc.password}"/>
>         <property name="maxActive" value="100"/>
>         <property name="maxWait" value="1000"/>
>         <property name="poolPreparedStatements" value="true"/>
>         <property name="defaultAutoCommit" value="true"/>
>     </bean>
>
> </beans>
>
>
> - In applicationContext-dao.xml: the sqlMapClient
>
> <?xml version="1.0" encoding="UTF-8"?>
>
>        xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"
>        default-lazy-init="true">
>
>     <bean id="transactionManager" class="
> org.springframework.jdbc.datasource.DataSourceTransactionManager">
>         <property name="dataSource" ref="dataSource"/>
>     </bean>
>
>     <bean id="sqlMapClient" class="
> org.springframework.orm.ibatis.SqlMapClientFactoryBean">
>         <property name="configLocation">
>             <value>classpath:/sql-map-config.xml</value>
>         </property>
>         <property name="dataSource" ref="dataSource"/>
>     </bean>
> </beans>
>
>
> - In applicationContext-service.xml: This is where I tell Spring to
> instanciate an iBatis based GPS device.
>
> <bean id="compass" class="org.compass.spring.LocalCompassBean">
>     <property name="classMappings">
>         <list>
>             <value>org.myorg.TestingPojo</value>
>         </list>
>     </property>
>     <property name="compassSettings">
>         <props>
>             <prop key="compass.engine.connection">file://compass</prop>
>             <prop key="compass.transaction.factory">
> org.compass.spring.transaction.SpringSyncT ransactionFactory</prop>
>             <prop key="compass.engine.analyzer.default.type
> ">snowball</prop>
>             <prop key="compass.engine.analyzer.default.name
> ">English</prop>
>             <!-- Mainly default values -->
>             <prop key="compass.engine.ramBufferSize">16</prop>
>             <prop key="compass.engine.maxBufferedDocs">10</prop>
>             <prop key="compass.engine.maxBufferedDeletedTerms">-1</prop>
>             <prop key="compass.engine.mergeFactor">10</prop>
>             <prop key="compass.engine.maxFieldLength">10000</prop>
>             <prop key="compass.engine.cacheIntervalInvalidation
> ">5000</prop>
>             <prop key="compass.engine.indexManagerScheduleInterval
> ">60</prop>
>         </props>
>     </property>
>     <property name="transactionManager" ref="transactionManager" />
> </bean>
>
> <bean id="iBatisGpsDevice" class="
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
>     <property name="name" value="iBatisDevice" />
>     <property name="sqlMapClient" ref="sqlMapClient" />
>     <property name="selectStatementsIds">
>         <list>
>             <value>getAllTestingPojos</value>
>         </list>
>     </property>
>     <property name="pageSize" value="50" />
> </bean>
>
> <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps"
> init-method="start" destroy-method="stop">
>     <property name="compass">
>         <ref bean="compass" />
>     </property>
>     <property name="gpsDevices">
>         <list>
>             <bean class="
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
>                 <property name="gpsDevice" ref="iBatisGpsDevice" />
>             </bean>
>         </list>
>     </property>
> </bean>
>
>
> Regarding your previous mail:
> - I emptied the handleRow method and ran the test - same result. The
> exception occurs before handleRow is invoked.
> - I'm not configuring any caches. Is there a default one? Should I config
> one?
> - Setting fetchSize, different resultSetTypes, use of <select> or
> <statement> elements didn't make any differences.
> - Additionally, I tried truncating the contents of each Wikipedia article
> to 500 characters, and I was able to index all the 650000 rows, so the
> number of records being selected is not the only factor here (I just wanted
> to make sure).
> - I can imagine others (including you) have successfully made it with
> large amounts of data (I'm new to iBatis and Compass/Lucene) so I'm sure
> there's a solution.
>
> Lots of thanks.
>
>
>
>
> ----- Original Message ----
> From: Clinton Begin <clinton.begin@gmail.com>
> To: user-java@ibatis.apache.org
> Sent: Monday, March 17, 2008 11:18:59 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> Wow, I just read that other thread on the Compass site.... This is not
> directly an iBATIS problem at all...
>
> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
>
> This line tells me that the execution is never returning from the prepared
> statement.  iBATIS never has a chance to even see the ResultSet, let alone
> get to the RowHandler. So swapping out the GpsDevice here won't make any
> difference at all. This is odd behavior for a JDBC driver and tells me that
> something is very wrong with the configuration.
>
> Which version of the JDBC driver are you using?  What are the rest of the
> iBATIS convfiguration details?
>
> Clinton
>
>
> On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <clinton.begin@gmail.com>
> wrote:
>
> > Thanks for the summary.  A few things:
> >
> > * PaginatedList was definitely the wrong solution for that, so I'm glad
> > it's gone.
> >
> > * A RowHandler is possibly the right solution, but will only store as
> > many records in memory as you tell it to.  iBATIS does not just arbitrarily
> > keep records around.  Quite the opposite actually.  Unless you keep a
> > reference or configure a cache, iBATIS will not keep the object around at
> > all.
> >
> > Looking at your implementation, there are a couple of things going on...
> >
> > session.create(o);
> > if(currentItem== pageSize){
> >   if (log.isDebugEnabled()) {
> >     log.debug("Indexing page number ["+ pageCount++ + "]");
> >   }
> > session.evictAll();
> >
> > Things to Try:
> >
> > 1)  These lines of code must be treated as guilty until proven innocent
> > (especially the logging).  The easiest way to do that is to delete them.
> > Create an empty handler/GpsDevice and run the test to see what happens.
> >
> > 2) Look at the iBATIS configuration files.  Is a cache configured for
> > this query?  What kind of cache?
> >
> > 3) You could try multiple queries of fixed sizes with
> > queryForList(String,Object,int offset,int limit) ... but I can't see how
> > that would be better than a RowHandler.
> >
> > 4) Next, just in case the driver's default fetch size is ridiculously
> > high (or unlimited), try setting <select ... fetchSize="100"/>.  You could
> > also try a scrollable SesultSet with <statement ... resultSetType /> of
> > SCROLL_INSENSITIVE or SCROLL_SENSITIVE.
> >
> > Finally, please know that I've used iBATIS to load tens of millions of
> > rows for ETL and even analysis (NetFlix Prize) and was able to do so with a
> > RowHandler or even simply multiple calls to queryForList(String,Object,int
> > offset,int limit).  I don't recall having to even think about the
> > configuration, although I did manage to optimize it to achieve a read/insert
> > performance of 15,000 records per second between two databases (on a single
> > machine too).
> >
> > I'm sure there's something about the configuration here that is causing
> > the problem, and it can be solved.
> >
> > I hope one of these helps.
> >
> > Clinton
> >
> >
> > On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com> wrote:
> >
> > >
> > > Hi, there. :-)
> > >
> > > The thing is, I'm testing a search engine called Compass. One of the
> > > tests consists of indexing a Wikipedia dump. In this case only the text of
> > > the articles, so I downloaded it and imported it into a MySQL database
> > > (about 650000 records - 1.5GB).
> > >
> > > You can configure Compass to access a database and index it's contents
> > > by configuring what they call a GPS Device. You can do that through an ORM
> > > such as iBatis, Hibernate or JPA.
> > >
> > > Compass provides an implementation of such a GPS Device called
> > > SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the
> > > results of a query and, so, index them. The query is just a "select * from
> > > articles_table".
> > >
> > > So I wired everything up and ran several tests with different amounts
> > > of data to be indexed and JVM stack space sizes. The result was an
> > > "OutOfMemoryError: java heap space" error message whenever the size of the
> > > selected rows was bigger than the amount of available heap space (I can not
> > > index the 650000 records having 1GB of stack).
> > >
> > > After doing some profiling I concluded iBatis was loading the whole
> > > amount of results into memory. So I decided to try Hibernate, which worked
> > > just fine.
> > >
> > > Finally I found queryForPaginatedList was deprecated and that I should
> > > use queryWithRowHandler instead, so I made my own implementation of the
> > > SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I had
> > > the same result (I profiled the application stack usage and I could see a
> > > line growing steadily and finally crash against the stack ceiling while
> > > performing the select statement).
> > >
> > > Shay Banon, Compass project leader, is planning to patch
> > > SqlMapClientGpsDevice by replacing queryForPaginatedList with
> > > queryWithRowHandler, but I'm doubtful it's gonna work.
> > >
> > > Finally you can, of course, devide the select stament into several
> > > selects (and it works) but, in my opinion, that shouldn't be necessary.
> > > You can read more on
> > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > >
> > > Thanks!
> > >
> > >
> > > ----- Original Message ----
> > > From: Clinton Begin <clinton.begin@gmail.com>
> > > To: user-java@ibatis.apache.org
> > > Sent: Monday, March 17, 2008 2:22:39 PM
> > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > >
> > > Wow, this thread is interesting.   I suppose I could read the past
> > > emails, but could someone summarize the problem and progress to date?  I may
> > > be able to shed some light on what is happening.
> > >
> > > Clinton
> > >
> > > On Sun, Mar 16, 2008 at 11:49 PM, nch <underscore_dot@yahoo.com>
> > > wrote:
> > >
> > > >
> > > > Another thing I forgot. This works perfectly well if we replace
> > > > iBatis by Hibernate, so I don't think this is being caused by the MySQL
> > > > driver.
> > > >
> > > > ----- Original Message ----
> > > > From: nch <underscore_dot@yahoo.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 10:18:42 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > Well, I modified the mediumblob into a mediumtext and removed all
> > > > other fields except the id one.
> > > >
> > > > ----- Original Message ----
> > > > From: nch <underscore_dot@yahoo.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 10:06:34 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > You're probably right. I'm using the table called "text" which
> > > > contains a field of type "mediumblob".
> > > >
> > > > CREATE TABLE /*$wgDBprefix*/text (
> > > >   old_id int unsigned NOT NULL auto_increment,
> > > >   old_text mediumblob NOT NULL,
> > > >   old_flags tinyblob NOT NULL,
> > > >   PRIMARY KEY old_id (old_id)
> > > > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
> > > >
> > > >
> > > >
> > > > You can find it's definition here:
> > > >
> > > > http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
> > > >
> > > > I'll try to modify that field, first, and see what happens.
> > > > I've already tried "select * from your_wikipedia_dump limit $start$,
> > > > $size$" and it seems to work fine, but I see a possible issue with this,
you
> > > > can not remove any of the records you've already read during the whole
> > > > process. This might not be an option.
> > > >
> > > > Thank you.
> > > >
> > > > ----- Original Message ----
> > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 4:19:43 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > > Yeah, I just noticed that - it looks like it's failing in the jdbc
> > > > driver when it tries to read a blob, maybe?
> > > >
> > > > I know that some versions of the postgresql jdbc driver load the
> > > > entire contents of a resultset into a massive byte[][] structure. I
> > > > wonder if that is what is happening here? I think you can get the
> > > > source for the MySQL driver and check that if you want to.
> > > >
> > > > If it is, you may need to structure your app to take smaller bites
> > > > of
> > > > this elephant.
> > > >
> > > > Try "select * from your_wikipedia_dump limit $start$, $size$"
> > > > instead
> > > > - that will limit the size of the results to $size$ rows, starting
> > > > on
> > > > row $start$ (the $start$ value is zero-based). If you experiment to
> > > > see how big $size$ can get, you should be able to get pretty decent
> > > > performance out of this.
> > > >
> > > > Psuedo-code would be like this...
> > > >
> > > > int start = 0;
> > > > int size = 1000; // see how big you can make this to improve
> > > > performance
> > > > while(still_more_data){
> > > >   still_more_data = index(start, size);
> > > >   start += size;
> > > > }
> > > >
> > > > Your index method will return true if the query returned any data.
> > > > If
> > > > the query returns no data, it'll return false.
> > > >
> > > > I can't imagine what Hibernate is doing differently here to make
> > > > this
> > > > work if the JDBC driver is failing when executing the query.
> > > >
> > > > Larry
> > > >
> > > >
> > > > On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > >
> > > > >
> > > > > Sure. Please, see attached.
> > > > > I don't think the problem is in the RowHandler, though, because
> > > > the
> > > > > OutOfMemoryError occurs before invoking RowHandle#handleRow.
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > To: user-java@ibatis.apache.org
> > > > >
> > > > > Sent: Sunday, March 16, 2008 3:13:27 AM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > >  Can you post the row handler you are using.
> > > > >
> > > > > Larry
> > > > >
> > > > >
> > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > > >
> > > > > >
> > > > > > Hi, Nathan.
> > > > > > I did so, but I'm still having the same issue. Perhaps I'm not
> > > > using it
> > > > > > correctly?
> > > > > > See my last post to this forum entry:
> > > > > >
> > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > >
> > > > > > See the stack trace:
> > > > > >
> > > > > > MemoryError: Java heap space:
> > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > > > > >        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java
> > > > :318)
> > > > > >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> > > > > >        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java
> > > > :2333)
> > > > > >        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> > > > > >        at
> > > > > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java
> > > > :2040)
> > > > > >        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java
> > > > :1443)
> > > > > >        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java
> > > > :1777)
> > > > > >        at com.mysql.jdbc.Connection.execSQL(Connection.java
> > > > :3249)
> > > > > >        at
> > > > > >
> > > > > com.mysql.jdbc.PreparedStatement.executeInternal(
> > > > PreparedStatement.java:1268)
> > > > > >        at
> > > > > > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java
> > > > :843)
> > > > > >        at
> > > > > >
> > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> > > > DelegatingPreparedStatement.java:169)
> > > > > >        at
> > > > > >
> > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> > > > DelegatingPreparedStatement.java:169)
> > > > > >        at
> > > > > >
> > > > > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(
> > > > SqlExecutor.java:186)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery
> > > > (GeneralStatement.java:205)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback
> > > > (GeneralStatement.java:173)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler
> > > > (GeneralStatement.java:133)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler
> > > > (SqlMapExecutorDelegate.java:649)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(
> > > > SqlMapSessionImpl.java:156)
> > > > > >        at
> > > > > >
> > > > > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(
> > > > SqlMapClientGpsDeviceWithRowHandler.java:33)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(
> > > > AbstractGpsDevice.java:93)
> > > > > >        at
> > > > > >
> > > > > org.compass.core.CompassCallbackWithoutResult.doInCompass(
> > > > CompassCallbackWithoutResult.java:29)
> > > > > >        at
> > > > > > org.compass.core.CompassTemplate.execute(CompassTemplate.java
> > > > :132)
> > > > > >        at
> > > > > >
> > > > > org.compass.gps.impl.SingleCompassGps.executeForIndex(
> > > > SingleCompassGps.java:161)
> > > > > >        at
> > > > > > org.compass.gps.device.AbstractGpsDevice.index(
> > > > AbstractGpsDevice.java:91)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult
> > > > (SpringSyncTransactionGpsDeviceWrapper.java:98)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction
> > > > (TransactionCallbackWithoutResult.java:33)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.springframework.transaction.support.TransactionTemplate.execute(
> > > > TransactionTemplate.java:128)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index
> > > > (SpringSyncTransactionGpsDeviceWrapper.java:96)
> > > > > >        at
> > > > > >
> > > > > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(
> > > > SingleCompassGps.java:133)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep
> > > > (DefaultLuceneSearchEngineIndexManager.java:233)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate
> > > > (DefaultLuceneSearchEngineIndexManager.java:182)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ----- Original Message ----
> > > > > > From: Nathan Maves <nathan.maves@gmail.com>
> > > > > > To: user-java@ibatis.apache.org
> > > > > >
> > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM
> > > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap
> > > > space"
> > > > > >
> > > > > >  this is only my 2 cents but I would throw that class out.
> > > > Write your own
> > > > > > implementation which uses a row handler.  This is the type of
> > > > situation in
> > > > > > which a row handler could really help.  almost no memory would
> > > > be used.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > > > >
> > > > > > > Yes, I agree that dividing the query into several queries
> > > > should do as
> > > > > > long as none of them uses more memory than available. Gracias,
> > > > Carlos.
> > > > > > >
> > > > > > > Nevertheless, I looked into
> > > > > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex
and,
> > > > > > fundamentally, what it does is a queryForPaginatedList and
> > > > iterates
> > > > > through
> > > > > > the resulting PaginatedList with nextPage()
> > > > > > >
> > > > > > > I guess that the reason why pages are kept in memory after
> > > > performing
> > > > > > paginatedList.nextPage() is because a paginatedList.previousPage()
> > > > can be
> > > > > > issued but, should't it free pages if heap space is running
out?
> > > > > > >
> > > > > > >
> > > > > > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>
wrote:
> > > > > > >
> > > > > > > Since we have a similar trouble around here i must say
that
> > > > you have lot
> > > > > > of options when going to compass..
> > > > > > > The first one is to index "object by object" by hand, the
> > > > second one is
> > > > > to
> > > > > > use the Gps wich will use the complete bunch of data and make
> > > > subindexes
> > > > > > (making parts of your "main index" with diferent "maps" for
each
> > > > subindex
> > > > > in
> > > > > > your gpsDriver.
> > > > > > > if your indexing is as strong as ours i should look forward
> > > > for numer 1
> > > > > or
> > > > > > number 3 depending of your needs. (More a Compass than iBatis
> > > > issue)
> > > > > > > Greetings
> > > > > > > Carlos de Luna
> > > > > > >
> > > > > > > ----- Mensaje original ----
> > > > > > > De: nch
> > > > > > >
> > > > > > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > > > > > Asunto: Re: select * causing " OutOfMemoryError: Java heap
> > > > space"
> > > > > > >
> > > > > > >
> > > > > > > Sorry. Yes, I need all the data at one time, because the
> > > > Compass API
> > > > > seems
> > > > > > to be meant in that way.
> > > > > > >
> > > > > > > Larry Meadors wrote: OK, so the answer to the question
is...[
> > > > yes, i
> > > > > need
> > > > > > all the data at
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > one time | no, i do not need all of the data at one time
].
> > > > > > >
> > > > > > > Larry
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I'm doing tests both under tomcat and jetty. I set
1024MB of
> > > > heap
> > > > > space
> > > > > > in
> > > > > > > > both cases. I am using a profiler to look into what's
> > > > happening during
> > > > > > > > execution of the indexing process and I can tell that's
the
> > > > exact
> > > > > amount
> > > > > > it
> > > > > > > > displays for the VM.
> > > > > > > >
> > > > > > > > As I describe in the Compass user forum (see link
bellow),
> > > > I'm
> > > > > indexing
> > > > > > a
> > > > > > > > Wikipedia dump of about 650000-675000 records. That's
sort
> > > > of 1-1.5GB
> > > > > of
> > > > > > > > data.
> > > > > > > >
> > > > > > > > Hibernate can handle all that data, I guess it's using
some
> > > > sort of
> > > > > > > > pagination. iBatis seems to divide the select statement
into
> > > > several
> > > > > > select
> > > > > > > > statements, but it also seems each of such statements
> > > > surpasses the
> > > > > > limit of
> > > > > > > > 1024MB when trying to index only 300000 of the database
> > > > records.
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Chris Lamey wrote:
> > > > > > > > What is you JVM heap size set to and how much data
is
> > > > returned by
> > > > > select
> > > > > > *
> > > > > > > > from table?
> > > > > > > >
> > > > > > > > If you're trying to pull back 1G worth of data into
a JVM
> > > > with a heap
> > > > > > size
> > > > > > > > set to 64M, you will hit the heap limit pretty quick.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > > > > > Sent: Thu 3/13/2008 8:03 AM
> > > > > > > > To: user-java@ibatis.apache.org
> > > > > > > > Subject: select * causing " OutOfMemoryError: Java
heap
> > > > space"
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi, everybody.
> > > > > > > > I'm testing the Compass search engine and, in particular,
> > > > how to index
> > > > > a
> > > > > > big
> > > > > > > > set of documents from a table in a MySQL database.
In order
> > > > to do this
> > > > > I
> > > > > > > > issue a "select * from table_name" using iBatis, but
this
> > > > seems to be
> > > > > > > > causing the application to use all heap space available.
> > > > > > > > I added a new entry into Compass user forum with details:
> > > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > > >
> > > > > > > > Can you figure out why is this happening?
> > > > > > > >
> > > > > > > > Many thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > ---------------------------------
> > > > > > > > Be a better friend, newshound, and know-it-all with
Yahoo!
> > > > Mobile. Try
> > > > > > it
> > > > > > > > now.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > ____________________________________________________________________________________
> > > > > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > > > > No te preocupes más por el espacio de tu cuenta con Correo
> > > > Yahoo!:
> > > > > > > http://correo.espanol.yahoo.com/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >  ________________________________
> > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > >
> > > > >
> > > > >  ________________________________
> > > > >
> > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile.
> > > > Try it
> > > > > now.
> > > >
> > > >
> > > > ------------------------------
> > > > Looking for last minute shopping deals? Find them fast with Yahoo!
> > > > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
> > > >
> > > >
> > > > ------------------------------
> > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> > > > it now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ%20>
> > > >
> > > >
> > > > ------------------------------
> > > > Looking for last minute shopping deals? Find them fast with Yahoo!
> > > > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
> > > >
> > >
> > >
> > >
> > > ------------------------------
> > > Never miss a thing. Make Yahoo your homepage.<http://us.rd.yahoo.com/evt=51438/*http://www.yahoo.com/r/hs>
> > >
> >
> >
>
>
> ------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ>
>
Mime
View raw message