ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clinton Begin" <clinton.be...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 18:53:43 GMT
That's why I get paid the big bucks.  i.e. $0   -- LOL.

I'm surprised nobody has run into it before, but I'm very glad that worked.
:-)

Thanks to the list for putting up with possibly our longest thread in
history. Thank you to MySQL for using magic numbers.

Cheers,
Clinton


On Tue, Mar 18, 2008 at 12:24 PM, nch <underscore_dot@yahoo.com> wrote:

>
> So-lu-tion.
> But, yeah, WOW!!
>
> Clinton, you've been really helpful.
> Lots of thanks to you and all people in this mailing list.
>
> ----- Original Message ----
> From: Clinton Begin <clinton.begin@gmail.com>
> To: user-java@ibatis.apache.org
> Sent: Tuesday, March 18, 2008 6:45:28 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> SOLUTION?
>
> A quick Google search yielded this thread from the MySQL boards which
> details the exact problem your having.
>
> http://lists.mysql.com/java/9137
>
> It's actually quite silly and definitely a MySQL driver issue.  I
> imagine that Hibernate's MySQL dialect configuration is aware of this
> problem, which is why it works.  Recall what I said earlier regarding
> the fact that the driver throws the exception from the
> PreparedStatement.execute() method... thus it's not even getting to
> the point of returning a result set.  This is why:
>
> // MySQL Driver code....
> protected boolean createStreamingResultSet() {
>   return ((this.resultSetType == java.sql.ResultSet.TYPE_FORWARD_ONLY)
>   && (this.resultSetConcurrency ==  java.sql.ResultSet.CONCUR_READ_ONLY)
>   && (this.fetchSize ==  Integer.MIN_VALUE)); // NOTICE THIS.... WTF!?
> }
>
> Turns out that for some reason the MySQL driver will always load all
> of the results in advance unless the fetchSize is set to
> Integer.MIN_VALUE.... wow.
>
> Try setting fetchSize="-2147483648"
>
> Silly, but wow...
>
> Clinton
>
> On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com> wrote:
> >
> >
> >
> >
> > Hi!
> >
> > Clinton, answering to your last mail:
> > - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currently
> using) with MySQL 5.0.45 (Debian). Tried also on Windows with the same
> drivers, but can not remember the MySQL version.
> > - Bellow are, I think I don't miss any, the different relevant
> configurations (some through Spring):
> >
> > - jdbc.properties: Database connection config
> >
> > jdbc.driverClassName=com.mysql.jdbc.Driver
> >
> jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=true&amp
> ;useUnicode=true&amp;characterEncoding=utf-8
> > jdbc.username=x
> > jdbc.password=y
> >
> >
> > - sql-map-config.xml: iBatis config file
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map Config
> 2.0//EN"
> >    "http://ibatis.apache.org/dtd/sql-map-config-2.dtd">
> >
> > <sqlMapConfig>
> >    <settings enhancementEnabled="true" maxTransactions="20"
> maxRequests="32" maxSessions="10"/>
> >
> >    <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/>
> >
> > </sqlMapConfig>
> >
> >
> > - TestingPojoSQL.xml: The operations needed for indexing:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "
> http://ibatis.apache.org/dtd/sql-map-2.dtd">
> >
> > <sqlMap namespace="TestingPojoSQL">
> >    <typeAlias alias="article" type="org.myorg.model.TestingPojo"/>
> >
> >    <resultMap id="articleResult" class="article">
> >        <result property="id" column="id"/>
> >        <result property="contents" column="contents"/>
> >    </resultMap>
> >
> >    <select id="getAllTestingPojos" resultMap="articleResult"
> fetchSize="100" resultSetType="FORWARD_ONLY">
> >    <![CDATA[
> >        select * from articles
> >    ]]>
> >    </select>
> >
> > </sqlMap>
> >
> >
> > - applicationContext-resources.xml: Here a DBCP data source is defined
> (I tried also C3P0. Made no difference):
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> >        xmlns:jee="http://www.springframework.org/schema/jee"
> >        xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >            http://www.springframework.org/schema/jee
> http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
> >
> >    <bean id="propertyConfigurer" class="
> org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >        <property name="locations">
> >            <list>
> >                <value>classpath:jdbc.properties</value>
> >            </list>
> >        </property>
> >    </bean>
> >
> >    <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> >        <property name="driverClassName" value="${jdbc.driverClassName
> }"/>
> >        <property name="url" value="${jdbc.url}"/>
> >        <property name="username" value="${jdbc.username}"/>
> >        <property name="password" value="${jdbc.password}"/>
> >        <property name="maxActive" value="100"/>
> >        <property name="maxWait" value="1000"/>
> >        <property name="poolPreparedStatements" value="true"/>
> >        <property name="defaultAutoCommit" value="true"/>
> >    </bean>
> >
> > </beans>
> >
> >
> > - In applicationContext-dao.xml: the sqlMapClient
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> >        xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"
> >        default-lazy-init="true">
> >
> >    <bean id="transactionManager" class="
> org.springframework.jdbc.datasource.DataSourceTransactionManager">
> >        <property name="dataSource" ref="dataSource"/>
> >    </bean>
> >
> >    <bean id="sqlMapClient" class="
> org.springframework.orm.ibatis.SqlMapClientFactoryBean">
> >        <property name="configLocation">
> >            <value>classpath:/sql-map-config.xml</value>
> >        </property>
> >        <property name="dataSource" ref="dataSource"/>
> >    </bean>
> > </beans>
> >
> >
> > - In applicationContext-service.xml: This is where I tell Spring to
> instanciate an iBatis based GPS device.
> >
> > <bean id="compass" class="org.compass.spring.LocalCompassBean">
> >    <property name="classMappings">
> >        <list>
> >            <value>org.myorg.TestingPojo</value>
> >        </list>
> >    </property>
> >    <property name="compassSettings">
> >        <props>
> >            <prop key="compass.engine.connection">file://compass</prop>
> >            <prop key="compass.transaction.factory">
> org.compass.spring.transaction.SpringSyncT ransactionFactory</prop>
> >            <prop key="compass.engine.analyzer.default.type
> ">snowball</prop>
> >            <prop key="compass.engine.analyzer.default.name
> ">English</prop>
> >            <!-- Mainly default values -->
> >            <prop key="compass.engine.ramBufferSize">16</prop>
> >            <prop key="compass.engine.maxBufferedDocs">10</prop>
> >            <prop key="compass.engine.maxBufferedDeletedTerms">-1</prop>
> >            <prop key="compass.engine.mergeFactor">10</prop>
> >            <prop key="compass.engine.maxFieldLength">10000</prop>
> >            <prop key="compass.engine.cacheIntervalInvalidation
> ">5000</prop>
> >            <prop key="compass.engine.indexManagerScheduleInterval
> ">60</prop>
> >        </props>
> >    </property>
> >    <property name="transactionManager" ref="transactionManager" />
> > </bean>
> >
> > <bean id="iBatisGpsDevice" class="
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
> >    <property name="name" value="iBatisDevice" />
> >    <property name="sqlMapClient" ref="sqlMapClient" />
> >    <property name="selectStatementsIds">
> >        <list>
> >            <value>getAllTestingPojos</value>
> >        </list>
> >    </property>
> >    <property name="pageSize" value="50" />
> > </bean>
> >
> > <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps"
> init-method="start" destroy-method="stop">
> >    <property name="compass">
> >        <ref bean="compass" />
> >    </property>
> >    <property name="gpsDevices">
> >        <list>
> >            <bean class="
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
> >                <property name="gpsDevice" ref="iBatisGpsDevice" />
> >            </bean>
> >        </list>
> >    </property>
> > </bean>
> >
> >
> > Regarding your previous mail:
> > - I emptied the handleRow method and ran the test - same result. The
> exception occurs before handleRow is invoked.
> > - I'm not configuring any caches. Is there a default one? Should I
> config one?
> > - Setting fetchSize, different resultSetTypes, use of <select> or
> <statement> elements didn't make any differences.
> > - Additionally, I tried truncating the contents of each Wikipedia
> article to 500 characters, and I was able to index all the 650000 rows, so
> the number of records being selected is not the only factor here (I just
> wanted to make sure).
> > - I can imagine others (including you) have successfully made it with
> large amounts of data (I'm new to iBatis and Compass/Lucene) so I'm sure
> there's a solution.
> >
> > Lots of thanks.
> >
> >
> >
> >
> >
> >
> > ----- Original Message ----
> > From: Clinton Begin <clinton.begin@gmail.com>
> > To: user-java@ibatis.apache.org
> >
> >
> >
> > Sent: Monday, March 17, 2008 11:18:59 PM
> > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> > Wow, I just read that other thread on the Compass site.... This is not
> directly an iBATIS problem at all...
> >
> > at com.mysql.jdbc.PreparedStatement.executeInternal(
> PreparedStatement.java:1268) at com.mysql.jdbc.PreparedStatement.execute(
> PreparedStatement.java:843)
> >
> > This line tells me that the execution is never returning from the
> prepared statement.  iBATIS never has a chance to even see the ResultSet,
> let alone get to the RowHandler. So swapping out the GpsDevice here won't
> make any difference at all. This is odd behavior for a JDBC driver and tells
> me that something is very wrong with the configuration.
> >
> > Which version of the JDBC driver are you using?  What are the rest of
> the iBATIS convfiguration details?
> >
> > Clinton
> >
> >
> >
> > On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <clinton.begin@gmail.com>
> wrote:
> >
> > > Thanks for the summary.  A few things:
> > >
> > > * PaginatedList was definitely the wrong solution for that, so I'm
> glad it's gone.
> > >
> > > * A RowHandler is possibly the right solution, but will only store as
> many records in memory as you tell it to.  iBATIS does not just arbitrarily
> keep records around.  Quite the opposite actually.  Unless you keep a
> reference or configure a cache, iBATIS will not keep the object around at
> all.
> > >
> > > Looking at your implementation, there are a couple of things going
> on...
> > >
> > > session.create(o);
> > > if(currentItem== pageSize){
> > >  if (log.isDebugEnabled()) {
> > >    log.debug("Indexing page number ["+ pageCount++ + "]");
> > >  }
> > > session.evictAll();
> > >
> > > Things to Try:
> > >
> > > 1)  These lines of code must be treated as guilty until proven
> innocent (especially the logging).  The easiest way to do that is to delete
> them.  Create an empty handler/GpsDevice and run the test to see what
> happens.
> > >
> > > 2) Look at the iBATIS configuration files.  Is a cache configured for
> this query?  What kind of cache?
> > >
> > > 3) You could try multiple queries of fixed sizes with
> queryForList(String,Object,int offset,int limit) ... but I can't see how
> that would be better than a RowHandler.
> > >
> > > 4) Next, just in case the driver's default fetch size is ridiculously
> high (or unlimited), try setting <select ... fetchSize="100"/>.  You could
> also try a scrollable SesultSet with <statement ... resultSetType /> of
> SCROLL_INSENSITIVE or SCROLL_SENSITIVE.
> > >
> > > Finally, please know that I've used iBATIS to load tens of millions of
> rows for ETL and even analysis (NetFlix Prize) and was able to do so with a
> RowHandler or even simply multiple calls to queryForList(String,Object,int
> offset,int limit).  I don't recall having to even think about the
> configuration, although I did manage to optimize it to achieve a read/insert
> performance of 15,000 records per second between two databases (on a single
> machine too).
> > >
> > > I'm sure there's something about the configuration here that is
> causing the problem, and it can be solved.
> > >
> > > I hope one of these helps.
> > >
> > > Clinton
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com> wrote:
> > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi, there. :-)
> > > >
> > > > The thing is, I'm testing a search engine called Compass. One of the
> tests consists of indexing a Wikipedia dump. In this case only the text of
> the articles, so I downloaded it and imported it into a MySQL database
> (about 650000 records - 1.5GB).
> > > >
> > > > You can configure Compass to access a database and index it's
> contents by configuring what they call a GPS Device. You can do that through
> an ORM such as iBatis, Hibernate or JPA.
> > > >
> > > > Compass provides an implementation of such a GPS Device called
> SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the
> results of a query and, so, index them. The query is just a "select * from
> articles_table".
> > > >
> > > > So I wired everything up and ran several tests with different
> amounts of data to be indexed and JVM stack space sizes. The result was an
> "OutOfMemoryError: java heap space" error message whenever the size of the
> selected rows was bigger than the amount of available heap space (I can not
> index the 650000 records having 1GB of stack).
> > > >
> > > > After doing some profiling I concluded iBatis was loading the whole
> amount of results into memory. So I decided to try Hibernate, which worked
> just fine.
> > > >
> > > > Finally I found queryForPaginatedList was deprecated and that I
> should use queryWithRowHandler instead, so I made my own implementation of
> the SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I
> had the same result (I profiled the application stack usage and I could see
> a line growing steadily and finally crash against the stack ceiling while
> performing the select statement).
> > > >
> > > > Shay Banon, Compass project leader, is planning to patch
> SqlMapClientGpsDevice by replacing queryForPaginatedList with
> queryWithRowHandler, but I'm doubtful it's gonna work.
> > > >
> > > > Finally you can, of course, devide the select stament into several
> selects (and it works) but, in my opinion, that shouldn't be necessary.
> > > > You can read more on
> http://forum.compass-project.org/thread.jspa?threadID=215278
> > > >
> > > > Thanks!
> > > >
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > > From: Clinton Begin <clinton.begin@gmail.com>
> > > > To: user-java@ibatis.apache.org
> > > >
> > > >
> > > >
> > > > Sent: Monday, March 17, 2008 2:22:39 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > > Wow, this thread is interesting.  I suppose I could read the past
> emails, but could someone summarize the problem and progress to date?  I may
> be able to shed some light on what is happening.
> > > >
> > > > Clinton
> > > >
> > > >
> > > > On Sun, Mar 16, 2008 at 11:49 PM, nch <underscore_dot@yahoo.com>
> wrote:
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Another thing I forgot. This works perfectly well if we replace
> iBatis by Hibernate, so I don't think this is being caused by the MySQL
> driver.
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: nch <underscore_dot@yahoo.com>
> > > > > To: user-java@ibatis.apache.org
> > > > >
> > > > >
> > > > >
> > > > > Sent: Sunday, March 16, 2008 10:18:42 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Well, I modified the mediumblob into a mediumtext and removed all
> other fields except the id one.
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: nch <underscore_dot@yahoo.com>
> > > > > To: user-java@ibatis.apache.org
> > > > > Sent: Sunday, March 16, 2008 10:06:34 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > You're probably right. I'm using the table called "text" which
> contains a field of type "mediumblob".
> > > > > CREATE TABLE /*$wgDBprefix*/text (
> > > > > old_id int unsigned NOT NULL auto_increment,
> > > > > old_text mediumblob NOT NULL,
> > > > > old_flags tinyblob NOT NULL,
> > > > > PRIMARY KEY old_id (old_id)
> > > > > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > You can find it's definition here:
> > > > >
> http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
> > > > >
> > > > > I'll try to modify that field, first, and see what happens.
> > > > > I've already tried "select * from your_wikipedia_dump limit
> $start$, $size$" and it seems to work fine, but I see a possible issue with
> this, you can not remove any of the records you've already read during the
> whole process. This might not be an option.
> > > > >
> > > > > Thank you.
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > To: user-java@ibatis.apache.org
> > > > > Sent: Sunday, March 16, 2008 4:19:43 PM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > > Yeah, I just noticed that - it looks like it's failing in the jdbc
> > > > > driver when it tries to read a blob, maybe?
> > > > >
> > > > > I know that some versions of the postgresql jdbc driver load the
> > > > > entire contents of a resultset into a massive byte[][] structure.
> I
> > > > > wonder if that is what is happening here? I think you can get the
> > > > > source for the MySQL driver and check that if you want to.
> > > > >
> > > > > If it is, you may need to structure your app to take smaller bites
> of
> > > > > this elephant.
> > > > >
> > > > > Try "select * from your_wikipedia_dump limit $start$, $size$"
> instead
> > > > > - that will limit the size of the results to $size$ rows, starting
> on
> > > > > row $start$ (the $start$ value is zero-based). If you experiment
> to
> > > > > see how big $size$ can get, you should be able to get pretty
> decent
> > > > > performance out of this.
> > > > >
> > > > > Psuedo-code would be like this...
> > > > >
> > > > > int start = 0;
> > > > > int size = 1000; // see how big you can make this to improve
> performance
> > > > > while(still_more_data){
> > > > >  still_more_data = index(start, size);
> > > > >  start += size;
> > > > > }
> > > > >
> > > > > Your index method will return true if the query returned any data.
> If
> > > > > the query returns no data, it'll return false.
> > > > >
> > > > > I can't imagine what Hibernate is doing differently here to make
> this
> > > > > work if the JDBC driver is failing when executing the query.
> > > > >
> > > > > Larry
> > > > >
> > > > >
> > > > > On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com>
> wrote:
> > > > > >
> > > > > >
> > > > > > Sure. Please, see attached.
> > > > > > I don't think the problem is in the RowHandler, though, because
> the
> > > > > > OutOfMemoryError occurs before invoking RowHandle#handleRow.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > > ----- Original Message ----
> > > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > > To: user-java@ibatis.apache.org
> > > > > >
> > > > > > Sent: Sunday, March 16, 2008 3:13:27 AM
> > > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap
> space"
> > > > > >
> > > > > >  Can you post the row handler you are using.
> > > > > >
> > > > > > Larry
> > > > > >
> > > > > >
> > > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com>
> wrote:
> > > > > > >
> > > > > > >
> > > > > > > Hi, Nathan.
> > > > > > > I did so, but I'm still having the same issue. Perhaps
I'm not
> using it
> > > > > > > correctly?
> > > > > > > See my last post to this forum entry:
> > > > > > >
> > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > >
> > > > > > > See the stack trace:
> > > > > > >
> > > > > > > MemoryError: Java heap space:
> > > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > > >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > > > > > >        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java
> :318)
> > > > > > >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> > > > > > >        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java
> :2333)
> > > > > > >        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java
> :435)
> > > > > > >        at
> > > > > > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(
> MysqlIO.java:2040)
> > > > > > >        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java
> :1443)
> > > > > > >        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java
> :1777)
> > > > > > >        at com.mysql.jdbc.Connection.execSQL(Connection.java
> :3249)
> > > > > > >        at
> > > > > > >
> > > > > > com.mysql.jdbc.PreparedStatement.executeInternal(
> PreparedStatement.java:1268)
> > > > > > >        at
> > > > > > > com.mysql.jdbc.PreparedStatement.execute(
> PreparedStatement.java:843)
> > > > > > >        at
> > > > > > >
> > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> DelegatingPreparedStatement.java:169)
> > > > > > >        at
> > > > > > >
> > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> DelegatingPreparedStatement.java:169)
> > > > > > >        at
> > > > > > >
> > > > > > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(
> SqlExecutor.java:186)
> > > > > > >        at
> > > > > > >
> > > > > >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery
> (GeneralStatement.java:205)
> > > > > > >        at
> > > > > > >
> > > > > >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback
> (GeneralStatement.java:173)
> > > > > > >        at
> > > > > > >
> > > > > >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler
> (GeneralStatement.java:133)
> > > > > > >        at
> > > > > > >
> > > > > >
> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(
> SqlMapExecutorDelegate.java:649)
> > > > > > >        at
> > > > > > >
> > > > > >
> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(
> SqlMapSessionImpl.java:156)
> > > > > > >        at
> > > > > > >
> > > > > > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(
> SqlMapClientGpsDeviceWithRowHandler.java:33)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(
> AbstractGpsDevice.java:93)
> > > > > > >        at
> > > > > > >
> > > > > > org.compass.core.CompassCallbackWithoutResult.doInCompass(
> CompassCallbackWithoutResult.java:29)
> > > > > > >        at
> > > > > > > org.compass.core.CompassTemplate.execute(CompassTemplate.java
> :132)
> > > > > > >        at
> > > > > > >
> > > > > > org.compass.gps.impl.SingleCompassGps.executeForIndex(
> SingleCompassGps.java:161)
> > > > > > >        at
> > > > > > > org.compass.gps.device.AbstractGpsDevice.index(
> AbstractGpsDevice.java:91)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult
> (SpringSyncTransactionGpsDeviceWrapper.java:98)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction
> (TransactionCallbackWithoutResult.java:33)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.springframework.transaction.support.TransactionTemplate.execute(
> TransactionTemplate.java:128)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(
> SpringSyncTransactionGpsDeviceWrapper.java:96)
> > > > > > >        at
> > > > > > >
> > > > > > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(
> SingleCompassGps.java:133)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep
> (DefaultLuceneSearchEngineIndexManager.java:233)
> > > > > > >        at
> > > > > > >
> > > > > >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate
> (DefaultLuceneSearchEngineIndexManager.java:182)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ----- Original Message ----
> > > > > > > From: Nathan Maves <nathan.maves@gmail.com>
> > > > > > > To: user-java@ibatis.apache.org
> > > > > > >
> > > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM
> > > > > > > Subject: Re: select * causing " OutOfMemoryError: Java
heap
> space"
> > > > > > >
> > > > > > >  this is only my 2 cents but I would throw that class out.
> Write your own
> > > > > > > implementation which uses a row handler.  This is the type
of
> situation in
> > > > > > > which a row handler could really help.  almost no memory
would
> be used.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com>
> wrote:
> > > > > > > >
> > > > > > > > Yes, I agree that dividing the query into several
queries
> should do as
> > > > > > > long as none of them uses more memory than available. Gracias,
> Carlos.
> > > > > > > >
> > > > > > > > Nevertheless, I looked into
> > > > > > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndexand,
> > > > > > > fundamentally, what it does is a queryForPaginatedList
and
> iterates
> > > > > > through
> > > > > > > the resulting PaginatedList with nextPage()
> > > > > > > >
> > > > > > > > I guess that the reason why pages are kept in memory
after
> performing
> > > > > > > paginatedList.nextPage() is because a
> paginatedList.previousPage() can be
> > > > > > > issued but, should't it free pages if heap space is running
> out?
> > > > > > > >
> > > > > > > >
> > > > > > > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>
wrote:
> > > > > > > >
> > > > > > > > Since we have a similar trouble around here i must
say that
> you have lot
> > > > > > > of options when going to compass..
> > > > > > > > The first one is to index "object by object" by hand,
the
> second one is
> > > > > > to
> > > > > > > use the Gps wich will use the complete bunch of data and
make
> subindexes
> > > > > > > (making parts of your "main index" with diferent "maps"
for
> each subindex
> > > > > > in
> > > > > > > your gpsDriver.
> > > > > > > > if your indexing is as strong as ours i should look
forward
> for numer 1
> > > > > > or
> > > > > > > number 3 depending of your needs. (More a Compass than
iBatis
> issue)
> > > > > > > > Greetings
> > > > > > > > Carlos de Luna
> > > > > > > >
> > > > > > > > ----- Mensaje original ----
> > > > > > > > De: nch
> > > > > > > >
> > > > > > > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > > > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > > > > > > Asunto: Re: select * causing " OutOfMemoryError: Java
heap
> space"
> > > > > > > >
> > > > > > > >
> > > > > > > > Sorry. Yes, I need all the data at one time, because
the
> Compass API
> > > > > > seems
> > > > > > > to be meant in that way.
> > > > > > > >
> > > > > > > > Larry Meadors wrote: OK, so the answer to the question
> is...[ yes, i
> > > > > > need
> > > > > > > all the data at
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > one time | no, i do not need all of the data at one
time ].
> > > > > > > >
> > > > > > > > Larry
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > > > > > > >
> > > > > > > > > Hi!
> > > > > > > > >
> > > > > > > > > I'm doing tests both under tomcat and jetty.
I set 1024MB
> of heap
> > > > > > space
> > > > > > > in
> > > > > > > > > both cases. I am using a profiler to look into
what's
> happening during
> > > > > > > > > execution of the indexing process and I can tell
that's
> the exact
> > > > > > amount
> > > > > > > it
> > > > > > > > > displays for the VM.
> > > > > > > > >
> > > > > > > > > As I describe in the Compass user forum (see
link bellow),
> I'm
> > > > > > indexing
> > > > > > > a
> > > > > > > > > Wikipedia dump of about 650000-675000 records.
That's sort
> of 1-1.5GB
> > > > > > of
> > > > > > > > > data.
> > > > > > > > >
> > > > > > > > > Hibernate can handle all that data, I guess it's
using
> some sort of
> > > > > > > > > pagination. iBatis seems to divide the select
statement
> into several
> > > > > > > select
> > > > > > > > > statements, but it also seems each of such statements
> surpasses the
> > > > > > > limit of
> > > > > > > > > 1024MB when trying to index only 300000 of the
database
> records.
> > > > > > > > >
> > > > > > > > > Thank you
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chris Lamey wrote:
> > > > > > > > > What is you JVM heap size set to and how much
data is
> returned by
> > > > > > select
> > > > > > > *
> > > > > > > > > from table?
> > > > > > > > >
> > > > > > > > > If you're trying to pull back 1G worth of data
into a JVM
> with a heap
> > > > > > > size
> > > > > > > > > set to 64M, you will hit the heap limit pretty
quick.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > > > > > > Sent: Thu 3/13/2008 8:03 AM
> > > > > > > > > To: user-java@ibatis.apache.org
> > > > > > > > > Subject: select * causing " OutOfMemoryError:
Java heap
> space"
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi, everybody.
> > > > > > > > > I'm testing the Compass search engine and, in
particular,
> how to index
> > > > > > a
> > > > > > > big
> > > > > > > > > set of documents from a table in a MySQL database.
In
> order to do this
> > > > > > I
> > > > > > > > > issue a "select * from table_name" using iBatis,
but this
> seems to be
> > > > > > > > > causing the application to use all heap space
available.
> > > > > > > > > I added a new entry into Compass user forum with
details:
> > > > > > > > >
> http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > > > >
> > > > > > > > > Can you figure out why is this happening?
> > > > > > > > >
> > > > > > > > > Many thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ---------------------------------
> > > > > > > > > Be a better friend, newshound, and know-it-all
with Yahoo!
> Mobile. Try
> > > > > > > it
> > > > > > > > > now.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ________________________________
> > > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > > >
> > > > > > > >
> > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> ____________________________________________________________________________________
> > > > > > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > > > > > No te preocupes más por el espacio de tu cuenta con
Correo
> Yahoo!:
> > > > > > > > http://correo.espanol.yahoo.com/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >  ________________________________
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > >
> > > > > >
> > > > > >  ________________________________
> > > > > >
> > > > > > Be a better friend, newshound, and know-it-all with Yahoo!
> Mobile. Try it
> > > > > > now.
> > > > >
> > > > >
> > > > > ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
> > > > >
> > > > > ________________________________
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> >
> >
> >
> > ________________________________
>
> > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it now.
>
>
> ------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
>

Mime
View raw message