ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nch <underscore_...@yahoo.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 18:10:26 GMT

Hello, Leon. I tried this with no success.
By the way, I'm using JDK 1.5.0_13.

----- Original Message ----
From: Leon Liu <ljw2083@alibaba-inc.com>
To: user-java@ibatis.apache.org
Sent: Tuesday, March 18, 2008 6:02:02 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"

  pls set enhancementEnabled=false  and test again.
Some cglib + some jdk version cause a big memory problem.   I met it inmy project that take
me a month to fix it.

Leon Liu


Nathan Maves 写道:did you try defaultAutoCommit=false for the connectionpool?
  
  On Tue, Mar 18, 2008 at 9:13 AM, nch <underscore_dot@yahoo.com>wrote:
              
Hi!
    
Clinton, answering to your last mail:
- I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currentlyusing) with MySQL 5.0.45
(Debian). Tried also on Windows with the samedrivers, but can not remember the MySQL version.
- Bellow are, I think I don't miss any, the different relevantconfigurations (some through
Spring):
    
- jdbc.properties: Database connection config
    
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8
jdbc.username=x
jdbc.password=y
    
    
- sql-map-config.xml: iBatis config file
    
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL MapConfig 2.0//EN"
        "http://ibatis.apache.org/dtd/sql-map-config-2.dtd">
    
<sqlMapConfig>
    <settings enhancementEnabled="true" maxTransactions="20"maxRequests="32" maxSessions="10"/>
    
    <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/>
    
</sqlMapConfig>
    
    
- TestingPojoSQL.xml: The operations needed for indexing:
    
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" "http://ibatis.apache.org/dtd/sql-map-2.dtd">
    
<sqlMap namespace="TestingPojoSQL">
    <typeAlias alias="article"type="org.myorg.model.TestingPojo"/>
    
    <resultMap id="articleResult" class="article">
        <result property="id" column="id"/>
        <result property="contents" column="contents"/>
    </resultMap>
    
    <select id="getAllTestingPojos" resultMap="articleResult"fetchSize="100" resultSetType="FORWARD_ONLY">
    <![CDATA[
        select * from articles
    ]]>
    </select>
    
</sqlMap>
    
    
- applicationContext-resources.xml: Here a DBCP data source is defined(I tried also C3P0.
Made no difference):
    
<?xml version="1.0" encoding="UTF-8"?>
    
           xmlns:jee="http://www.springframework.org/schema/jee"
           xsi:schemaLocation="http://www.springframework.org/schema/beans    http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
                http://www.springframework.org/schema/jee    http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
    
    <bean id="propertyConfigurer"class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <list>
                <value>classpath:jdbc.properties</value>
            </list>
        </property>
    </bean>
    
    <bean id="dataSource"class="org.apache.commons.dbcp.BasicDataSource"destroy-method="close">
        <property name="driverClassName"value="${jdbc.driverClassName}"/>
        <property name="url" value="${jdbc.url}"/>
        <property name="username" value="${jdbc.username}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="maxActive" value="100"/>
        <property name="maxWait" value="1000"/>
        <property name="poolPreparedStatements" value="true"/>
        <property name="defaultAutoCommit" value="true"/>
    </bean>
    
</beans>
    
    
- In applicationContext-dao.xml: the sqlMapClient
    
<?xml version="1.0" encoding="UTF-8"?>
    
           xsi:schemaLocation="http://www.springframework.org/schema/beans    http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"
       default-lazy-init="true">
    
    <bean id="transactionManager"class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
        <property name="dataSource" ref="dataSource"/>
    </bean>
    
    <bean id="sqlMapClient"class="org.springframework.orm.ibatis.SqlMapClientFactoryBean">
        <property name="configLocation">
            <value>classpath:/sql-map-config.xml</value>
        </property>
        <property name="dataSource" ref="dataSource"/>
    </bean>
</beans>
    
    
- In applicationContext-service.xml: This is where I tell Spring toinstanciate an iBatis based
GPS device. 
    
<bean id="compass" class="org.compass.spring.LocalCompassBean">
    <property name="classMappings">
        <list>
            <value>org.myorg.TestingPojo</value>
        </list>
    </property>
    <property name="compassSettings">
        <props>
            <propkey="compass.engine.connection">file://compass</prop>
            <propkey="compass.transaction.factory">org.compass.spring.transaction.SpringSyncTransactionFactory</prop>
            <propkey="compass.engine.analyzer.default.type">snowball</prop>
            <prop key="compass.engine.analyzer.default.name">English</prop>
            <!-- Mainly default values -->
            <propkey="compass.engine.ramBufferSize">16</prop>
            <propkey="compass.engine.maxBufferedDocs">10</prop>
            <propkey="compass.engine.maxBufferedDeletedTerms">-1</prop>
            <prop key="compass.engine.mergeFactor">10</prop>
            <propkey="compass.engine.maxFieldLength">10000</prop>
            <propkey="compass.engine.cacheIntervalInvalidation">5000</prop>
            <propkey="compass.engine.indexManagerScheduleInterval">60</prop>
        </props>
    </property>
    <property name="transactionManager" ref="transactionManager"/>
</bean>
    
<bean id="iBatisGpsDevice"class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
    <property name="name" value="iBatisDevice" />
    <property name="sqlMapClient" ref="sqlMapClient" />
    <property name="selectStatementsIds">
        <list>
            <value>getAllTestingPojos</value>
        </list>
    </property>
    <property name="pageSize" value="50" />
</bean>
    
<bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps"init-method="start"
destroy-method="stop">
    <property name="compass">
        <ref bean="compass" />
    </property>
    <property name="gpsDevices">
        <list>
            <beanclass="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
                <property name="gpsDevice" ref="iBatisGpsDevice"/>
            </bean>
        </list>
    </property>
</bean>
    
    
Regarding your previous mail:
- I emptied the handleRow method and ran the test - same result. Theexception occurs before
handleRow is invoked.
- I'm not configuring any caches. Is there a default one? Should Iconfig one?
- Setting fetchSize, different resultSetTypes, use of <select> or<statement> elements
didn't make any differences.
- Additionally, I tried truncating the contents of each Wikipediaarticle to 500 characters,
and I was able to index all the 650000 rows,so the number of records being selected is not
the only factor here (Ijust wanted to make sure).
- I can imagine others (including you) have successfully made it withlarge amounts of data
(I'm new to iBatis and Compass/Lucene) so I'msure there's a solution.
    
Lots of thanks.
    
    
    
    
        ----- Original Message ----
From: Clinton Begin <clinton.begin@gmail.com>
To: user-java@ibatis.apache.org
    
        Sent: Monday, March 17, 2008 11:18:59 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
    
Wow, I just read that other thread on the Compass site.... This is notdirectly an iBATIS problem
at all...
    
atcom.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
    
This line tells me that the execution is never returning from theprepared statement.  iBATIS
never has a chance to even see theResultSet, let alone get to the RowHandler. So swapping
out theGpsDevice here won't make any difference at all. This is odd behaviorfor a JDBC driver
and tells me that something is very wrong with theconfiguration.  
    
Which version of the JDBC driver are you using?  What are the rest ofthe iBATIS convfiguration
details?
    
Clinton
    
    
    On Mon, Mar 17, 2008 at 4:12 PM, ClintonBegin <clinton.begin@gmail.com>wrote:
    Thanksfor the summary.  A few things:
      
* PaginatedList was definitely the wrong solution for that, so I'm gladit's gone.
      
* A RowHandler is possibly the right solution, but will only store asmany records in memory
as you tell it to.  iBATIS does not justarbitrarily keep records around.  Quite the opposite
actually.  Unlessyou keep a reference or configure a cache, iBATIS will not keep theobject
around at all.  
      
Looking at your implementation, there are a couple of things going on...
      
session.create(o);
if(currentItem== pageSize){
  if (log.isDebugEnabled()) {
    log.debug("Indexing page number ["+ pageCount++ + "]");
  }
session.evictAll();
      
Things to Try:
      
1)  These lines of code must be treated as guilty until proven innocent(especially the logging).
 The easiest way to do that is to deletethem.  Create an empty handler/GpsDevice and run the
test to see whathappens.
      
2) Look at the iBATIS configuration files.  Is a cache configured forthis query?  What kind
of cache?
      
3) You could try multiple queries of fixed sizes withqueryForList(String,Object,int offset,int
limit) ... but I can't seehow that would be better than a RowHandler.
      
4) Next, just in case the driver's default fetch size is ridiculouslyhigh (or unlimited),
try setting <select ... fetchSize="100"/>. You could also try a scrollable SesultSet
with <statement ...resultSetType /> of  SCROLL_INSENSITIVE or SCROLL_SENSITIVE. 
      
Finally, please know that I've used iBATIS to load tens of millions ofrows for ETL and even
analysis (NetFlix Prize) and was able to do sowith a RowHandler or even simply multiple calls
toqueryForList(String,Object,int offset,int limit).  I don't recallhaving to even think about
the configuration, although I did manage tooptimize it to achieve a read/insert performance
of 15,000 records persecond between two databases (on a single machine too).  
      
I'm sure there's something about the configuration here that is causingthe problem, and it
can be solved.
      
I hope one of these helps.
      
Clinton            
      
      On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com>wrote:
                              
Hi, there. :-)
        
The thing is, I'm testing a search engine called Compass. One of thetests consists of indexing
a Wikipedia dump. In this case only the textof the articles, so I downloaded it and imported
it into a MySQLdatabase (about 650000 records - 1.5GB).
        
You can configure Compass to access a database and index it's contentsby configuring what
they call a GPS Device. You can do that through anORM such as iBatis, Hibernate or JPA.
        
Compass provides an implementation of such a GPS Device calledSqlMapClientGpsDevice which
uses iBatis queryForPaginatedList to getthe results of a query and, so, index them. The query
is just a "select* from articles_table".
        
So I wired everything up and ran several tests with different amountsof data to be indexed
and JVM stack space sizes. The result was an"OutOfMemoryError: java heap space" error message
whenever the size ofthe selected rows was bigger than the amount of available heap space (Ican
not index the 650000 records having 1GB of stack).
        
After doing some profiling I concluded iBatis was loading the wholeamount of results into
memory. So I decided to try Hibernate, whichworked just fine.
        
Finally I found queryForPaginatedList was deprecated and that I shoulduse queryWithRowHandler
instead, so I made my own implementation of theSqlMapClientGpsDevice based on queryWithRowHandler
and tested it, but Ihad the same result (I profiled the application stack usage and I couldsee
a line growing steadily and finally crash against the stack ceilingwhile performing the select
statement).
        
Shay Banon, Compass project leader, is planning to patchSqlMapClientGpsDevice by replacing
queryForPaginatedList withqueryWithRowHandler, but I'm doubtful it's gonna work.
        
Finally you can, of course, devide the select stament into severalselects (and it works) but,
in my opinion, that shouldn't be necessary.
        You can read more on http://forum.compass-project.org/thread.jspa?threadID=215278
        
Thanks!
        
        
                ----- Original Message ----
From: Clinton Begin <clinton.begin@gmail.com>
To: user-java@ibatis.apache.org
        
                Sent: Monday, March 17, 2008 2:22:39 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
        
Wow, this thread is interesting.   I suppose I could read the pastemails, but could someone
summarize the problem and progress to date? I may be able to shed some light on what is happening.
        
Clinton
        
        On Sun, Mar 16, 2008 at 11:49 PM, nch<underscore_dot@yahoo.com>wrote:
                                      
Another thing I forgot. This works perfectly well if we replace iBatisby Hibernate, so I don't
think this is being caused by the MySQL driver.
          
                    ----- Original Message ----
From: nch <underscore_dot@yahoo.com>
To: user-java@ibatis.apache.org
          
                    Sent: Sunday, March 16, 2008 10:18:42 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
          
                    
Well, I modified the mediumblob into a mediumtext and removed all otherfields except the id
one.
          
          -----Original Message ----
From: nch <underscore_dot@yahoo.com>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 10:06:34 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
          
                    
You're probably right. I'm using the table called "text" which containsa field of type "mediumblob".
          CREATE TABLE /*$wgDBprefix*/text (
  old_id int unsigned NOT NULL auto_increment,
  old_text mediumblob NOT NULL,
  old_flags tinyblob NOT NULL,
  PRIMARY KEY old_id (old_id)
) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;




          You can find it's definition here:
          http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
          
I'll try to modify that field, first, and see what happens.
I've already tried "select * from your_wikipedia_dump limit $start$,$size$" and it seems to
work fine, but I see a possible issue withthis, you can not remove any of the records you've
already read duringthe whole process. This might not be an option.
          
Thank you.
          
          -----Original Message ----
From: Larry Meadors <larry.meadors@gmail.com>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 4:19:43 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
          
Yeah, I just noticed that - it looks like it's failing in the jdbc
driver when it tries to read a blob, maybe?
          
I know that some versions of the postgresql jdbc driver load the
entire contents of a resultset into a massive byte[][] structure. I
wonder if that is what is happening here? I think you can get the
source for the MySQL driver and check that if you want to.
          
If it is, you may need to structure your app to take smaller bites of
this elephant.
          
Try "select * from your_wikipedia_dump limit $start$, $size$" instead
- that will limit the size of the results to $size$ rows, starting on
row $start$ (the $start$ value is zero-based). If you experiment to
see how big $size$ can get, you should be able to get pretty decent
performance out of this.
          
Psuedo-code would be like this...
          
int start = 0;
int size = 1000; // see how big you can make this to improve performance
while(still_more_data){
  still_more_data = index(start, size);
  start += size;
}
          
Your index method will return true if the query returned any data. If
the query returns no data, it'll return false.
          
I can't imagine what Hibernate is doing differently here to make this
work if the JDBC driver is failing when executing the query.
          
Larry
          
          
On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com>wrote:
>
>
> Sure. Please, see attached.
> I don't think the problem is in the RowHandler, though, because the
> OutOfMemoryError occurs before invoking RowHandle#handleRow.
>
> Cheers
>
>
> ----- Original Message ----
> From: Larry Meadors <larry.meadors@gmail.com>
> To: user-java@ibatis.apache.org
>
> Sent: Sunday, March 16, 2008 3:13:27 AM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
>  Can you post the row handler you are using.
>
> Larry
>
>
> On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com>wrote:
> >
> >
> > Hi, Nathan.
> > I did so, but I'm still having the same issue. Perhaps I'mnot using it
> > correctly?
> > See my last post to this forum entry:
> >
> > http://forum.compass-project.org/thread.jspa?threadID=215278
> >
> > See the stack trace:
> >
> > MemoryError: Java heap space:
> > java.lang.OutOfMemoryError: Java heap space
> >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> >        atcom.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> >        atcom.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> >        atcom.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> >        at
> >com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> >        atcom.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> >        atcom.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> >        atcom.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> >        at
> >
>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> >        at
> >com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> >        at
> >
>org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> >        at
> >
>org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> >        at
> >
>com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
> >        at
> >
>com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
> >        at
> >
>com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
> >        at
> >
>com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
> >        at
> >
>com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
> >        at
> >
>com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
> >        at
> >
>org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
> >        at
> >
>org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
> >        at
> >
>org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
> >        at
> >org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> >        at
> >
>org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
> >        at
> >org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
> >        at
> >
>org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
> >        at
> >
>org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
> >        at
> >
>org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
> >        at
> >
>org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
> >        at
> >
>org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
> >        at
> >
>org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
> >        at
> >
>org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
> >
> >
> >
> >
> >
> > ----- Original Message ----
> > From: Nathan Maves <nathan.maves@gmail.com>
> > To: user-java@ibatis.apache.org
> >
> > Sent: Saturday, March 15, 2008 5:43:04 PM
> > Subject: Re: select * causing " OutOfMemoryError: Java heapspace"
> >
> >  this is only my 2 cents but I would throw that class out. Write your own
> > implementation which uses a row handler.  This is the type ofsituation in
> > which a row handler could really help.  almost no memorywould be used.
> >
> >
> >
> > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com>wrote:
> > >
> > > Yes, I agree that dividing the query into severalqueries should do as
> > long as none of them uses more memory than available.Gracias, Carlos.
> > >
> > > Nevertheless, I looked into
> > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndexand,
> > fundamentally, what it does is a queryForPaginatedList anditerates
> through
> > the resulting PaginatedList with nextPage()
> > >
> > > I guess that the reason why pages are kept in memoryafter performing
> > paginatedList.nextPage() is because apaginatedList.previousPage() can be
> > issued but, should't it free pages if heap space is runningout?
> > >
> > >
> > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>wrote:
> > >
> > > Since we have a similar trouble around here i must saythat you have lot
> > of options when going to compass..
> > > The first one is to index "object by object" by hand,the second one is
> to
> > use the Gps wich will use the complete bunch of data and makesubindexes
> > (making parts of your "main index" with diferent "maps" foreach subindex
> in
> > your gpsDriver.
> > > if your indexing is as strong as ours i should lookforward for numer 1
> or
> > number 3 depending of your needs. (More a Compass than iBatisissue)
> > > Greetings
> > > Carlos de Luna
> > >
> > > ----- Mensaje original ----
> > > De: nch
> > >
> > > Para: user-java@ibatis.apache.org;          larry.meadors@gmail.com
> > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > Asunto: Re: select * causing " OutOfMemoryError: Javaheap space"
> > >
> > >
> > > Sorry. Yes, I need all the data at one time, because theCompass API
> seems
> > to be meant in that way.
> > >
> > > Larry Meadors wrote: OK, so the answer to the questionis...[ yes, i
> need
> > all the data at
> > >
> > >
> > >
> > > one time | no, i do not need all of the data at one time].
> > >
> > > Larry
> > >
> > >
> > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > >
> > > > Hi!
> > > >
> > > > I'm doing tests both under tomcat and jetty. I set1024MB of heap
> space
> > in
> > > > both cases. I am using a profiler to look intowhat's happening during
> > > > execution of the indexing process and I can tellthat's the exact
> amount
> > it
> > > > displays for the VM.
> > > >
> > > > As I describe in the Compass user forum (see linkbellow), I'm
> indexing
> > a
> > > > Wikipedia dump of about 650000-675000 records.That's sort of 1-1.5GB
> of
> > > > data.
> > > >
> > > > Hibernate can handle all that data, I guess it'susing some sort of
> > > > pagination. iBatis seems to divide the selectstatement into several
> > select
> > > > statements, but it also seems each of suchstatements surpasses the
> > limit of
> > > > 1024MB when trying to index only 300000 of thedatabase records.
> > > >
> > > > Thank you
> > > >
> > > >
> > > >
> > > > Chris Lamey wrote:
> > > > What is you JVM heap size set to and how much datais returned by
> select
> > *
> > > > from table?
> > > >
> > > > If you're trying to pull back 1G worth of data intoa JVM with a heap
> > size
> > > > set to 64M, you will hit the heap limit prettyquick.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > Sent: Thu 3/13/2008 8:03 AM
> > > > To: user-java@ibatis.apache.org
> > > > Subject: select * causing " OutOfMemoryError: Javaheap space"
> > > >
> > > >
> > > > Hi, everybody.
> > > > I'm testing the Compass search engine and, inparticular, how to index
> a
> > big
> > > > set of documents from a table in a MySQL database.In order to do this
> I
> > > > issue a "select * from table_name" using iBatis,but this seems to be
> > > > causing the application to use all heap spaceavailable.
> > > > I added a new entry into Compass user forum withdetails:
> > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > >
> > > > Can you figure out why is this happening?
> > > >
> > > > Many thanks
> > > >
> > > >
> > > > ---------------------------------
> > > > Be a better friend, newshound, and know-it-all withYahoo! Mobile. Try
> > it
> > > > now.
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>____________________________________________________________________________________
> > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > No te preocupes más por el espacio de tu cuenta conCorreo Yahoo!:
> > > http://correo.espanol.yahoo.com/
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ________________________________
> > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >
> >
> >  ________________________________
> > Never miss a thing. Make Yahoo your homepage.
>
>
>  ________________________________
>
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile.Try it
> now.
          
          
          
          
          
          Looking for last minute shopping deals?  Find them fast with Yahoo! Search.
          
          
          
          
          Be a better friend, newshound, and know-it-allwith Yahoo! Mobile.  Try it now.
          
          
          
          
          
                    
          Looking for last minute shopping deals?  Find them fast with Yahoo! Search.
          
          
                
        
        
        
        
        
        
        
                
        Never miss a thing.  Make Yahoo your homepage.        
        
        
            
      
      
      
        
    
    
    
    
    
    
    
    
        Be a better friend, newshound, and know-it-allwith Yahoo! Mobile.  Try it now.
    
    
  







      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ


Mime
View raw message