cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrus Adamchik <and...@objectstyle.org>
Subject Re: out of memory using ResultIterator
Date Thu, 24 Dec 2009 08:52:45 GMT
This is likely an issue with MySQL driver caching returned data. The  
solution was implemented in Cayenne 3.0 (which is now in beta):

    query.setStatementFetchSize(1000);

This propagates the fetch size setting to the JDBC statement, reducing  
the driver memory footprint.

Andrus

On Dec 24, 2009, at 9:43 AM, Emanuele Maiarelli wrote:

> im using cayenne 2.0.4, and i need to fetch an huge ammout of data,  
> i did that way:
>
>
> public static StringBuffer traceProts;
>
>
>   static {
>       traceProts = new StringBuffer();
>       traceProts.append("SELECT ");
>       traceProts.append("PROTOCOLLI.* ");
>       traceProts.append("FROM PROTOCOLLI, ");
>       traceProts.append("( ");
>       traceProts.append("   SELECT ");
>       traceProts.append("   FORM_PROT_FK ");
>       traceProts.append("   FROM ");
>       traceProts.append("   ( ");
>       traceProts.append("      SELECT ");
>       traceProts.append("      FORM_PROT_FK,SUM(IF(ME_PK is not null, 
> 1,0)) AS ESPORTAZIONI ");
>       traceProts.append("      FROM FORMULARI ");
>       traceProts.append("      LEFT JOIN MOV_EXPORTED ON  
> (FORM_PK=ME_FORMULARI_FK) ");
>       traceProts.append("      GROUP BY FORM_PROT_FK ");
>       traceProts.append("   ) ");
>       traceProts.append("   EXPS ");
>       traceProts.append("   WHERE ESPORTAZIONI=0 ");
>       traceProts.append(") ");
>       traceProts.append("UNEXP ");
>       traceProts.append("where PROT_PK=UNEXP.FORM_PROT_FK");
>   }
>
> this query returns an huge ammont of row,
>
> and im fetching row using IteratedQuery, that's the code im using
>
>       StringBuffer mq=new StringBuffer();
>        Vector<EXPProtocolli> toRet=new  Vector<EXPProtocolli>();
>       mq.append(traceProts);
>       mq.append(" AND PROT_CHIUSO=true AND PROT_RIPARTITO=true");
>       SQLTemplate rawSelect = new SQLTemplate(Protocolli.class,  
> mq.toString());
>             ResultIterator v=ctx.performIteratedQuery(rawSelect);
>       while (v.hasNextRow())
>       {
>           EXPProtocolli prt=new EXPProtocolli();
>           DataRow dataRow = (DataRow) v.nextDataRow();
>
>           Protocolli p=(Protocolli)  
> ctx.objectFromDataRow(Protocolli.class,  dataRow,false);
>
>           this.bindEXPProtMsg(p,prt);
>           prt.setStato(ProtStato.PROT_CARICO_ESPORTABILE);
>           toRet.add(prt);
>       }
>       return toRet;
>
> when running this method i always get an heap space problem
>
> java.lang.OutOfMemoryError: Java heap space
>       at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
>       at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
>       at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
>       at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
>       at  
> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
>       at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
>       at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java: 
> 2542)
>       at  
> com 
> .mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java: 
> 1734)
>       at  
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995)
>       at  
> org 
> .apache 
> .cayenne 
> .access.jdbc.SQLTemplateAction.execute(SQLTemplateAction.java:135)
>       at  
> org 
> .apache 
> .cayenne 
> .access.jdbc.SQLTemplateAction.performAction(SQLTemplateAction.java: 
> 107)
>       at  
> org 
> .apache 
> .cayenne 
> .access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:59)
>       at  
> org.apache.cayenne.access.DataNode.performQueries(DataNode.java:273)
>       at  
> org 
> .apache 
> .cayenne 
> .access 
> .DataDomainLegacyQueryAction 
> .execute(DataDomainLegacyQueryAction.java:81)
>       at org.apache.cayenne.access.DataDomain 
> $1.transform(DataDomain.java:725)
>       at  
> org 
> .apache.cayenne.access.DataDomain.runInTransaction(DataDomain.java: 
> 826)
>       at  
> org.apache.cayenne.access.DataDomain.performQueries(DataDomain.java: 
> 722)
>       at  
> org 
> .apache 
> .cayenne 
> .access.DataContext.internalPerformIteratedQuery(DataContext.java: 
> 1329)
>       at  
> org 
> .apache 
> .cayenne.access.DataContext.performIteratedQuery(DataContext.java: 
> 1295)
>       at services.EXP.Factory.getProtocolliCaricoNonEsp(Factory.java: 
> 85)
>       at services.EXP.EXPServicePBE.getData(EXPServicePBE.java:54)
>       at services.EXPExporter.processRequest(EXPExporter.java:52)
> ....
>
> the error is generated by:     ResultIterator  
> v=ctx.performIteratedQuery(rawSelect);
>
> any hints?
>
>
>
>
>


Mime
View raw message