cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Razumovsky <razumovsky.and...@gmail.com>
Subject Re: out of memory using ResultIterator
Date Thu, 24 Dec 2009 09:36:13 GMT
ROP's became better, e.g. it now supports nesting of contexts same way as
DataContexts do

2009/12/24 Emanuele Maiarelli <emanuele@mengozzi.com>

>   I'll give a try once the project will me stable on the 2.0.4. This
> project uses ROP with a swing rich application, plus some webservices for
> batchjobs, are ROP still working good on 3.0
> as they works on 2.0.4?
>
> Andrus Adamchik ha scritto:
>
>  Sure. Hopefully within a couple of weeks 3.0 becomes Release Candidate and
>> final after that, so you may give it another look.
>>
>> For now I guess the only way is to write a custom query extending
>> SQLTemplate and overriding 'createSQLAction' to call 'setFetchSize' on the
>> JDBC statement. Implementing a custom SQLAction can be pretty involved,
>> however if you are doing a one-off thing, it is doable.
>>
>> Also I would recommend to switch to Cayenne 3.0 at least temporarily to
>> confirm that the fix I described is indeed working for you. There may be
>> some other things at play here. Who knows.
>>
>> Andrus
>>
>>
>> On Dec 24, 2009, at 10:48 AM, Emanuele Maiarelli wrote:
>>
>>>
>>>  I'm trying to figureout a solution using 2.0.4.
>>>
>>>  I don't feel confortable in switching to 3.0 at this stage of the
>>> project.
>>>
>>>
>>>  Thank you,
>>>
>>>  Emanuele
>>> Andrus Adamchik ha scritto:
>>>
>>>> This is likely an issue with MySQL driver caching returned data. The
>>>> solution was implemented in Cayenne 3.0 (which is now in beta):
>>>>
>>>>  query.setStatementFetchSize(1000);
>>>>
>>>> This propagates the fetch size setting to the JDBC statement, reducing
>>>> the driver memory footprint.
>>>>
>>>> Andrus
>>>>
>>>> On Dec 24, 2009, at 9:43 AM, Emanuele Maiarelli wrote:
>>>>
>>>>  im using cayenne 2.0.4, and i need to fetch an huge ammout of data, i
>>>>> did that way:
>>>>>
>>>>>
>>>>> public static StringBuffer traceProts;
>>>>>
>>>>>
>>>>>  static {
>>>>>     traceProts = new StringBuffer();
>>>>>     traceProts.append("SELECT ");
>>>>>     traceProts.append("PROTOCOLLI.* ");
>>>>>     traceProts.append("FROM PROTOCOLLI, ");
>>>>>     traceProts.append("( ");
>>>>>     traceProts.append("   SELECT ");
>>>>>     traceProts.append("   FORM_PROT_FK ");
>>>>>     traceProts.append("   FROM ");
>>>>>     traceProts.append("   ( ");
>>>>>     traceProts.append("      SELECT ");
>>>>>     traceProts.append("      FORM_PROT_FK,SUM(IF(ME_PK is not
>>>>> null,1,0)) AS ESPORTAZIONI ");
>>>>>     traceProts.append("      FROM FORMULARI ");
>>>>>     traceProts.append("      LEFT JOIN MOV_EXPORTED ON
>>>>> (FORM_PK=ME_FORMULARI_FK) ");
>>>>>     traceProts.append("      GROUP BY FORM_PROT_FK ");
>>>>>     traceProts.append("   ) ");
>>>>>     traceProts.append("   EXPS ");
>>>>>     traceProts.append("   WHERE ESPORTAZIONI=0 ");
>>>>>     traceProts.append(") ");
>>>>>     traceProts.append("UNEXP ");
>>>>>     traceProts.append("where PROT_PK=UNEXP.FORM_PROT_FK");
>>>>>  }
>>>>>
>>>>> this query returns an huge ammont of row,
>>>>>
>>>>> and im fetching row using IteratedQuery, that's the code im using
>>>>>
>>>>>     StringBuffer mq=new StringBuffer();
>>>>>      Vector<EXPProtocolli> toRet=new  Vector<EXPProtocolli>();
>>>>>     mq.append(traceProts);
>>>>>     mq.append(" AND PROT_CHIUSO=true AND PROT_RIPARTITO=true");
>>>>>     SQLTemplate rawSelect = new SQLTemplate(Protocolli.class,
>>>>> mq.toString());
>>>>>           ResultIterator v=ctx.performIteratedQuery(rawSelect);
>>>>>     while (v.hasNextRow())
>>>>>     {
>>>>>         EXPProtocolli prt=new EXPProtocolli();
>>>>>         DataRow dataRow = (DataRow) v.nextDataRow();
>>>>>
>>>>>         Protocolli p=(Protocolli)
>>>>> ctx.objectFromDataRow(Protocolli.class,  dataRow,false);
>>>>>
>>>>>         this.bindEXPProtMsg(p,prt);
>>>>>         prt.setStato(ProtStato.PROT_CARICO_ESPORTABILE);
>>>>>         toRet.add(prt);
>>>>>     }
>>>>>     return toRet;
>>>>>
>>>>> when running this method i always get an heap space problem
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>     at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
>>>>>     at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
>>>>>     at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
>>>>>     at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
>>>>>     at
>>>>> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
>>>>>     at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
>>>>>     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
>>>>>     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
>>>>>     at
>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
>>>>>
>>>>>     at
>>>>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995)
>>>>>     at
>>>>> org.apache.cayenne.access.jdbc.SQLTemplateAction.execute(SQLTemplateAction.java:135)
>>>>>
>>>>>     at
>>>>> org.apache.cayenne.access.jdbc.SQLTemplateAction.performAction(SQLTemplateAction.java:107)
>>>>>
>>>>>     at
>>>>> org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:59)
>>>>>
>>>>>     at
>>>>> org.apache.cayenne.access.DataNode.performQueries(DataNode.java:273)
>>>>>     at
>>>>> org.apache.cayenne.access.DataDomainLegacyQueryAction.execute(DataDomainLegacyQueryAction.java:81)
>>>>>
>>>>>     at
>>>>> org.apache.cayenne.access.DataDomain$1.transform(DataDomain.java:725)
>>>>>     at
>>>>> org.apache.cayenne.access.DataDomain.runInTransaction(DataDomain.java:826)
>>>>>     at
>>>>> org.apache.cayenne.access.DataDomain.performQueries(DataDomain.java:722)
>>>>>     at
>>>>> org.apache.cayenne.access.DataContext.internalPerformIteratedQuery(DataContext.java:1329)
>>>>>
>>>>>     at
>>>>> org.apache.cayenne.access.DataContext.performIteratedQuery(DataContext.java:1295)
>>>>>
>>>>>     at services.EXP.Factory.getProtocolliCaricoNonEsp(Factory.java:85)
>>>>>     at services.EXP.EXPServicePBE.getData(EXPServicePBE.java:54)
>>>>>     at services.EXPExporter.processRequest(EXPExporter.java:52)
>>>>> ....
>>>>>
>>>>> the error is generated by:     ResultIterator
>>>>> v=ctx.performIteratedQuery(rawSelect);
>>>>>
>>>>> any hints?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>


-- 
Andrey

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message