cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Gentry <mgen...@masslight.net>
Subject Re: Fetching and processing a large amount of objects
Date Wed, 12 Dec 2012 17:32:26 GMT
Hi Simon, some questions:

1) How many records are you talking about?
2) Are you updating your object with a flag/etc you can query on again
later (to exclude objects you've already processed)?
3) What version of Cayenne are you using and what database?
4) When you convert your Map (from the iterated query) into a DataObject,
are you creating a new DataContext or using the old one over and over again?

For #4, if you are using the same DataContext repeatedly, try changing your
logic to something more like:

while (iterator.hasNextRow())
{
    DataContext context = DataContext.createDataContext();
    Map row = (Map) iterator.nextRow();
    CayenneObject object = (CayenneObject)
context.objectFromDataRow("CayenneObject", row);
    ...
    object.doStuff();
    ...
    context.commitChanges();
}

This way you won't build up a ton of objects in a single DataContext and
possibly run out of memory.

mrg


On Wed, Dec 12, 2012 at 8:18 AM, Simon Schneider <sschneider@mackoy.de>
wrote:
>
> Hi list,
>
> my task is to fetch and process a large amount of objects. Reading the
documentation my best choice
> seemed to do a dataContext.performIteratedQuery(query). The problem is
that I get a Java Heap Space
> error, on executing this statement. Now it seems to me the problem is
that performIteratedQuery loads
> all data into a ResultSet.
> My question is how to handle such cases, is there any best practice?
Would it be ok to use a paginated
> query. Then unregistering processed objects from the DataContext they
belong to and also removing
> them from the array that resulted from the paginated query.
>
> Regards Simon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message