cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Zeigler <robert.zeig...@gmail.com>
Subject Re: Extract large data from database
Date Wed, 18 Mar 2009 14:17:24 GMT
Problem is that the data context will cache the objects (up to the max  
object limit); this is alleviated somewhat in 3.0 with the use of weak  
references
in the caching, but there's still potential for running out of memory.

You can certainly use ResultIterator, with the caveat that you'll be  
getting DataRows instead of DataObjects.
Alternatively, periodically discard the context that you're using and  
replace it with a fresh context, something like:

Query query = createMyQuery();//SQLTemplate, or SelectQuery, or  
EJBQLQuery, or whatever.
while(notDone) {
     ObjectContext context = ...;//create a new object context here.
     //do things like set the query fetch limit and fetch offset.
     List objs = context.performQuery(query);
     for(Obj o : objs) {
        //stream out the obj...
     }
}

Robert

On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote:

> Hi,i'm developing an application that must load a large ammount of  
> data
> (100.000 records) and stream them out  from a servlet.
>
> I created a "for" cycle that for every step load 200 record from the
> database (using datacontext.performquery method), and then flush the  
> data in
> the serlvet output stream.
> In this way i suppose that the max ammount of memory allocated by  
> cayenne is
> :
>
> object size * 200
>
> but seems that this is not true. When i try to get the data from the
> servlet, after few seconds, i receive an "out of memory" error. (my  
> 500Mb
> allocated for the Heap are reached in a few time... :-(....)
>
> Which is the best practice in this case?
> I can increase the heap, but i want to find a workaround that allow  
> me to
> have a small heap allocated during this operation.
>
> I found this article:
> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
> Is this the right way?
>
> The servlet i'm implementig will be called by a different webapp
> (backoffice.. in a server 2 server way ) with a low traffic but i'm  
> warried
> about the warning reported in the article:
>
> "In web applications, programmers must ensure that no open  
> ResultIterators
> are kept between HTTP requests. Failure to do so may result in too  
> many
> database connections being locked, thus quickly exhausting  
> connection pool.
> In general, an application with Web GUI is NOT a good candidate for
> implementation using ResultIterators."
>
> Thanks
> Davide


Mime
View raw message