cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mr.abanjo" <mr.aba...@gmail.com>
Subject Re: Extract large data from database
Date Wed, 18 Mar 2009 15:54:50 GMT
HI all,thanks for your answers!
Actually i'm using Cayenne 2.0. Resultiterator works well! The memory
doesn't grow, and the data are retrived in a resonable time (100.000 record,
 60Mb in 50/60 seconds... this is the worst case).
So i think that this is a good solution that also allow me to maintain the
same logic i actually use for manage datacontext.

I was sure that Cayenne can solve my problem. Thanks for this great product!

;-)

Davide

On Wed, Mar 18, 2009 at 3:52 PM, Andrus Adamchik <andrus@objectstyle.org>wrote:

> Actually using a ResultIterator should be entirely safe in 3.0 in respect
> to memory management. If a user doesn't do something bad (like storing
> obtained objects in a list himself), and only one object is processed at a
> time, there won't be any memory issues.
>
>  The servlet i'm implementig will be called by a different webapp
>>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>>> warried
>>> about the warning reported in the article:
>>>
>>> "In web applications, programmers must ensure that no open
>>> ResultIterators
>>> are kept between HTTP requests. Failure to do so may result in too many
>>> database connections being locked, thus quickly exhausting connection
>>> pool.
>>>
>>
> This one is still valid. Keep ResultIterator request-scoped, and close it
> at the end.
>
>  In general, an application with Web GUI is NOT a good candidate for
>>> implementation using ResultIterators."
>>>
>>
> Well, there are always exceptions... The general warning here is based on
> the assumption that large results are usually taking lots of time to fetch
> and process. So users may not see the response fast enough. If the response
> time is reasonable in your case, you can use ResultIterator.
>
> Andrus
>
>
> On Mar 18, 2009, at 4:17 PM, Robert Zeigler wrote:
>
>  Problem is that the data context will cache the objects (up to the max
>> object limit); this is alleviated somewhat in 3.0 with the use of weak
>> references
>> in the caching, but there's still potential for running out of memory.
>>
>> You can certainly use ResultIterator, with the caveat that you'll be
>> getting DataRows instead of DataObjects.
>> Alternatively, periodically discard the context that you're using and
>> replace it with a fresh context, something like:
>>
>> Query query = createMyQuery();//SQLTemplate, or SelectQuery, or
>> EJBQLQuery, or whatever.
>> while(notDone) {
>>   ObjectContext context = ...;//create a new object context here.
>>   //do things like set the query fetch limit and fetch offset.
>>   List objs = context.performQuery(query);
>>   for(Obj o : objs) {
>>      //stream out the obj...
>>   }
>> }
>>
>> Robert
>>
>> On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote:
>>
>>  Hi,i'm developing an application that must load a large ammount of data
>>> (100.000 records) and stream them out  from a servlet.
>>>
>>> I created a "for" cycle that for every step load 200 record from the
>>> database (using datacontext.performquery method), and then flush the data
>>> in
>>> the serlvet output stream.
>>> In this way i suppose that the max ammount of memory allocated by cayenne
>>> is
>>> :
>>>
>>> object size * 200
>>>
>>> but seems that this is not true. When i try to get the data from the
>>> servlet, after few seconds, i receive an "out of memory" error. (my 500Mb
>>> allocated for the Heap are reached in a few time... :-(....)
>>>
>>> Which is the best practice in this case?
>>> I can increase the heap, but i want to find a workaround that allow me to
>>> have a small heap allocated during this operation.
>>>
>>> I found this article:
>>> http://cayenne.apache.org/doc20/iterating-through-data-rows.html
>>> Is this the right way?
>>>
>>> The servlet i'm implementig will be called by a different webapp
>>> (backoffice.. in a server 2 server way ) with a low traffic but i'm
>>> warried
>>> about the warning reported in the article:
>>>
>>> "In web applications, programmers must ensure that no open
>>> ResultIterators
>>> are kept between HTTP requests. Failure to do so may result in too many
>>> database connections being locked, thus quickly exhausting connection
>>> pool.
>>> In general, an application with Web GUI is NOT a good candidate for
>>> implementation using ResultIterators."
>>>
>>> Thanks
>>> Davide
>>>
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message