Mailing-List: contact user-help@cayenne.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cayenne.apache.org
Received-SPF: pass (athena.apache.org: domain of h.pikkemaat@tsi-solutions.nl
 designates 217.119.229.242 as permitted sender)
Message-ID: <4AFBCAC9.9060706@tsi-solutions.nl>
Date: Thu, 12 Nov 2009 09:43:53 +0100
From: Hans Pikkemaat <h.pikkemaat@tsi-solutions.nl>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: user@cayenne.apache.org
Subject: Re: Object Caching
References: <4AFBC247.5000909@tsi-solutions.nl>
 <24078BC3-F5DF-40C2-BD44-9CDDD96D9A97@objectstyle.org>
In-Reply-To: <24078BC3-F5DF-40C2-BD44-9CDDD96D9A97@objectstyle.org>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

Yes, the paginated query would indeed be the only way for me to go forward.
The problem however is that I get the exception I posted earlier.

tx

Hans

Andrus Adamchik wrote:
> For paginated queries we contemplated a strategy of a list with  
> constant size of fully resolved objects. I.e. when a page is swapped  
> in, some other (LRU?) page is swapped out. We decided against it, as  
> in a general case it is hard to consistently predict which page should  
> be swapped out.
>
> However it should be rather easy to write such a list for a specific  
> case with a known access order (e.g. a standard iteration order). In  
> fact I would vote to even include such implementation in Cayenne going  
> forward.
>
> More specifically, you can extend IncrementalFaultList [1], overriding  
> 'resolveInterval' to swap out previously read pages, turning them back  
> into ids. And the good part is that you can use your extension  
> directly without any need to modify the rest of Cayenne.
>
> Andrus
>
>
> [1] http://cayenne.apache.org/doc/api/org/apache/cayenne/access/IncrementalFaultList.html
>
>
> On Nov 12, 2009, at 10:07 AM, Hans Pikkemaat wrote:
>   
>> Hi,
>>
>> So this means that if I use a generic query that the query results  
>> are always stored
>> completely in the object store (or the query cache if I configure it).
>>
>> Objects are returned in a list so as long I have a reference to this  
>> list (because I'm
>> traversing it) these objects are not garbage collected.
>>
>> If I use the query cache the full query results are cached. This  
>> means that I can only
>> tell it to remove the whole query.
>>
>> Effectively this means I'm unable to run a big query and process the  
>> results as a stream.
>> So I cannot process the first results and then somehow make them  
>> available for
>> garbage collection.
>>
>> The only option I have would be the iterated query but this is only  
>> usefull for queries
>> one 1 table without any relations because it is not possible to use  
>> prefetching nor is
>> it possible to manually construct relations between obects.
>>
>> My conclusion here is that cayenne is simply not suitable for doing  
>> large batch wise
>> query processing because of the memory implications.
>>
>> tx
>>
>> HPI
>>
>> Andrus Adamchik wrote:
>>     
>>> As mentioned in the docs, individual objects and query lists are
>>> cached independently. Of course query lists contain a subset of  
>>> cached
>>> object store objects inside the lists. An object won't get gc'd if it
>>> is also stored in the query list.
>>>
>>> Now list cache expiration is controlled via query cache factory. By
>>> default this is an LRU map, so as long as the map has enough space to
>>> hold lists (its capacity == # of lists, not # of objects), the  
>>> objects
>>> won't get gc'd.
>>>
>>> You can explicitly remove entries from the cache via QueryCache  
>>> remove
>>> and removeGroup methods. Or you can use a different QueryCacheFactory
>>> that implements some custom expiration/cleanup mechanism.
>>>
>>> Andrus
>>>
>>> On Nov 11, 2009, at 3:43 PM, Hans Pikkemaat wrote:
>>>
>>>
>>>       
>>>> Hi,
>>>>
>>>> I use the latest version of cayenne, 3.0b and am experimenting with
>>>> the object caching features.
>>>>
>>>> The documentation states that committed objects are purged from the
>>>> cache because it uses weak references.
>>>> (http://cayenne.apache.org/doc/individual-object-caching.html)
>>>>
>>>> If I however run a query using SQLTemplate which caches the objects
>>>> into the dataContext local cache (objectstore),
>>>> the objects don't seem to be purged at all. If I simply run the
>>>> query dump the contents using an iterator on the resulting
>>>> List then the nr of registered objects in the objectstore stays the
>>>> same (dataContext.getObjectStore().registeredObjectsCount()).
>>>> Even if I manually run System.gc() I don't see any changes (I know
>>>> this can be normal as gc() doesn't guarantee anything)
>>>>
>>>> What am I doing wrong? Under which circumstances will cayenne purge
>>>> the cache?
>>>>
>>>> tx
>>>>
>>>> Hans
>>>>
>>>>
>>>>         
>>>