geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Schuchardt <bschucha...@pivotal.io>
Subject Re: [DISCUSS] TTL setting on WAN
Date Tue, 26 Mar 2019 15:19:23 GMT
I've been thinking along those lines as well Suranjan.  Since conflation 
and expiry-forwarding don't solve the problem of running out of disk 
space the solution needs to involve the dispatch thread.

For the session-state caching scenario that raised this whole issue I 
think what you've described will work.  Looking at it with a wider lens 
I'm a little concerned about a TTL on the queue because multiple regions 
can feed into the same queue and you might not have the same TTL 
settings on all of those regions.

On 3/25/19 4:53 PM, Suranjan Kumar wrote:
> Hi,
>   I think the one approach for a user would be to 'filter' the events while
> dispatching. If I remember correctly, we can attach a filter at dispatch
> time and filter the events based on creationTime of the GatewayEvent. We
> can provide a pre created filter and use it based on some so that user
> doesn't have to write his/her own.
>
> Something like,
> /**
> All the events which spend timeToLive or more time in queue will be deleted
> from the queue
> and will not be sent to remote site.
> Possible consequence is that two sites can be inconsistent in case
> */
> public GatewaySenderFactory setEntryTimeToLive(long timeToLive);
>
> As queues will be read in LRU way this would be faster too. Only drawback
> is that there will be only one thread (not sure if we have concurrent
> dispatcher yet) clearing the queue.
>
> As Udo/Dan mentioned above, user needs to be aware of the consequences.
>
>
> On Tue, Mar 26, 2019 at 3:09 AM Bruce Schuchardt <bschuchardt@pivotal.io>
> wrote:
>
>> I've walked through the code to forward expiration actions to async
>> event listeners & don't see how to apply it to removal of queue entries
>> for WAN.  The current implementation just queues the expiration
>> actions.  If we wanted to remove any queued events associated with the
>> expired region entry we'd have to scan the whole queue, which would be
>> too slow if we're overflowing the queue to disk.
>>
>> I've also walked through the conflation code.  It applies only to the
>> current batch being processed by the gateway sender.  The data structure
>> used to perform conflation is just a Map that is created in the sender's
>> batch processing method and then thrown away.
>>
>> On 3/20/19 11:15 AM, Dan Smith wrote:
>>>> 2) The developer wants to replicate _state_.  This means that implicit
>>>> state changes (expiration or eviction w/ destroy) could allow us to
>>>> optimize the queue size.  This is very similar to conflation, just a
>>>> different kind of optimization.
>>>>
>>>> For this second case, does it make sense to allow the user to specify a
>>>> different TTL than the underlying region?  It seems like what the user
>>>> wants is to not replicate stale data and having an extra TTL attribute
>>>> would just be another value to mis-configure.  What do you think about
>> just
>>>> providing a boolean flag?
>>>>
>>>>
>>> This kinda jogged my memory. AsyncEventQueues actually *do* have a
>> boolean
>>> flag to allow you to forward expiration events to the queue. I have no
>> idea
>>> how this interacts with conflation though -
>>>
>> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueueFactory.html#setForwardExpirationDestroy-boolean-

Mime
View raw message