geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Baker <aba...@pivotal.io>
Subject Re: [DISCUSS] TTL setting on WAN
Date Wed, 20 Mar 2019 20:30:56 GMT
An important use case for this is session caching.  Obviously it’s pointless to replicate
an expired session—the user has already gone away.  Copying the bits to the remote cluster
is just creating unnecessary work.

> On Mar 20, 2019, at 11:22 AM, Bruce Schuchardt <bschuchardt@pivotal.io> wrote:
> 
> I don't know why the users didn't use conflation but I suspect they're generating events
with new keys.  Conflation only applies if you're operating on the same keys all the time.
 If you're generating new keys it doesn't help at all.
> 
> On 3/20/19 11:10 AM, Udo Kohlmeyer wrote:
>> If all that the customer is concerned about is that the receiving side gets the "latest"
data, conflation is definitely the best bet. How do we classify old? The only classification
that I have of old (in this context) is that there is a newer version of a data entry. This
classification is not time based, as a TTL approach would require, but state.
>> 
>> What is the use of WAN replication if we are proposing to only replicate *some* of
the data. How would the clusters ever know if they are in-sync? I believe we are just opening
a door that we will never be able to close again and will cause us endless issues.
>> 
>> --Udo
>> 
>> On 3/20/19 10:56, Bruce Schuchardt wrote:
>>> Udo, in the cases I've looked at the user is okay with inconsistency because
they don't really care about the old data. They're most interested in getting the newest data
and keeping the sending site from going down.  I guess the docs for TTL should make it very
clear that it will cause inconsistencies.
>>> 
>>> Conflation does seem like an appropriate thing to try if the same keys are being
updated - I'll do some investigation and see why it wasn't appropriate.
>>> 
>>> 
>>> On 3/20/19 10:51 AM, Udo Kohlmeyer wrote:
>>>> -1, I don't believe this is a feature that we should support. IF a client
is experiencing slow WAN replication and users only care about the "latest" data, then maybe
the user should use "conflation".
>>>> 
>>>> With a TTL model, we are messing with our consistency tenet. I'm am NOT in
support of a setting that can cause inconsistency.
>>>> 
>>>> Dead-letter queues is another area that WILL cause data/site inconsistency.
I think we really have to take a step back, think about WHAT tenets are important to GEODE
and then act accordingly.
>>>> 
>>>> --Udo
>>>> 
>>>> On 3/20/19 10:46, Bruce Schuchardt wrote:
>>>>> IDK Anil, we'll figure that out in the implementation.  I was thinking
it would be in the dispatch threads, so if distribution is need that will happen as it does
now.  I'm hopeful that this won't perturb the code too much.
>>>>> 
>>>>> One thing that was brought up to me in person was the Dead Letter Queue
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478> initiative
that seems to have stalled.  That seems like a very similar idea though it's reacting to errors
coming from the receiving side and not a local condition.  I like the callback, stats, gfsh
and cluster config integration in that write-up & think they might be useful here.  There
is also relevant discussion in that page about things like PDX registrations.  Is that initiative
going to move forward at some point or is it off the boards?
>>>>> 
>>>>> On 3/20/19 10:25 AM, Anilkumar Gingade wrote:
>>>>>> +1. Will the expiration (destroy) be applied on local queues or the
>>>>>> expiration will be replicated (for both serial and parallel)?
>>>>>> 
>>>>>> -Anil.
>>>>>> 
>>>>>> 
>>>>>> On Wed, Mar 20, 2019 at 8:59 AM Bruce Schuchardt <bschuchardt@pivotal.io>
>>>>>> wrote:
>>>>>> 
>>>>>>> We've seen situations where the receiving side of a WAN gateway
is slow
>>>>>>> to accept data or is not accepting any data.  This can cause
queues to
>>>>>>> fill up on the sending side.  If disk-overflow is being used
this can
>>>>>>> even lead to an outage.  Some users are concerned more with the
latest
>>>>>>> data and don't really care if old data is thrown away in this
>>>>>>> situation.  They may have set a TTL on their Regions and would
like to
>>>>>>> be able to do the same thing with their GatewaySenders.
>>>>>>> 
>>>>>>> With that in mind I'd like to add this method to GatewaySenderFactory:
>>>>>>> 
>>>>>>> /** * Sets the timeToLive expiration attribute for queue entries
for the
>>>>>>> next * {@code GatewaySender} created. * * @param timeToLive the
>>>>>>> timeToLive ExpirationAttributes for entries in this region *
@return a
>>>>>>> reference to this GatewaySenderFactory object * @throws
>>>>>>> IllegalArgumentException if timeToLive is null * @see
>>>>>>> RegionFactory#setEntryTimeToLive */ public GatewaySenderFactory
>>>>>>> setEntryTimeToLive(ExpirationAttributes timeToLive);
>>>>>>> 
>>>>>>> The exact implementation may not be the same as for Regions since
we
>>>>>>> probably want to expire the oldest entries first and make sure
we do so
>>>>>>> in their order in the queue.
>>>>>>> 
>>>>>>> 
>>>>> 


Mime
View raw message