geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anilkumar Gingade <aging...@pivotal.io>
Subject Re: [DISCUSS] TTL setting on WAN
Date Tue, 26 Mar 2019 17:16:15 GMT
Yes. From our experiment that looked like a possibility.

-Anil.


On Tue, Mar 26, 2019 at 9:59 AM Dan Smith <dsmith@pivotal.io> wrote:

> Following up on the conflation thing - Anil and I did an experiment.
> Conflation definitely *does* happen on everything in the queue, not just
> the last batch. But we didn't see destroys get conflated with updates.
>
> So one thing that might make this use case work is to conflate the destroys
> with the updates. Then the disk space would be freed up when the expiration
> events are conflated in the queue.
>
> -Dan
>
> On Tue, Mar 26, 2019 at 8:19 AM Bruce Schuchardt <bschuchardt@pivotal.io>
> wrote:
>
> > I've been thinking along those lines as well Suranjan.  Since conflation
> > and expiry-forwarding don't solve the problem of running out of disk
> > space the solution needs to involve the dispatch thread.
> >
> > For the session-state caching scenario that raised this whole issue I
> > think what you've described will work.  Looking at it with a wider lens
> > I'm a little concerned about a TTL on the queue because multiple regions
> > can feed into the same queue and you might not have the same TTL
> > settings on all of those regions.
> >
> > On 3/25/19 4:53 PM, Suranjan Kumar wrote:
> > > Hi,
> > >   I think the one approach for a user would be to 'filter' the events
> > while
> > > dispatching. If I remember correctly, we can attach a filter at
> dispatch
> > > time and filter the events based on creationTime of the GatewayEvent.
> We
> > > can provide a pre created filter and use it based on some so that user
> > > doesn't have to write his/her own.
> > >
> > > Something like,
> > > /**
> > > All the events which spend timeToLive or more time in queue will be
> > deleted
> > > from the queue
> > > and will not be sent to remote site.
> > > Possible consequence is that two sites can be inconsistent in case
> > > */
> > > public GatewaySenderFactory setEntryTimeToLive(long timeToLive);
> > >
> > > As queues will be read in LRU way this would be faster too. Only
> drawback
> > > is that there will be only one thread (not sure if we have concurrent
> > > dispatcher yet) clearing the queue.
> > >
> > > As Udo/Dan mentioned above, user needs to be aware of the consequences.
> > >
> > >
> > > On Tue, Mar 26, 2019 at 3:09 AM Bruce Schuchardt <
> bschuchardt@pivotal.io
> > >
> > > wrote:
> > >
> > >> I've walked through the code to forward expiration actions to async
> > >> event listeners & don't see how to apply it to removal of queue
> entries
> > >> for WAN.  The current implementation just queues the expiration
> > >> actions.  If we wanted to remove any queued events associated with the
> > >> expired region entry we'd have to scan the whole queue, which would be
> > >> too slow if we're overflowing the queue to disk.
> > >>
> > >> I've also walked through the conflation code.  It applies only to the
> > >> current batch being processed by the gateway sender.  The data
> structure
> > >> used to perform conflation is just a Map that is created in the
> sender's
> > >> batch processing method and then thrown away.
> > >>
> > >> On 3/20/19 11:15 AM, Dan Smith wrote:
> > >>>> 2) The developer wants to replicate _state_.  This means that
> implicit
> > >>>> state changes (expiration or eviction w/ destroy) could allow us
to
> > >>>> optimize the queue size.  This is very similar to conflation, just
a
> > >>>> different kind of optimization.
> > >>>>
> > >>>> For this second case, does it make sense to allow the user to
> specify
> > a
> > >>>> different TTL than the underlying region?  It seems like what the
> user
> > >>>> wants is to not replicate stale data and having an extra TTL
> attribute
> > >>>> would just be another value to mis-configure.  What do you think
> about
> > >> just
> > >>>> providing a boolean flag?
> > >>>>
> > >>>>
> > >>> This kinda jogged my memory. AsyncEventQueues actually *do* have a
> > >> boolean
> > >>> flag to allow you to forward expiration events to the queue. I have
> no
> > >> idea
> > >>> how this interacts with conflation though -
> > >>>
> > >>
> >
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueueFactory.html#setForwardExpirationDestroy-boolean-
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message