fluo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: third party service to poll Fluo for absence of event
Date Thu, 02 Feb 2017 03:58:38 GMT
On Wed, Feb 1, 2017 at 9:53 AM, Meier, Caleb <Caleb.Meier@parsons.com> wrote:
> Hey Keith,
> Right now the project is in its infancy so I don't have concrete examples of how events
are distributed, but I think we would like everything to scale as needed.  Right now, the
user is in charge of how events are modeled and how many different event types are included.
> Regarding the use cases that you mentioned, I think we would be more interested in find
which event types haven't been updated or haven't occurred within the last 24 hours.  I'm
not ruling out the possibility of tracking updates to an individual event though.
> More generally, we're interested in issuing periodic notifications.  The non-event event
is kind of a canonical example.  Another use case would be "generate a notification if more
than 100 events of a given type have occurred within the last 24 hours",  and the user specifies
that they want a notification to be generated every hour.  In the case that 1000 events happen
in the first hour and then no other events occur during that 24 hour period, 1000 notifications
would be generated by Fluo during the first hour, but then nothing would happen the remaining
23 hours.  We would need some service (that possibly utilizes an observer if that makes sense)
to periodically detect how many events have occurred within the specified window and generate
a notification.  So in this case, every hour for the next 23 hours, something would check
and find that 1000 events occurred within the window and generate a notification.
> Did that clarify things?

Yeah.  I feel like this could be broken into two parts.

 * A many to few update recipe with short circuiting.
 * Something that periodically scans the "few" row/cols and possibly
injects a new action.

Below is an example to show what I mean by short circuiting.

 1. Processing event 100934 with event type X
 2. Update the last update hour for event type X only if its value is
less than current hour

The Collision Free Map (CFM) could be used for these many to few
updates.  However, it lacks the short circuit optimization.  It will
always queue the updates, even when the applied update will not change
the current value.

The external scan service periodically scan the few event type last
update times and inject new notifications.  This mechanism could use
transactions to avoid problems from accidentally running multiple
external services.

> ________________________________________
> From: Keith Turner <keith@deenlo.com>
> Sent: Tuesday, January 31, 2017 6:43 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
> On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Caleb.Meier@parsons.com> wrote:
>> Hello,
>> I’m looking into using Fluo to develop an event based notification system that
incrementally generates events of increasing complexity.  The one issue that I’m running
into is how to handle the non-event event.  That is, Fluo (as I understand it) is not well-suited
to handle the following request: “generate a notification if no events of a given type have
occurred within the last 24 hours”.  This is because it is a push based notification framework
that only generates notifications when things actually happen.  So the question is, has anyone
looked into developing a service for generating notifications at regular intervals (even if
something doesn’t happen) that works with Fluo?  I’m toying with the idea of creating
some sort of Twill application that tells Fluo to wake up at regular intervals to generate
a notification about the set of events falling within the given time window. Before doing
this I just wanted to make sure that something like this does not already exist, and I also
want to get a sense of how bad an idea it is to delegate some of the logic of this periodic
notification service to Fluo.   Would it be better to separate out the temporal portion of
my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
> At one point in time I was thinking of delayed notifications, but I
> can't remember what my specific use case was. I am not sure if this is
> good idea or not.
> How many event types are there in your use case?  Just curious about
> the order of magnitude.  Like is it ~100 event types and millions of
> events?
> Just to make sure I understand your use case, are you interested in
> finding which event types were not updated in the last 24hrs?  Or are
> you interested in finding which events were not updated in the last
> 24hrs?  If none of these, would you be able to describe the use case
> you want with an example?
>> Caleb A. Meier, Ph.D.
>> Software Engineer II ♦ Analyst
>> Parsons Corporation
>> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> Office:  (703)797-3066
>> Caleb.Meier@Parsons.com<mailto:Caleb.Meier@Parsons.com> ♦ www.parsons.com<http://www.parsons.com/>

View raw message