Mailing-List: contact dev-help@fluo.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@fluo.incubator.apache.org
MIME-Version: 1.0
References: <4ce0516a7eca4d78a6e1e25bf3481d39@ALHUN12EXCH02.Parsons.com>
 <CAL5zq9YbsWFOEJdVNT+5FFEOjk1AeTPhifaHuiY2AcC2xaMkmQ@mail.gmail.com>
 <1485961333946.54190@parsons.com> <CAL5zq9bk9h7-6_x2t+vqvPD0YOtphdbE67ffv=60w-+UT9W95A@mail.gmail.com>
 <CAGUtCHqrqjKCQA4A+335MoCu6ngydh9k2EQnZcvdW0L-+-efdQ@mail.gmail.com> <082e08d83ed34993a3da4d02a99c16c8@ALHUN12EXCH02.Parsons.com>
In-Reply-To: <082e08d83ed34993a3da4d02a99c16c8@ALHUN12EXCH02.Parsons.com>
From: Mike Walch <mwalch@apache.org>
Date: Thu, 02 Feb 2017 23:18:00 +0000
Message-ID: <CAL4xCsmGhLpX+irhDA+6cxM_DbksSKkUWbO5=9YwkGH94j+XNQ@mail.gmail.com>
Subject: Re: third party service to poll Fluo for absence of event
To: dev@fluo.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1141e074f19e7105479460df
archived-at: Thu, 02 Feb 2017 23:19:25 -0000

--001a1141e074f19e7105479460df
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Caleb,

I wouldn't try to follow our Twill code too closely.  We have a lot of
abstraction in the code that I don't think you need.  We do use TwillRunner
and TwillController.  One major tip is that you should consider whether or
not you want to pull in all of Twill dependencies (which may cause
dependency conflicts). You can avoid pulling in Twill's dependencies by
creating a bundled jar and using Twill's BundleJarRunnable.  We didn't do
this in Fluo right now but I would like to in the future.  If you want to
see an example, take a look at the new accumulo-testing repo[1] which uses
Twill to launch tests using bundled jars. In particular, look at the class
YarnAccumuloTestRunner[2].

Best,
Mike

[1]: https://github.com/apache/accumulo-testing
[2]:
https://github.com/apache/accumulo-testing/blob/master/yarn/src/main/java/o=
rg/apache/accumulo/testing/yarn/YarnAccumuloTestRunner.java


On Thu, Feb 2, 2017 at 5:15 PM Meier, Caleb <Caleb.Meier@parsons.com> wrote=
:

> Thanks for the input.  I'm currently looking at creating some sort of
> coordinator (which wraps a ScheduledExecutorService to generate periodic
> notifications) and a collection of workers (to process the periodic queri=
es
> as they are issued).  Most of the interaction between the workers and
> coordinator will be via Kafka (develop some sort of protocol to ensure th=
at
> more than one worker isn't getting assigned the same query).  At any rate=
,
> I was thinking of implementing these components as TwillRunnables.
> However, it seems like the Twill documentation is a bit sparse.  Given th=
at
> you guys implemented Fluo as a TwillApplication, do you have any
> insight/advice for writing TwillApplications?  In particular, how is your
> FluoTwillApp being run?  All of the examples I've seen create a client wi=
th
> a TwillRunner and TwillController.  It seems like you 've created your ow=
n
> version of a YarnAppRunner -- what role is that playing in running the
> FluoTwillApp?  Moreover, it is also unclear to me whether the
> TwillRunnables are bound to the client -- if the client terminates do the
> runnables terminate as well?  So essentially, it is unclear to me how
> create a long running application in Twill that is not bound to a
> particular client.  Sorry that this is a little off topic, but any help,
> references to documentation/examples would be very appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II =E2=99=A6 Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 =E2=99=A6 Arlington, VA 22209
> Office:  (703)797-3066 <(703)%20797-3066>
> Caleb.Meier@Parsons.com =E2=99=A6 www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ctubbsii@apache.org> wrote:
> > On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Caleb.Meier@parsons.com>
> > wrote:
> >
> >> Yeah, this seems pretty reasonable to me.  I guess it then boils down
> >> to the nitty gritty of do I store results in Fluo and have my service
> >> query Fluo (I think you guys actually advise against that in your
> >> documentation), or export results and then have the service query
> >> some external index that I am exporting to.
> >>
> >>
> > I'm not sure we advise against it, so much as recognize that it may
> > not be suitable for certain use cases and may not meet query
> > performance expectations (
> >
> https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__fluo.apache.org_doc=
s_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=3DCwIFaQ&c=3DNwf-pp4x=
tYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=3DvuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4=
WXTD0qo8&m=3DzqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=3DspmwJN_FBTO6TB=
BT2dne8sbE7MRMrlhz8lLPpfPZBbs&e=3D
> ).
> >
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and
> injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions.   Failed transactions are not cleaned
> up until something tries to read the data, which could significantly dela=
y
> reads.
>
> > In any case, your observer need not write the final "last occurrence"
> > entries into a Fluo table. It could write them anywhere.
> >
> >
> >> Regarding timestamps, does the oracle server provide actual
> >> timestamps or just logical timestamps?  That is, could I use the
> >> timestamps that the server provides to define some sort of now()
> >> function to obtain the current time to compare with the times of
> incoming events?
> >>
> >
> > Just logical time, and it delivers batches to limit locking, so it can
> > appear to jump ahead spontaneously. I'm not sure the OracleServer is
> > suitable for this purpose. What level of precision are you going for?
> > It might be enough to just run NTP, if you don't need more precision
> > than "within seconds".
> >
> >
> >> ________________________________________
> >> From: Christopher <ctubbsii@apache.org>
> >> Sent: Tuesday, January 31, 2017 5:08 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: third party service to poll Fluo for absence of event
> >>
> >> You could write an observer which rolls up timestamps from all the
> >> events you are concerned about, and puts the most recent event
> >> timestamp into a centralized place, which you could poll. If there is
> >> no ingest of these events, then the last timestamp in this central
> >> place will exceed some threshold and the poller could detect that and
> trigger additional actions.
> >>
> >> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
> >> <Caleb.Meier@parsons.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I=E2=80=99m looking into using Fluo to develop an event based notifi=
cation
> >> > system that incrementally generates events of increasing
> >> > complexity.  The one issue that I=E2=80=99m running into is how to h=
andle
> >> > the non-event event.  That
> >> is,
> >> > Fluo (as I understand it) is not well-suited to handle the
> >> > following
> >> > request: =E2=80=9Cgenerate a notification if no events of a given ty=
pe have
> >> > occurred within the last 24 hours=E2=80=9D.  This is because it is a=
 push
> >> > based notification framework that only generates notifications when
> >> > things actually happen.  So the question is, has anyone looked into
> >> > developing a service for generating notifications at regular
> >> > intervals (even if something doesn=E2=80=99t happen) that works with=
 Fluo?
> >> > I=E2=80=99m toying with the idea of creating some sort of Twill appl=
ication
> >> > that tells Fluo to wake up at regular intervals to generate a
> >> > notification about the set of events falling within the given time
> >> > window. Before doing this I just wanted to make sure that something
> >> > like this does not already exist, and I also
> >> want
> >> > to get a sense of how bad an idea it is to delegate some of the logi=
c
> of
> >> > this periodic notification service to Fluo.   Would it be better to
> >> > separate out the temporal portion of my notification request to be
> >> > processed entirely outside of Fluo to avoid transactional overhead?
> >> >
> >> > Caleb A. Meier, Ph.D.
> >> > Software Engineer II =E2=99=A6 Analyst
> >> > Parsons Corporation
> >> > 1911 N. Fort Myer Drive, Suite 800 =E2=99=A6 Arlington, VA 22209
> >> > Office:  (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
> <(703)%20797-3066>
> >> > Caleb.Meier@Parsons.com<mailto:Caleb.Meier@Parsons.com> =E2=99=A6
> >> www.parsons.com<
> >> > http://www.parsons.com/>
> >> >
> >> > --
> >> Christopher
> >>
> > --
> > Christopher
>

--001a1141e074f19e7105479460df--