incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: Demux trigger
Date Mon, 19 Apr 2010 23:23:03 GMT
Sure, I can make that changes. I named it processDataLoaders since it
handles a collection.

I created a JIRA with a first pass of the implementation, along with a few
points of discussion:

https://issues.apache.org/jira/browse/CHUKWA-477

Let me know what you think.

thanks,
Bill

On Mon, Apr 19, 2010 at 10:23 AM, Eric Yang <eyang@yahoo-inc.com> wrote:

> Yes this is indeed the case.  +1 on adding hook inside the if block, to
> generate post move triggers.  Could you also rename processDemuxPigOutput
> to
> processDataLoader?  The name is more concise.  Thanks
>
> Regards,
> Eric
>
>
> On 4/19/10 9:54 AM, "Bill Graham" <billgraham@gmail.com> wrote:
>
> > Thanks, Eric.
> >
> > Looking into the PostProcessorManager code a little more, it seems the
> > chukwa.post.demux.data.loader loaders get called before the post
> processor
> > moves the finished files into place. I need a trigger that fires after
> they're
> > in place in the repos/ dir.
> >
> > This is the code I'm referring to from PostProcessorManager.
> >
> > if ( processDemuxPigOutput(directoryToBeProcessed) == true) {
> >   if (movetoMainRepository(directoryToBeProcessed,chukwaRootReposDir) ==
> true)
> > {
> >     deleteDirectory(directoryToBeProcessed);
> >     ...
> >     continue;
> >   }
> > }
> >
> > The data loaders get called as part of processDemuxPigOutput. Is this
> sequence
> > correct or am I missing something?
> >
> > If this is in fact the case, I'd like to add a hook to take some post
> action
> > once the files are in the /repos dir. From a users perspective, that's
> what
> > 'post demux' implies. I'm open for suggestions re the best way to do that
> and
> > what to call the configs. One thought is to follow a similar pattern as
> how
> > DataLoaders are configured, but use a new interface that's more generic
> than
> > loading data. Not 'Action', but something that denotes that.
> >
> > thanks,
> > Bill
> >
> >
> > On Sat, Apr 17, 2010 at 1:35 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> >> Yes.
> >>
> >> Regards,
> >> Eric
> >>
> >>
> >>
> >> On 4/16/10 9:56 PM, "Bill Graham" <billgraham@gmail.com
> >> <http://billgraham@gmail.com> > wrote:
> >>
> >>> Thanks Eric, I'm glad I emailed before writing code.
> >>>
> >>> I can see how data loaders get triggered, but I don't see one that
> makes an
> >>> HTTP request like I'm proposing. Are you suggesting I implement a new
> >>> DataLoader that doesn't actually load data, but makes an HTTP request
> >>> instead?
> >>>
> >>>
> >>> On Fri, Apr 16, 2010 at 7:30 PM, Eric Yang <eyang@yahoo-inc.com
> >>> <http://eyang@yahoo-inc.com> > wrote:
> >>>> Hi Bill,
> >>>>
> >>>> This already exist in Chukwa.  Take a look in DataLoaderFactory.java
> and
> >>>> SocketDataLoader.java, they are triggered after demux jobs.  Hence,
> you can
> >>>> use PostProcessManager as triggers, and configure it through
> >>>> chukwa-demux-conf.xml, chukwa.post.demux.data.loader.  Hope this
> helps.
> >>>>
> >>>> Regards,
> >>>> Eric
> >>>>
> >>>>
> >>>> On 4/16/10 4:39 PM, "Bill Graham" <billgraham@gmail.com
> >>>> <http://billgraham@gmail.com>  <http://billgraham@gmail.com>
> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I'd like to add a feature to the DemuxManager where you can configure
> an
> >>>>> HTTP request to be fired after a Demux run. It would be similar
to
> what's
> >>>>> currently there for Nagios alerts, only this would be HTTP (the
> Nagios
> >>>>> alert is a raw TCP socket call). You'd configure the host, port,
> >>>>> (POST|GET) and uri for this first pass.
> >>>>>
> >>>>> Some metadata about the job would also go along for the ride. Maybe
> things
> >>>>> like status code and job name.
> >>>>>
> >>>>> The use case is to trigger a dependent job to run elsewhere upon
> >>>>> completion. The same functionality could potentially be ported to
> some of
> >>>>> the other chukwa processor jobs if the need arose.
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> thanks,
> >>>>> Bill
> >>>>>
> >>>
> >>>
> >
> >
>
>

Mime
View raw message