flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Capwell <dcapw...@gmail.com>
Subject Re: dynamic hostnames for sinks
Date Wed, 15 Aug 2012 00:28:01 GMT
What I did was modified the agent to register itself with ZK if its source
listens to a port.  I then have it setup so that the the agent will check
ZK if the sink defines a config called "cluster".  This is used to discover
the host to talk to.  If the sink defines cluster, the host/port will be
replaced.

This was all done in the agent itself so i could inherit all source/sinks
out there.

I was thinking of forking the FailoverSinkProcessor so on failover discover
a new host to talk to and create a new sink.


On Mon, Aug 13, 2012 at 1:11 PM, Ralph Goers <ralph.goers@dslextreme.com>wrote:

> I'd be interested in knowing how you dynamically located the new host.  I
> would also imagine you could create a "smart" sink that could failover
> itself.
>
> Ralph
>
> On Aug 13, 2012, at 11:33 AM, David Capwell wrote:
>
> > I am looking at flume for data ingestion on AWS.  I was wondering how
> people handle dynamic hostnames in flume?
> >
> > I have created my own ConfigurationProvider and looking to mimic
> FailoverSinkProcessor but on failover to dynamically find a new host to
> talk to.
> >
> > Is there a better route to go than the one I went down?
> >
> > Thanks for your time reading this email.
>
>

Mime
View raw message