flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Goers <ralph.go...@dslextreme.com>
Subject Re: Flume-NG Channels
Date Tue, 10 Jan 2012 12:31:01 GMT
When you speak of flow isolation are you doing that for security, failure protection or for
some other reason?  From a failure protection case you would need physically different Flume
agents, not just channels. I'm not sure what the security gains are in isolation, if any.

I guess to give you a proper response I would want to know what your actual requirements are
and possibly why. 

For what its worth, I also work in a multi-tenant environment and this has never been a requirement.


Ralph



On Jan 10, 2012, at 12:42 AM, Praveen Ramachandra wrote:

> Hi arvind,
> 
> Thanks for responding.
> 
> if we want to model separation not only in transit but also at rest i.e., if channel
has a filechannel/jdbcchannel/memorychannel backing separation is required when data resides
in those channels before they are shipped to the next hop.
> 
> on multi-tenant, I was trying to figure out from isolation perspective. Flow isolation
is required from one collecting agent tier, to aggregating agent-tier and a tier that is going
to deposit/deliver the events.
> 
> "How do you propose the platform be modified in order to support this use-case?" you
ask, Thinking out loud now :-).
> One option is to have a notion of a flow that is visible at flume-ng level, applications
will map channels to flows and sources/sinks across agent tiers, can mux/demux it appropriately.
> 
> This will also decouple mapping across agent tiers i.e., 
> 
> If you smell scribe in my above description, I wouldn't hold it against you :-). Honestly
the simplicity of scribe let us prototype for our use case in a matter of hour or two, compared
to many days that it took to get almost similar thing prototyped with flume. We even struggle
today to model the use cases seamlessly in flume (og or ng).
> 
> 
> --
> Regards,
> Praveen Ramachandra
> 
> 
> 
> 
> From: Arvind Prabhakar <arvind@apache.org>
> To: flume-user@incubator.apache.org; Praveen Ramachandra <praveen_ramachandra@yahoo.com>

> Sent: Monday, January 9, 2012 11:15 PM
> Subject: Re: Flume-NG Channels
> 
> Hi Praveen,
> 
> First to your question:
> 
> > Did I get the modeling right with flume-ng
> 
> More-or-less yes. The one distinction that I would like to point out
> is that having separate source-sink end points for individual channels
> is stemming more from your requirement than by design of flume. A
> channel in flume implementation does not care how many sources write
> to it or how many sink's read from it.
> 
> > 2. Is there a better way to do it at a platform level
> >             2.1 I know if I can write a bunch of custom sinks/sources and
> > embed a notion of channel to which each events belong to in the message, I
> > can effectively mux and demux the events at either ends.
> 
> The key issue here is the layering of a multi-tenant semantic on top
> of flows. Since fundamentally flume is not aware of the contents of
> the events in a flow, and does not expose any client auth/id model -
> there is no inherent support of doing this out of the box.
> 
> Moreover, from your description it seems that the channels that
> logically separate out the flows will operate within the same agent.
> If that is the case, then it may be a better option to use a single
> channel and have a multiplexing terminal sink that can route the
> messages to the correct destination.
> 
> >             2.2 Which means the default support for channel is also not of
> > much use
> 
> How do you propose the platform be modified in order to support this use-case?
> 
> Thanks,
> Arvind
> 
> 
> 
> On Mon, Jan 9, 2012 at 9:36 PM, Praveen Ramachandra
> <praveen_ramachandra@yahoo.com> wrote:
> > They are in low 100's in the best case scenario, and could be in 1000 in the
> > worst case scenario.
> >
> > I believe this aspect can be pretty much shielded from application if the
> > underlying platform has the right set of responsibilities.
> >
> >
> > --
> > Regards,
> > Praveen Ramachandra
> >
> >
> >
> > ________________________________
> > From: Ralph Goers <ralph.goers@dslextreme.com>
> > To: flume-user@incubator.apache.org
> > Sent: Monday, January 9, 2012 6:53 PM
> > Subject: Re: Flume-NG Channels
> >
> >
> > On Jan 9, 2012, at 2:28 AM, Praveen Ramachandra wrote:
> >
> > Hi,
> >
> > We were trying to design a multi-tenanted system using flume-ng, where each
> > logically independent data set is modelled through a channel going through
> > the system of collectors, aggregators and delivery agents (to end
> > destination). Each channel will carry data that logically belong together.
> > The requirement is that we should be able to bring up and tear down a
> > channel with ease.
> >
> >
> > When we completed the exercise, it turned out that we have to run a separate
> > Source/Sink, at a designated host/port combination for each channel. The
> > issue with this is that, it is an operational overhead that we have work
> > with net-ops to punch holes in the firewall to let tcp traffic flow on
> > non-standard ports. I would imagine that it would be the case in many
> > organizations as well.
> >
> > Two questions.
> >
> > 1. Did I get the modeling right with flume-ng
> > 2. Is there a better way to do it at a platform level
> >             2.1 I know if I can write a bunch of custom sinks/sources and
> > embed a notion of channel to which each events belong to in the message, I
> > can effectively mux and demux the events at either ends.
> >             2.2 Which means the default support for channel is also not of
> > much use
> >
> >
> > What is your target destination(s) for the tenants?  Can they all flow
> > through a single channel in Flume and then be delivered to the correct
> > destination by a smarter sink at the end?
> >
> > Ralph
> >
> >
> 
> 


Mime
View raw message