flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sammer <esam...@cloudera.com>
Subject Re: Do I have to use 2 collector ports if I have 2 different flows?
Date Wed, 09 May 2012 20:59:20 GMT

It's been a while since I've looked at OG, to be perfectly honest, so take
this with a grain of salt. If I recall, you do need to use separate ports.
Flow names were primarily used to detect where separation was necessary in
autoChains. In other words, if you properly labeled flows and used
autoChains, we were able to figure out that flow A and B can't be sent to
the same group of collectors. When not using autoChains (and few people
could due to some of the limitations) these labels don't mean much.
Separate ports were required so the internals could be kept relatively
simple (e.g. a source need not make routing decisions except in highly
customized cases).

Hope this helps.

On Wed, May 9, 2012 at 9:59 AM, Steve Hoffman <steveh@goofy.net> wrote:

> I had setup 1 flow using flow isolation from several nodes to a single
> collector (default port 35853).  Things have been running just fine.
> When I added a second flow to the same collector, not only did the
> second collector sink not receive any data, but the first went into
> ERROR state (I assume because the first flow uses a different format
> than the first and isn't written to deal with the second flow's
> format).
> When I moved the second flow-id to another port (35854) my data came
> thru which makes me believe you can't send different flow-ids to the
> same collector port.  That would seem very odd.  After all, that is
> the point of the flow identifier -- if I had to have a separate port
> for every flow id, why bother with flow names?
> This is cloudera cdh3u3 flume (not NG).  And before you suggest moving
> to NG, not an option at this time.
> Thanks,
> Steve

Eric Sammer
twitter: esammer
data: www.cloudera.com

View raw message