flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Ye <denny...@gmail.com>
Subject Re: AvroSink and LoadBalancingRpcClient
Date Thu, 10 Jan 2013 07:55:06 GMT
hi Hari,
    I cannot judge the situation that using method you raised. I would like
to explain my case and need your comments. Thanks a lot!
    What I need is load balancing while event transferring.  Assume that I
have single local Flume server (located with application) named
'localFlumeA', configured with single AvroSink and Channel. Meanwhile, two
central Flume servers (collectors) named 'centralFlumeE' and
'centralFlumeF'. Under this case, I would like to configure load balancing
between 'centralFlumeE' and 'centralFlumeF' for events coming from
'localFlumeA', and load can be dispatched averagely for that two central
Flume servers.
    Can it be configured by LoadBalancingSinkProcessor in your mind? Wish
your advice

Denny Ye

2013/1/10 Hari Shreedharan <hshreedharan@cloudera.com>

>  The LoadBalancing capability similar to the LoadBalancingRpcClient can be
> configured for multiple Avro Sinks using a LoadBalancingSinkProcessor, if
> you are looking for that functionality.
> Hari
> --
> Hari Shreedharan
> On Wednesday, January 9, 2013 at 11:05 PM, Connor Woodson wrote:
> Short answer: there is no way in the current AvroSink to configure the
> RpcClient, limiting you to just a single host connection (I'm not sure how
> well it recovers if that host goes down).
> The AvroSink is incredibly simplified from what the RPCClient can do and
> exposes none of the background functionality. Right now, the only way
> around that is to create a custom sink based off of the AvroSink source
> code and instead of setting the RPCClient up the way it currently is, you
> pass into the RPCClient.getInstance() a set of user supplied properties. To
> implement this in an unsafe way (not checking any of the user's values)
> would only take a couple lines of code I believe. It is a work around, but
> it will enable all of the various RPCClient capabilities such as failover
> or loadbalancing mode and allow it to connect to multiple hosts.
> This is something that (I think) there is a JIRA filed for; but if not, it
> would be very helpful for this to be implemented into the actual AvroSink
> (and something that should be linked to that is RPCClient.getInstance
> accepting a Context object, simply for ease of use).
> - Connor
> On Wed, Jan 9, 2013 at 10:55 PM, Denny Ye <dennyy99@gmail.com> wrote:
> hi all,
>     I didn't find the relationship between AvroSink and other types of
> RpcClient, including LoadBalancingRpcClient. In my opinion, user can set
> the specified RpcClient type from AvroSink with several strategies and host
> selectors. Also, I cannot get information from source code and user guide.
> Did I miss something about this?
>      Wish someone can support, thanks!
> -Regards
> Denny Ye

View raw message