flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: flume failover only support two nodes?
Date Thu, 14 Aug 2014 03:55:13 GMT
Each sink needs to have a different priority. If multiple sinks have 
same priority, only one of them will be used.

基勇 wrote:
>
> Hello,guys
> I tested flume failover feature found failover support only two nodes,
> may I ask whether it is so?‍
>
> config file:
> storm@storm01:~/apache-flume-1.5.0-bin/conf$ more flume-sink.properties
> #Name the compents on this agent
> a1.sources = r1
> a1.sinks = k1 k2 k3 k4
> a1.channels = c1
>
> #Describe the sinkgroups
> a1.sinkgroups = g1 g2
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 10
> a1.sinkgroups.g1.processor.priority.k2 = 5
> a1.sinkgroups.g1.processor.maxpenalty = 10000
>
> #a1.sinkgroups.g2.sinks = k3 k4
> #a1.sinkgroups.g2.processor.type = load_balance
> #a1.sinkgroups.g2.processor.backoff = true
> #a1.sinkgroups.g2.processor.selector = round_robin
>
> #Describe/config the source
> a1.sources.r1.type = syslogtcp
> a1.sources.r1.port = 5140
> a1.sources.r1.host = localhost
> a1.sources.r1.channels = c1
>
> #Describe the sink
> a1.sinks.k1.type = avro
> a1.sinks.k1.channel = c1
> a1.sinks.k1.hostname = 192.168.220.159
> a1.sinks.k1.port = 44411
>
> a1.sinks.k2.type = avro
> a1.sinks.k2.channel = c1
> a1.sinks.k2.hostname = 192.168.220.159
> a1.sinks.k2.port = 44422
>
> a1.sinks.k3.type = avro
> a1.sinks.k3.channel = c1
> a1.sinks.k3.hostname = 192.168.220.159
> a1.sinks.k3.port = 44433
>
> a1.sinks.k4.type = avro
> a1.sinks.k4.channel = c1
> a1.sinks.k4.hostname = 192.168.220.159
> a1.sinks.k4.port = 44444
> #Use a channel which butters events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100‍
>
> only k3 and k2 node received ‍the data.
> i stop k2 k3. k1 k4 can receive the data;
>
> exception infomation:
> 2014-07-08 08:10:06,403
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [ERROR -
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)]
> Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: All sinks failed to process,
> nothing left to failover to
> at
> org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:191)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> 2014-07-08 08:10:11,408
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO -
> org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:206)] 
>
> Rpc sink k2: Building RpcClient with hostname: 192.168.220.159, port:
> 44422
> 2014-07-08 08:10:11,408
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO -
> org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)]
> Attempting to create Avro Rpc client.
> 2014-07-08 08:10:11,408
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [WARN -
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:620)] 
>
> Using default maxIOWorkers
> 2014-07-08 08:10:11,417
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [ERROR -
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)]
> Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: All sinks failed to process,
> nothing left to failover to
> at
> org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:191)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> 2014-07-08 08:10:16,422
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO -
> org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:206)] 
>
> Rpc sink k3: Building RpcClient with hostname: 192.168.220.159, port:
> 44433
> 2014-07-08 08:10:16,423
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO -
> org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)]
> Attempting to create Avro Rpc client.
> 2014-07-08 08:10:16,425
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [WARN -
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:620)] 
>
> Using default maxIOWorkers
> 2014-07-08 08:10:16,431
> (SinkRunner-PollingRunner-FailoverSinkProcessor) [ERROR -
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)]
> Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: All sinks failed to process,
> nothing left to failover to
> at
> org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:191)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)‍

Mime
View raw message