incubator-flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-865) Implement failover sink
Date Tue, 21 Feb 2012 04:15:36 GMT

    [ https://issues.apache.org/jira/browse/FLUME-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212341#comment-13212341
] 

jiraposter@reviews.apache.org commented on FLUME-865:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3750/#review5232
-----------------------------------------------------------

Ship it!


+1. Please attach the patch to the Jira. And thanks for pulling through this patiently!

- Arvind


On 2012-02-20 06:31:35, Juhani Connolly wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3750/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-02-20 06:31:35)
bq.  
bq.  
bq.  Review request for Flume.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  This includes the changes from FLUME-945 and is thus subject to change with it. I have
a separate patch for the diff from 945->865 if someone wants it.
bq.  
bq.  A failover sink runner is added, along with a SinkRunnerFactory. The configuration reading
is modified so as to read the runner configuration and pass it to the SinkRunnerFactory.
bq.  A failover sink is used by setting the runner type to failover for every participating
sink. In addition each sink assigned to a runner must set the same runner.name and a unique
runner.priority
bq.  The runner will poll only the highest priority live sink. Should it fail by throwing
EventDeliveryException, it will throw it into a pile of dead sinks. Only once all sinks are
exhausted are the dead sinks revived.
bq.  
bq.  Some issues that are outstanding
bq.  - Not all sinks throw EventDeliveryException, or have a clear time when they should be
considered "dead". Coupling such sinks with the failover runner will not result in failover
bq.  - Sinks do not have a clear mechanism to test for their liveliness, other than perhaps
polling LifecycleState. However for most sinks lifecyclestate tends to remain in START even
if the sink is unable to do anything(e.g. avro sink failing to connect)
bq.  
bq.  
bq.  This addresses bug FLUME-865.
bq.      https://issues.apache.org/jira/browse/FLUME-865
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java PRE-CREATION 
bq.    flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java PRE-CREATION 
bq.    flume-ng-core/src/main/java/org/apache/flume/SinkRunner.java 074aab3 
bq.    flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java PRE-CREATION

bq.    flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java PRE-CREATION

bq.    flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java PRE-CREATION 
bq.    flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java PRE-CREATION

bq.    flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java PRE-CREATION

bq.    flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
1ee1f8e 
bq.    flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
3d0e366 
bq.    flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
34d6010 
bq.    flume-ng-node/src/test/resources/flume-conf.properties eebf03d 
bq.  
bq.  Diff: https://reviews.apache.org/r/3750/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  All unit tests pass.
bq.  A new unit test was added. It creates a memory channel and a failover runner with 3 sinks
that consume a preset number of events then send EventDeliveryException. Events are fed to
the channel and assertions are made that the runner is failing over to the correct sinks.
This test also passes.
bq.  
bq.  I also tried to run this on a real cluster, with one master source using a failover runner
to three avro sinks which would each fed into separate agents which I would kill off to test
the failover. Unfortunately because of AvroSink not throwing EventDeliveryException, this
test could not be completed succesfully, and I felt it was beyond the scope of this ticket
to modify AvroSink
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Juhani
bq.  
bq.


                
> Implement failover sink 
> ------------------------
>
>                 Key: FLUME-865
>                 URL: https://issues.apache.org/jira/browse/FLUME-865
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>    Affects Versions: NG alpha 2
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Juhani Connolly
>             Fix For: v1.1.0
>
>         Attachments: FLUME-865.4.patch
>
>
> It would be nice if the flume-ng would have ability to failover to different sink in
case that the active one is not responding (e.g. before failing the transaction).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message