chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CHUKWA-670) Use ZooKeeper for collector list and adaptor check list
Date Sat, 01 Dec 2012 23:57:58 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508109#comment-13508109
] 

Eric Yang commented on CHUKWA-670:
----------------------------------

ZooKeeper has some nice properties that can be used for coordination.  The current design
requires to use puppet or chef to make changes to collector list, and issue refresh collector
list command.  It would be nice to have ZooKeeper watcher node for agent to get notified and
reshuffle the list, without the 2 step process.

When a collector is started, it automatically register itself into the ZooKeeper list as an
ephemeral node.  Agent who is watching the list, get notified automatically, and will take
advantage of the new list.  The centralized list helps to refresh the collector list more
easily.

Second use case, ZooKeeper configuration will enable possibilities to manages pipeline writer
in collector to channel data to different destination.

                
> Use ZooKeeper for collector list and adaptor check list
> -------------------------------------------------------
>
>                 Key: CHUKWA-670
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-670
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: Data Collection
>         Environment: Java, ZooKeeper, Chukwa
>            Reporter: Eric Yang
>
> Agent is currently using local disk to storage check point file, and collector list.
 This can be hard to manage for large scale system that needs to be updated more frequently.
 The proposal is to have a new configuration in chukwa agent configuration file to enable
ZooKeeper as the location to store check point file and collector list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message