chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CHUKWA-707) Replace Chukwa collector with Apache Kafka
Date Wed, 23 Jul 2014 23:30:39 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072541#comment-14072541
] 

Eric Yang commented on CHUKWA-707:
----------------------------------

Kafka topic is designed to have one consumer per topic.  Kafka topics should be partitioned
by host +  data type.  Otherwise, n consumers are running inside n Chukwa agents, and causes
the same message to be consumed n times and send to writer n times.

> Replace Chukwa collector with Apache Kafka
> ------------------------------------------
>
>                 Key: CHUKWA-707
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-707
>             Project: Chukwa
>          Issue Type: New Feature
>            Reporter: Eric Yang
>            Assignee: shreyas subramanya
>         Attachments: CHUKWA-707.patch1
>
>
> Chukwa collector has stopped evolving since 2010.  Newer framework has offer better features
of message queues, and Apache Kafka looks like a good replacement for Chukwa collector.
> Chukwa agent can implement a connector to Apache Kafka to replace Chukwa collector, and
HBase consumer to write data to HBase.  HICC REST API change to new HBase storage format.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message