chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CHUKWA-707) Replace Chukwa collector with Apache Kafka
Date Tue, 29 Jul 2014 19:24:39 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

shreyas subramanya updated CHUKWA-707:
--------------------------------------

    Attachment: CHUKWA-707.patch2

I have modified the patch to create topics as CHUKWA-datatype-host. Each chukwa agent consumer
thread will only subscribe to topics matching its own host according to the regex CHUKWA-(.*)-currentHost.


The messages will have timestamp as their key. This design will also ensure that any streaming
software can perform computation on a datatype stream by subscribing to topics CHUKWA-datatype-.*

Currently the topic name and key format is defined by KafkaKeyFormatter class. This can be
made configurable in future when there is a need for it.



> Replace Chukwa collector with Apache Kafka
> ------------------------------------------
>
>                 Key: CHUKWA-707
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-707
>             Project: Chukwa
>          Issue Type: New Feature
>            Reporter: Eric Yang
>            Assignee: shreyas subramanya
>         Attachments: CHUKWA-707.patch1, CHUKWA-707.patch2
>
>
> Chukwa collector has stopped evolving since 2010.  Newer framework has offer better features
of message queues, and Apache Kafka looks like a good replacement for Chukwa collector.
> Chukwa agent can implement a connector to Apache Kafka to replace Chukwa collector, and
HBase consumer to write data to HBase.  HICC REST API change to new HBase storage format.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message