chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-674) Integrate Chukwa collector feature to Chukwa agent
Date Thu, 26 Jun 2014 02:43:24 GMT


Eric Yang commented on CHUKWA-674:


SeqFileWriter was built prior to introduction of PipelinableWriter.  This is the reason that
it can not write and blocks the chunk to be passed to the next writer.  If the configuration
is done with SeqFileWriter being last, it will work fine.  In the event, if the writer fails
for bad data, the chunk can be dropped.  In the event that writer failed due to down stream
unavailability, then the same chunk can be retried.  It is possible to have duplicated data
this way, and the sequence id helps to eliminate the duplication.  Hence, this should be working
as designed.

> Integrate Chukwa collector feature to Chukwa agent
> --------------------------------------------------
>                 Key: CHUKWA-674
>                 URL:
>             Project: Chukwa
>          Issue Type: Improvement
>          Components: Data Collection
>         Environment: MacOSX, Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: CHUKWA-674.patch
> Feature offered in Chukwa collector can be integrated into Chukwa agent, and use multi-tier
Chukwa agent to collect data for large scale cluster.  For small cluster, agents can talk
directly to HDFS cluster to reduce the complexity of deployment.  The required features to
reduce the need of Chukwa collectors are: 
> - Enhance agent rest api to receive chunk data.
> - Pipeline writer to channel data to storage destinations (HDFS, HBASE).
> - Improve connector interface and replace http connector with collector connector for
bandwidth balance.

This message was sent by Atlassian JIRA

View raw message