chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-674) Integrate Chukwa collector feature to Chukwa agent
Date Tue, 24 Jun 2014 02:56:26 GMT


shreyas subramanya commented on CHUKWA-674:

Hi Eric,
1. SeqFileWriter is not passing data to next stage of the pipeline
2. Error in one pipeline stage will affect the remaining stages. For example, HBaseWriter
throws exception when HBase is down and this will prevent other writers from seeing these
chunks. So instead of a pipeline, how about we spawn new threads for each pipeline stage and
merge the results? This will also help us in future when we do kafka integration. Each new
writer can be a kafka client thread and perform at its own pace.
3. Since we currently do processing for JMX and other monitoring metrics within the demux
map processors, maybe we should move the demux phase to be called inside the PipelineConnector
rather than in HBaseWriter? This will ensure the same data will be uniformly available to
all writers (for example hbase and alert writers) 

> Integrate Chukwa collector feature to Chukwa agent
> --------------------------------------------------
>                 Key: CHUKWA-674
>                 URL:
>             Project: Chukwa
>          Issue Type: Improvement
>          Components: Data Collection
>         Environment: MacOSX, Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: CHUKWA-674.patch
> Feature offered in Chukwa collector can be integrated into Chukwa agent, and use multi-tier
Chukwa agent to collect data for large scale cluster.  For small cluster, agents can talk
directly to HDFS cluster to reduce the complexity of deployment.  The required features to
reduce the need of Chukwa collectors are: 
> - Enhance agent rest api to receive chunk data.
> - Pipeline writer to channel data to storage destinations (HDFS, HBASE).
> - Improve connector interface and replace http connector with collector connector for
bandwidth balance.

This message was sent by Atlassian JIRA

View raw message