Mailing-List: contact dev-help@chukwa.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@chukwa.apache.org
Date: Thu, 26 Jun 2014 02:43:24 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: dev@chukwa.apache.org
Message-ID: <JIRA.12618986.1354769801301.51574.1403750604809@arcas>
In-Reply-To: <JIRA.12618986.1354769801301@arcas>
References: <JIRA.12618986.1354769801301@arcas>
Subject: [jira] [Commented] (CHUKWA-674) Integrate Chukwa collector feature
 to Chukwa agent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CHUKWA-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044290#comment-14044290 ] 

Eric Yang commented on CHUKWA-674:
----------------------------------

Shreyas,

SeqFileWriter was built prior to introduction of PipelinableWriter.  This is the reason that it can not write and blocks the chunk to be passed to the next writer.  If the configuration is done with SeqFileWriter being last, it will work fine.  In the event, if the writer fails for bad data, the chunk can be dropped.  In the event that writer failed due to down stream unavailability, then the same chunk can be retried.  It is possible to have duplicated data this way, and the sequence id helps to eliminate the duplication.  Hence, this should be working as designed.

> Integrate Chukwa collector feature to Chukwa agent
> --------------------------------------------------
>
>                 Key: CHUKWA-674
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-674
>             Project: Chukwa
>          Issue Type: Improvement
>          Components: Data Collection
>         Environment: MacOSX, Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: CHUKWA-674.patch
>
>
> Feature offered in Chukwa collector can be integrated into Chukwa agent, and use multi-tier Chukwa agent to collect data for large scale cluster.  For small cluster, agents can talk directly to HDFS cluster to reduce the complexity of deployment.  The required features to reduce the need of Chukwa collectors are: 
> - Enhance agent rest api to receive chunk data.
> - Pipeline writer to channel data to storage destinations (HDFS, HBASE).
> - Improve connector interface and replace http connector with collector connector for bandwidth balance.


--
This message was sent by Atlassian JIRA
(v6.2#6252)