chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-369) proposed reliability mechanism
Date Tue, 18 Aug 2009 22:42:14 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744760#action_12744760
] 

Eric Yang commented on CHUKWA-369:
----------------------------------

What is the algorithm used to identify the written bytes from chunk 2 from server a as oppose
to chunk 2 from server b if the data are stacked together?  There doesn't seem to be an easy
way to identify if one of the chunk is missing.

Our deployment has collector mapped to exact number of datanode deployed.  100kb/sec from
25000 machines easily met the 2GB/sec data, Chukwa is designed to process data in parallel
with stateless transport.  We need to ensure that this solution does not impact parallelism.

> proposed reliability mechanism
> ------------------------------
>
>                 Key: CHUKWA-369
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-369
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.3.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.3.0
>
>         Attachments: delayedAcks.patch
>
>
> We like to say that Chukwa is a system for reliable log collection. It isn't, quite,
since we don't handle collector crashes.  Here's a proposed reliability mechanism.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message