Mailing-List: contact chukwa-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@hadoop.apache.org
Message-ID: <1490508385.1250395874815.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 21:11:14 -0700 (PDT)
From: "Ari Rabkin (JIRA)" <jira@apache.org>
To: chukwa-dev@hadoop.apache.org
Subject: [jira] Commented: (CHUKWA-369) proposed reliability mechanism
In-Reply-To: <2099955847.1249412175025.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CHUKWA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743808#action_12743808 ] 

Ari Rabkin commented on CHUKWA-369:
-----------------------------------

Patch has basically five pieces; I'm happy to split them up and commit separately if some are uncontroversial.

1) Sender and Connector are refactored to allow the HttpClient to be reused, and used more generally.
2) Writers now return an instance of ChukwaWriter.CommitStatus.  This is either OK, Failure, or Pending. The first two are singletons, the latter includes a list of strings.
3) SeqFileWriter returns a CommitPending on writes.
4) A new servlet, CommitCheckServlet for periodically scanning HDFS.
5) A new Sender, the AsyncAckSender, that doesn't automatically commit, but only does so when it either receives an OK, or else after a pending commit becomes stable.  The Sender periodically asks a CommitCheckServlet what's been committed.

I think (1), and possibly (2+3) may make sense even without 4 and 5, which are the bits that I think need serious testing before we should even discuss committing them.

> proposed reliability mechanism
> ------------------------------
>
>                 Key: CHUKWA-369
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-369
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.3.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.3.0
>
>         Attachments: delayedAcks.patch
>
>
> We like to say that Chukwa is a system for reliable log collection. It isn't, quite, since we don't handle collector crashes.  Here's a proposed reliability mechanism.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.