chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <>
Subject [jira] Commented: (CHUKWA-369) proposed reliability mechanism
Date Sun, 16 Aug 2009 04:11:14 GMT


Ari Rabkin commented on CHUKWA-369:

Patch has basically five pieces; I'm happy to split them up and commit separately if some
are uncontroversial.

1) Sender and Connector are refactored to allow the HttpClient to be reused, and used more
2) Writers now return an instance of ChukwaWriter.CommitStatus.  This is either OK, Failure,
or Pending. The first two are singletons, the latter includes a list of strings.
3) SeqFileWriter returns a CommitPending on writes.
4) A new servlet, CommitCheckServlet for periodically scanning HDFS.
5) A new Sender, the AsyncAckSender, that doesn't automatically commit, but only does so when
it either receives an OK, or else after a pending commit becomes stable.  The Sender periodically
asks a CommitCheckServlet what's been committed.

I think (1), and possibly (2+3) may make sense even without 4 and 5, which are the bits that
I think need serious testing before we should even discuss committing them.

> proposed reliability mechanism
> ------------------------------
>                 Key: CHUKWA-369
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.3.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.3.0
>         Attachments: delayedAcks.patch
> We like to say that Chukwa is a system for reliable log collection. It isn't, quite,
since we don't handle collector crashes.  Here's a proposed reliability mechanism.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message