chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <>
Subject [jira] Commented: (CHUKWA-369) proposed reliability mechanism
Date Wed, 05 Aug 2009 16:00:15 GMT


Ari Rabkin commented on CHUKWA-369:

Eric -- 

The difficulty is that with HDFS, there's no easy way to know if data has committed: it can
happen that a call to write() succeeds, but the collector then crashes before the data left
the process.  Flush in 0.21 may help, but I'm not sure we want to flush on every write.  The
purpose of the proposed mechanism is to decouple commit-detection from the collector, and
let the Agent determine on its own if data committed.

> proposed reliability mechanism
> ------------------------------
>                 Key: CHUKWA-369
>                 URL:
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.3.0
>            Reporter: Ari Rabkin
>             Fix For: 0.3.0
> We like to say that Chukwa is a system for reliable log collection. It isn't, quite,
since we don't handle collector crashes.  Here's a proposed reliability mechanism.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message