incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <>
Subject Re: Embedding Chukwa agent/adapter?
Date Fri, 02 Apr 2010 18:07:19 GMT
On Fri, Apr 2, 2010 at 10:55 AM, Kirk True <> wrote:
> Hi Ari,
> Thanks for the info.
> So the recommended model is to have my app write out to a directory on local
> storage and on the same machine have an adapter that tails files from that
> same directory and handles the communication to the collector? I can see
> your point about loss of network connectivity - my assumption was that the
> Chukwa adapter buffered/persisted the data internally on network failures.

No.  It wouldn't be too hard to implement store-on-collector-failure
at this point, but it isn't there yet.
There is experimental support in 0.4 and trunk for "pessimistically
write everything to disk."

The preferred model is indeed to write to local files, and then add
adaptors to taste.

> If the above is correct, then my next question is - how would I know when I
> can reclaim the files on the local storage? How do I know when they've been
> sent to the collector and thus can be deleted from local storage so as to
> prevent unbounded growth of the files?

If you connect to the agent and say "list", you can see how  much data
from each adaptor has been committed. If it matches the length of your
file, you can safely deregister the adaptor and delete the file.

Ari Rabkin
UC Berkeley Computer Science Department

View raw message