incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <>
Subject Re: Embedding Chukwa agent/adapter?
Date Fri, 02 Apr 2010 07:49:33 GMT
The benefit of the agent model is that it makes it much easier to do
failure tolerance. Networks go down, but writes to local disk are
about as reliable as anything can be. Also, it means that your app
won't fail, even if you hit a bug in Chukwa, and you won't lose the
collected data, even if your app crashes.

That said, it is certainly possible to embed the guts of the Chukwa
adaptors and agent inside your process.  There's two plausible ways I
see to do it:

Approach 1)
- Instantiate a Connector, the piece that talks to the collector.
- The Connector reads from a static queue; you can get ahold of this
queue via DataFactory.
- Just push Chunks into the queue, and they'll get sent to the collector.

Approach 2)
Instantiate a ChukwaAgent, and then call its add(...) method to start adaptors.

Approach 1 is best if all the data you want is inside your
application; approach 2 is more flexible, but if you go for approach
two, you might just run Chukwa as a normal external process and be
done with it.

The backfilling loader (in class BackfillingLoader) may be worth inspecting


On Thu, Apr 1, 2010 at 11:33 PM, Kirk True <> wrote:
> Hi all,
> We'd like to push data to the collectors straight from our application as it
> receives data from its clients. In a sense we'd like to embed a Chukwa agent
> and our adapters inside our web application. Does this make sense or is this
> approach flawed in some way? Otherwise we'd have to save the data to a file
> in a shared directory that's then picked up by a stand-alone Chukwa agent
> via a 'file tailer' or something. Or do we want to use the UDPAdaptor and
> proxy our data to it, which in turn forwards it to the collector?
> Thanks,
> Kirk

Ari Rabkin
UC Berkeley Computer Science Department

View raw message