chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome Boulon <jbou...@netflix.com>
Subject Re: Embedding Chukwa agent/adapter?
Date Fri, 02 Apr 2010 15:13:15 GMT
Hi Kirk,

I've built an agent less solution for Netflix similar in concept with Chukwa
and also compatible at the dataSink level.
It's basically a log4J appender that you have to use in your application
using standard log4 configuration.

I'm using Thrift as the transport protocol and will add Avro support later.
This setup has been benchmark at 120 Millions events/day using 4 collectors
on ec2 (small instances) and the goal is to collect over 600M events/day by
the end of next month.

The code is not yet available but you can read more here:
http://wiki.github.com/jboulon/Honu/ and this is a presentation I did at the
last Facebook User Group:
http://www.slideshare.net/evamtse/hive-user-group-presentation-from-netflix-
3182010-3483386

Send me an email if you want more information since I don't want to hijack
Chukwa ML.

/Jerome.


On 4/1/10 11:33 PM, "Kirk True" <kirk@mustardgrain.com> wrote:

> Hi all,
> 
> We'd like to push data to the collectors straight from our application
> as it receives data from its clients. In a sense we'd like to embed a
> Chukwa agent and our adapters inside our web application. Does this make
> sense or is this approach flawed in some way? Otherwise we'd have to
> save the data to a file in a shared directory that's then picked up by a
> stand-alone Chukwa agent via a 'file tailer' or something. Or do we want
> to use the UDPAdaptor and proxy our data to it, which in turn forwards
> it to the collector?
> 
> Thanks,
> Kirk
> 


Mime
View raw message