incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corbin Hoenes <cor...@tynt.com>
Subject Re: SocketTeeWriter
Date Tue, 11 May 2010 02:39:50 GMT
Jerome what is honu?  You also mention a client library? Is that  
something running at the adaptor/agent level?

Sent from my iPhone

On May 10, 2010, at 6:18 PM, Jerome Boulon <jboulon@netflix.com> wrote:

> I may start working soon on a chunk forwarder that will work at the
> collector level exactly to send monitoring metrics to a real time  
> system.
> The requirements are the same as the ones I have for my client  
> library:
> should be able:
> - to filter the data that will be sent over
> - to buffer some chunks in memory
> - to retry/fail over automatically
> - drop on the floor if the data cannot be sent over
>
> I was planning to reuse most of the code that I have built for Honu  
> here at
> Netflix
>
> I've reach my 2d milestone on Honu (sending 700M rows per day) so this
> should be the next thing on my todo list.
>
> - So what is your time range to get something?
> - Is the SocketTee something that is production ready? (I mean able to
> support more than 300M events/day)?
>
>
> Thanks,
>  /Jerome.
>
> On 5/10/10 4:37 PM, "Ariel Rabkin" <asrabkin@gmail.com> wrote:
>
>> That's how we use it at Berkeley, to process metrics from hundreds of
>> machines; total data rate less than a megabyte per second, though.
>> What scale of data are you looking at?
>>
>> The intent of SocketTee was if you need some subset of the data now,
>> while write-to-HDFS-and-process-with-Hadoop is still the default  
>> path.
>> What sort of low-latency processing do you need?
>>
>> --Ari
>>
>> On Mon, May 10, 2010 at 4:28 PM, Corbin Hoenes <corbin@tynt.com>  
>> wrote:
>>> Has anyone used the "Tee" in a larger scale deployment to try to get
>>> real-time/low latency data?  Interested in how feasible it would  
>>> be to use it
>>> to pipe data into another system to handle these low latency  
>>> requests and
>>> leave the long term analysis to hadoop.
>>>
>>>
>>
>>
>

Mime
View raw message