flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bhaskar V. Karambelkar" <bhaska...@gmail.com>
Subject Re: Non blocking IO using Flume client
Date Thu, 29 Jan 2015 15:00:05 GMT
Loic,
One thing you can do is write data to a queue (say Redis) using a non
blocking client, and use a redis source to feed to flume. That'll give you
non blocking push operations on the client side. It does add another
component to the mix however.
We do something similar, and it scales quite well.

On Thu, Jan 29, 2015 at 1:45 AM, Loic Descotte <loic.descotte@gmail.com>
wrote:

> Thanks for the link, I'll take a look.
>
> 2015-01-29 7:43 GMT+01:00 Hari Shreedharan <hshreedharan@cloudera.com>:
>
>> In your case, you'd end up having too many threads - that is right. I
>> don't know if there is anything you can do right now
>>
>> The interesting thing is that the underlying Flume Avro RPC client is
>> actually non-blocking. It is exposed as a blocking client for ease of use.
>> Also, there is a bug in Avro that can cause the client to block during an
>> initial handshake - https://issues.apache.org/jira/browse/AVRO-1122
>>
>> For now, I am not sure what we can do. But if you do happen to have some
>> time, please take a look!
>>
>>
>> Hari
>>
>> On Wed, Jan 28, 2015 at 10:35 PM, Loic Descotte <loic.descotte@gmail.com>
>> wrote:
>>
>>> Of course this kind of problem can always be solved on the server side,
>>> but it would be better be able to handle latency with non blocking IO on
>>> the client. In particular if you are working with asynchronous web
>>> frameworks like Play Framework (or Spray.io) that work better with a low
>>> number of threads.
>>>
>>>
>>
>>
>
>
> --
> Loïc Descotte
> http://about.me/loicd
>

Mime
View raw message