flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincentius Martin <vincentiusmar...@gmail.com>
Subject Re: All channels in an agent get slower after a channel is full
Date Thu, 13 Nov 2014 21:01:50 GMT
Right now, I am using FileChannel.

Thanks


Regards,
Vincentius Martin

On Fri, Nov 14, 2014 at 4:00 AM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> Are you using MemoryChannel or File Channel?
>
> Thanks,
> Hari
>
>
> On Thu, Nov 13, 2014 at 12:59 PM, Vincentius Martin <
> vincentiusmartin@gmail.com> wrote:
>
>>  Yes, they are sharing the same disk
>>
>> I used to try it with memory channel, it also produced the same impact
>> when a channel in an agent with many channels reaches its channel capacity.
>> It caused ChannelException and made other channels slower.
>>
>>
>>
>> Regards,
>> Vincentius Martin
>>
>> On Fri, Nov 14, 2014 at 3:47 AM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>> Are all the channels sharing the same disk(s)?
>>>
>>> Thanks,
>>> Hari
>>>
>>>
>>> On Thu, Nov 13, 2014 at 12:44 PM, Vincentius Martin <
>>> vincentiusmartin@gmail.com> wrote:
>>>
>>>>  it is between agents, I am using avro sinks and file channels while
>>>> all of those channels write the checkpoint to a disk.
>>>>
>>>> For the rest, I am using default configuration.
>>>>
>>>>
>>>> Regards,
>>>> Vincentius Martin
>>>>
>>>> On Fri, Nov 14, 2014 at 1:39 AM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>> What does your configuration look like? What sink are you using?
>>>>>
>>>>> On Thu, Nov 13, 2014 at 8:23 AM, Vincentius Martin <
>>>>> vincentiusmartin@gmail.com> wrote:
>>>>>
>>>>>>  Hi,
>>>>>>
>>>>>> In my cluster, I have an agent with one source connected to multiple
>>>>>> channels. Each channel connected to different sink (1 channel paired
with 1
>>>>>> sink) which send events to different agents (like one to many relation).
>>>>>> Just like the multiplexing flow example in Flume user guide website.
>>>>>>
>>>>>> However, when a channel reaches its capacity (already full)  I see
>>>>>> that the agent performance gets slower.
>>>>>>
>>>>>> What I mean by getting slower is that, all other channel-sink pairs
>>>>>> in that agent also get slower when sending events to their destination.
I
>>>>>> can understand if the overfilled channel-sink pair get slower, but
why it
>>>>>> affects another channel-sink pairs in that agent? From what I see
here, the
>>>>>> other pairs should be independent with the overfilled channel except
that
>>>>>> they use the same source, right?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Regards,
>>>>>> Vincentius Martin
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message