flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Re: Performance on Widows vs *NIX
Date Mon, 04 Aug 2014 21:57:37 GMT
The one we are using is the bzip2 codec. That is something we could test.

On Monday, August 4, 2014, Roshan Naik <roshan@hortonworks.com> wrote:

> One of the hdfs compression codecs had a memory leak. dont recall which
> right now. if you are using one, try changing to a diff codec and see if
> the leak goes away.
> -roshan
>
>
> On Sat, Aug 2, 2014 at 6:19 AM, Christopher Shannon <cshannon108@gmail.com
> <javascript:_e(%7B%7D,'cvml','cshannon108@gmail.com');>> wrote:
>
>> I do want to add that the Windows agents in our configuration were
>> upstream from the agent using the HDFS sink, and the upstream agents would
>> not recover gracefully when the downstream agent disconnected them on
>> encountering an out of memory error.
>>
>>
>> On Sat, Aug 2, 2014 at 8:18 AM, Christopher Shannon <
>> cshannon108@gmail.com
>> <javascript:_e(%7B%7D,'cvml','cshannon108@gmail.com');>> wrote:
>>
>>> Roshan,
>>>
>>> After more testing, it became plain that the Windows environment was not
>>> the problem. The simultaneous JVMs were behaving well in the Windows
>>> environment. There does appear to be a documented memory leak problem with
>>> the HDFS sink for version 1.3.0. Our distro comes from IBM BigInsights, and
>>> yes, there do appear to be some differences between some dependencies from
>>> the open source and what comes with the IBM distro. So we are looking into
>>> working with them to fix the problem.
>>>
>>> In the meantime, if you can think of any threads on the list pointing to
>>> HDFS sink and out of memory errors, I would be grateful if some can be sent
>>> my way.
>>>
>>> All the best,
>>>
>>> Chris.
>>>
>>>
>>>
>>> On Fri, Aug 1, 2014 at 6:50 PM, Roshan Naik <roshan@hortonworks.com
>>> <javascript:_e(%7B%7D,'cvml','roshan@hortonworks.com');>> wrote:
>>>
>>>> Is that a custom Flume build or from some distro ? Hard to say without
>>>> additional info. anything interesting in  the logs when it crashes ?
>>>>
>>>>
>>>> On Tue, Jul 29, 2014 at 10:15 AM, Christopher Shannon <
>>>> cshannon108@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','cshannon108@gmail.com');>> wrote:
>>>>
>>>>> For development and testing, I sometimes have to run multiple agents
>>>>> on the same server / workstation. Recently I had to load test a second
>>>>> agent in a Windows environment with a near identical configuration (except
>>>>> for paths and more memory allocated to the JVM), but the agent in the
>>>>> second JVM soon ground to a halt whereas the first JVM with less memory
>>>>> worked without slowdown.
>>>>>
>>>>> The total physicsl memory on the machine is 32 gb, and the total
>>>>> physical RAM used never exceds 14GB. Disk space is plentiful. Agent A
>>>>> (started first) has a max JVM heap space of 4 gb, the second JVM (Agent
B)
>>>>> has 8 gb allocated. Both use a spooling file source, a memory channel,
and
>>>>> an Avro sink.
>>>>>
>>>>> Yet agent B is the one that slows down and evetually hangs. I have
>>>>> stood up multiple Flume agents on Linux without this problem.
>>>>>
>>>>> What could account for the degrading performance in a Windows
>>>>> environment for the second agent but not the first?
>>>>>
>>>>> All the best, Chris
>>>>>
>>>>> p.s. The agent version is 1.3.0, and I can't change it for contractual
>>>>> reasons.
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message