flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Deepak8 " <deepak8.ku...@citi.com>
Subject RE: Flume Service Takes Lot of Memory
Date Thu, 17 Jan 2013 09:40:13 GMT
Hi Alex,
The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG & MASTERI_WATCHDOG
at flume master will work for agent memory as well? Or do we have some other option as flume
agent?

Regards,
Deepak

-----Original Message-----
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com] 
Sent: Thursday, January 17, 2013 2:18 AM
To: user@flume.apache.org
Subject: Re: Flume Service Takes Lot of Memory

Hi,

To add JVM options in flume we have to edit the watchdog line in bin/flume into: 
MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID
$WATCHDOG_CLASS" 
MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID
$WATCHDOG_CLASS"

and restart Flume master.

flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and
FLUME_IDENT_STRING

Ensure that you have provided the jvm with sufficient heap space. By default Flume starts
the jvm with it's default heap allocation, which differs depending on the jvm version, the
host type (os, 32/64 bit, etc...), total host memory available, as well as other issues.

The environment variable UOPTS can be used to pass additional jvm parameters when running
Flume. e.g.

$ UOPTS="-Xms1g -Xmx2g" bin/flume node

which starts a flume node with an initial heap of one gig and a max heap of two gig. See "java
-h" or "java -X" for more details on available jvm options.

cheers,
 Alex

On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <deepak8.kumar@citi.com> wrote:

> Hi Alex,
> Thanks for assisting. I do have the following settings in flume-env.sh 
> file export UOPTS="-Xms256m -Xmx512m"
> 
> But once I do top, it shows this much virtual memory
> 
> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume
 4957 root   
> 
> Regards,
> Deepak
> 
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> Sent: Thursday, January 17, 2013 1:58 AM
> To: user@flume.apache.org
> Subject: Re: Flume Service Takes Lot of Memory
> 
> Hi Deepak,
> 
> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
> 
> UOPTS="-Xms1g -Xmx2g" bin/flume node
> 
> Of course you can also tweak the bin/flume script.
> 
> - Alex
> 
> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <deepak8.kumar@citi.com> wrote:
> 
>> Hi,
>> Could you please guide?
>> 
>> Regards,
>> Deepak
>> 
>> -----Original Message-----
>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>> Sent: Wednesday, January 16, 2013 1:29 PM
>> To: cdh-user@cloudera.org
>> Subject: Flume Service Takes Lot of Memory
>> 
>> Hi,
>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory.
Is it uses this much normally or do we have some configuration settings?  
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume
 4957 root   
>> 
>> Regards,
>> Deepak
> 
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> 

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message