flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: Flume Service Takes Lot of Memory
Date Thu, 17 Jan 2013 10:03:59 GMT
No, this will work for the master only. To pass this to agents you have to tweak the startup
script after line 104 ($FLUME_HOME/bin/flume):
https://github.com/cloudera/flume/blob/master/bin/flume#L104

JOPTS="$JOPTS -Xms1g -Xmx2g " 

as example.

- Alex

On Jan 17, 2013, at 10:40 AM, "Kumar, Deepak8" <deepak8.kumar@citi.com> wrote:

> Hi Alex,
> The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG &
MASTERI_WATCHDOG at flume master will work for agent memory as well? Or do we have some other
option as flume agent?
> 
> Regards,
> Deepak
> 
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com] 
> Sent: Thursday, January 17, 2013 2:18 AM
> To: user@flume.apache.org
> Subject: Re: Flume Service Takes Lot of Memory
> 
> Hi,
> 
> To add JVM options in flume we have to edit the watchdog line in bin/flume into: 
> MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID
$WATCHDOG_CLASS" 
> MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID
$WATCHDOG_CLASS"
> 
> and restart Flume master.
> 
> flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR
and FLUME_IDENT_STRING
> 
> Ensure that you have provided the jvm with sufficient heap space. By default Flume starts
the jvm with it's default heap allocation, which differs depending on the jvm version, the
host type (os, 32/64 bit, etc...), total host memory available, as well as other issues.
> 
> The environment variable UOPTS can be used to pass additional jvm parameters when running
Flume. e.g.
> 
> $ UOPTS="-Xms1g -Xmx2g" bin/flume node
> 
> which starts a flume node with an initial heap of one gig and a max heap of two gig.
See "java -h" or "java -X" for more details on available jvm options.
> 
> cheers,
> Alex
> 
> On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <deepak8.kumar@citi.com> wrote:
> 
>> Hi Alex,
>> Thanks for assisting. I do have the following settings in flume-env.sh 
>> file export UOPTS="-Xms256m -Xmx512m"
>> 
>> But once I do top, it shows this much virtual memory
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume
 4957 root   
>> 
>> Regards,
>> Deepak
>> 
>> -----Original Message-----
>> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
>> Sent: Thursday, January 17, 2013 1:58 AM
>> To: user@flume.apache.org
>> Subject: Re: Flume Service Takes Lot of Memory
>> 
>> Hi Deepak,
>> 
>> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx
params:
>> 
>> UOPTS="-Xms1g -Xmx2g" bin/flume node
>> 
>> Of course you can also tweak the bin/flume script.
>> 
>> - Alex
>> 
>> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <deepak8.kumar@citi.com> wrote:
>> 
>>> Hi,
>>> Could you please guide?
>>> 
>>> Regards,
>>> Deepak
>>> 
>>> -----Original Message-----
>>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>>> Sent: Wednesday, January 16, 2013 1:29 PM
>>> To: cdh-user@cloudera.org
>>> Subject: Flume Service Takes Lot of Memory
>>> 
>>> Hi,
>>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory.
Is it uses this much normally or do we have some configuration settings?  
>>> 
>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume
 4957 root   
>>> 
>>> Regards,
>>> Deepak
>> 
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>> 
> 
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> 

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message