hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Will <stefan.w...@gmx.net>
Subject Re: How to do load control of MapReduce
Date Tue, 12 May 2009 15:25:22 GMT
Yes, I think the JVM uses way more memory than just its heap. Now some of it
might be just reserved memory, but not actually used (not sure how to tell
the difference). There are also things like thread stacks, jit compiler
cache, direct nio byte buffers etc. that take up process space outside of
the Java heap. But none of that should imho add up to Gigabytes...

-- Stefan 


> From: zsongbo <zsongbo@gmail.com>
> Reply-To: <core-user@hadoop.apache.org>
> Date: Tue, 12 May 2009 20:06:37 +0800
> To: <core-user@hadoop.apache.org>
> Subject: Re: How to do load control of MapReduce
> 
> Yes, I also found that the TaskTracker should not use so much memory.
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 
> 32480 schubert  35  10 1411m 172m 9212 S    0  2.2   8:54.78 java
> 
> The previous 1GB is the default value, I have just change the heap of TT to
> 384MB one hours ago.
> 
> I also found DataNode also need not too much memory.
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 32399 schubert  25   0 1638m 372m 9208 S    2  4.7  32:46.28 java
> 
> 
> In fact, I define the -Xmx512m in child opt for MapReduce tasks. But I found
> the child task use more memory than the definition:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 
> 10577 schubert  30  10  942m 572m 9092 S   46  7.2  51:02.21 java
> 
> 10507 schubert  29  10  878m 570m 9092 S   48  7.1  50:49.52 java
> 
> Schubert
> 
> On Tue, May 12, 2009 at 6:53 PM, Steve Loughran <stevel@apache.org> wrote:
> 
>> zsongbo wrote:
>> 
>>> Hi Stefan,
>>> Yes, the 'nice' cannot resolve this problem.
>>> 
>>> Now, in my cluster, there are 8GB of RAM. My java heap configuration is:
>>> 
>>> HDFS DataNode : 1GB
>>> HBase-RegionServer: 1.5GB
>>> MR-TaskTracker: 1GB
>>> MR-child: 512MB   (max child task is 6, 4 map task + 2 reduce task)
>>> 
>>> But the memory usage is still tight.
>>> 
>> 
>> does TT need to be so big if you are running all your work in external VMs?
>> 



Mime
View raw message