hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sultan Alamro <sultan.ala...@gmail.com>
Subject Re: Physical memory (bytes) snapshot counter question - how to get maximum memory used in reduce task
Date Thu, 06 Apr 2017 01:48:21 GMT
Hi Nico,

Did you check the jhist file?
It has all details about each task.

Best,
Sultan

> On Apr 5, 2017, at 9:15 PM, Nico Pappagianis <nico.pappagianis@salesforce.com>
wrote:
> 
> Hi all
> 
> I've made some memory optimizations on the reduce task and I would like to compare the
old reducer vs new reducer in terms of maximum memory consumption.
> 
> I have a question regarding the description of the following counter:
> 
> PHYSICAL_MEMORY_BYTES | Physical memory (bytes) snapshot | Total physical memory used
by all tasks including spilled data.
> 
> I'm assuming this means the aggregate of memory used throughout the entire reduce task
(if viewing at the reduce task-level). 
> Please correct me if I'm wrong on this assumption (the description seems pretty straightforward).
> 
> Is there a way to get the maximum (not total) memory used by a reduce task from the default
counters?
> 
> Thanks!
> 
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message