hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nico Pappagianis <nico.pappagia...@salesforce.com>
Subject Physical memory (bytes) snapshot counter question - how to get maximum memory used in reduce task
Date Thu, 06 Apr 2017 01:15:49 GMT
Hi all

I've made some memory optimizations on the reduce task and I would like to
compare the old reducer vs new reducer in terms of maximum memory
consumption.

I have a question regarding the description of the following counter:

PHYSICAL_MEMORY_BYTES | Physical memory (bytes) snapshot | Total physical
memory used by all tasks including spilled data.

I'm assuming this means the aggregate of memory used throughout the entire
reduce task (if viewing at the reduce task-level).
Please correct me if I'm wrong on this assumption (the description seems
pretty straightforward).

Is there a way to get the maximum (not total) memory used by a reduce task
from the default counters?

Thanks!

Mime
View raw message