hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh Chouraria" <qwertyman...@gmail.com>
Subject Re: Review Request: Add load average support to TaskTrackerStatus
Date Thu, 04 Nov 2010 17:27:17 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated 2010-11-04 10:27:17.486796)

Review request for hadoop-mapreduce and Dhruba Borthakur.


Impl'd the ResourceCalculatorPlugin's getSystemLoadAverage method (previously abstract) to
use an OS-independent function to retrieve the last minute load average.

For the Linux plugin, we still read /proc/loadavg.


Extend TaskTrackerStatus.ResourceStatus and ResourceCalculatorPlugin to allow for sending
last-one-minute load average reports.

Here's a link on load and how they could help with scheduling: http://en.wikipedia.org/wiki/Load_(computing)

This addresses bug MAPREDUCE-2170.

Diffs (updated)

  /trunk/src/java/org/apache/hadoop/mapred/TaskTrackerStatus.java 1031069 
  /trunk/src/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java 1031069

  /trunk/src/java/org/apache/hadoop/mapreduce/util/ResourceCalculatorPlugin.java 1031069 
  /trunk/src/test/mapred/org/apache/hadoop/mapred/DummyResourceCalculatorPlugin.java 1031069

  /trunk/src/test/mapred/org/apache/hadoop/mapred/TestTTResourceReporting.java 1031069 

Diff: https://reviews.apache.org/r/20/diff


/proc/loadavg is available on all GNU/Linux systems.

Post editing and recompiling, "ant clean; ant test" passes.

I've added a test case for the Linux plugin, and also extended the existing TT Reporting test
to beam out and receive load averages. Please let me know if I need to add any more tests.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message