Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
	boundary="===============1823500510440995406=="
MIME-Version: 1.0
Subject: Re: Review Request: Log more Hadoop task counter values in the
 MapRedStats
 object.
From: "Kevin Wilfong" <kevinwilfong@fb.com>
To: "Yongqiang He" <heyongqiangict@gmail.com>,
	"Ramkumar Vadali" <ramkumar.vadali@gmail.com>
Date: Tue, 04 Oct 2011 17:38:18 -0000
Message-ID: <20111004173818.4917.34095@reviews.apache.org>
Cc: "Kevin Wilfong" <kevinwilfong@fb.com>,"hive" <dev@hive.apache.org>
In-Reply-To: <20111004172743.4917.92950@reviews.apache.org>
References: <20111004172743.4917.92950@reviews.apache.org>

--===============1823500510440995406==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2167/
-----------------------------------------------------------

(Updated 2011-10-04 17:38:18.076011)


Review request for hive, Ramkumar Vadali and Yongqiang He.


Changes
-------

I added everything from Task$Counter except CPU_MILLISECONDS because curren=
tly that receives special treatment in HadoopJobExecHelper.


Summary
-------

I added the counters mentioned in the task to the MapRedStats class, and mo=
dified HadoopJobExecHelper to collect them.

I got tired of writing the same code over and over again, so I modified the=
 way MapRedStats and HadoopJobExecHelper treat task counters.  MapRedStats =
now has an enum with all of the task counters we want to collect, it is a s=
ubset of the enum in Task$Counter.  Task is package private so the enum in =
it is unavailable.  MapRedStats now contains a map from the enum values to =
the values of the counters, if they were set.  HadoopJobExecHelper loops ov=
er the enum values and tries to get a value for each counter.  As long as t=
he new getter and setter methods are used the functionality is the same, in=
 particular for the getter, if a counter was set, it returns the value of t=
he counter, otherwise it returns -1.


This addresses bug Hive-2479.
    https://issues.apache.org/jira/browse/Hive-2479


Diffs (updated)
-----

  trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1178612 =

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java=
 1178612 =


Diff: https://reviews.apache.org/r/2167/diff


Testing
-------

I ran some queries to verify the counters were being populated.

I also ran a few of the unit test queries to verify I hadn't broken anythin=
g.


Thanks,

Kevin


--===============1823500510440995406==--