Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 16 Sep 2011 17:45:10 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: 
 <1581053976.35575.1316195110722.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <572879018.28381.1316035449242.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HIVE-2446) Introduction of client statistics
 publishers possibility
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HIVE-2446?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13106=
625#comment-13106625 ]=20

jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/#review1935
-----------------------------------------------------------

Ship it!


- Ning


On 2011-09-16 17:35:58, Robert Sur=C3=B3wka wrote:
bq. =20
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1909/
bq.  -----------------------------------------------------------
bq. =20
bq.  (Updated 2011-09-16 17:35:58)
bq. =20
bq. =20
bq.  Review request for hive and Ning Zhang.
bq. =20
bq. =20
bq.  Summary
bq.  -------
bq. =20
bq.  The purpose of this change is to allow publication or storage of count=
ers while the job is running.
bq. =20
bq.  Introduced two new variables to hive-default.xml and HiveConf.java: "h=
ive.client.stats.publishers" and "hive.client.stats.counters". First one sp=
ecifies classes names, whose instances will be executed by HadoopJobExecHel=
per.java (similarly as hooks are) in its method progress(ExecDriverTaskHand=
le): MapRedStats. Second one specifies list of counters that any client sta=
t publishers should publish or stored. Details regarding format of this lis=
t is up to a specific deployment (it is up to client stats publishers to pa=
rse it), yet it is required to use display names of counter groups and coun=
ters.
bq. =20
bq.  Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.sta=
ts package, that must be implemented by all stats publishers.
bq. =20
bq.  Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJ=
obExecHelper.java that puts counters' values to a Java map and then execute=
s registered client stats publishers giving them that map and running job i=
d. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Co=
unters) and getClientStatsPublishers() that are used by code from previous =
sentence.
bq. =20
bq.  Made cosmetic changes in one other class
bq. =20
bq. =20
bq.  This addresses bug HIVE-2446.
bq.      https://issues.apache.org/jira/browse/HIVE-2446
bq. =20
bq. =20
bq.  Diffs
bq.  -----
bq. =20
bq.    trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171=
283=20
bq.    trunk/conf/hive-default.xml 1171283=20
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper=
.java 1171283=20
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java=
 1171283=20
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublish=
er.java PRE-CREATION=20
bq. =20
bq.  Diff: https://reviews.apache.org/r/1909/diff
bq. =20
bq. =20
bq.  Testing
bq.  -------
bq. =20
bq.  Run some random tests, and still running the unit tests.
bq. =20
bq. =20
bq.  Thanks,
bq. =20
bq.  Robert
bq. =20
bq.


> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
>                 Key: HIVE-2446
>                 URL: https://issues.apache.org/jira/browse/HIVE-2446
>             Project: Hive
>          Issue Type: Improvement
>          Components: Clients, Statistics
>            Reporter: Robert Sur=C3=B3wka
>            Assignee: Robert Sur=C3=B3wka
>            Priority: Minor
>         Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.pa=
tch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters=
 while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive=
.client.stats.publishers" and "hive.client.stats.counters". First one speci=
fies classes names, whose instances will be executed by HadoopJobExecHelper=
.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle)=
: MapRedStats. Second one specifies list of counters that any client stat p=
ublishers should publish or stored. Details regarding format of this list i=
s up to a specific deployment (it is up to client stats publishers to parse=
 it), yet it is required to use display names of counter groups and counter=
s.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats =
package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobE=
xecHelper.java that puts counters' values to a Java map and then executes r=
egistered client stats publishers giving them that map and running job id. =
Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Count=
ers) and getClientStatsPublishers() that are used by code from previous sen=
tence.
> Made cosmetic changes in two other classes
> =20

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira