hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <>
Subject Re: Review Request 40318: HIVE-12271 : Add metrics around HS2 query execution and job submission for Hive
Date Mon, 16 Nov 2015 19:21:18 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Nov. 16, 2015, 7:21 p.m.)

Review request for hive.



Bugs: HIVE-12271

Repository: hive-git


Add following metrics:

1.  Size of async thread pool and queue. (in SessionManager)
2.  Number/Duration of all HS2 operations in each state, and count of what state they finished
(in Operation)
3.  Number/Duration of all calls recorded by PerfLogger (in PerfLogger)

In the metrics, cleaned up the definition of MetricsScope, extended it to also record number
of active calls instead of just time.  MetricsScope are stored by framework itself for metastore/PerfLogger
cases, and kept by the code in Operation (due to nature of operations being run on async threads).

Minor note: removed a PerfLogBegin call for task with specific task-id.  It is missing PerfLogEnd
so never recorded anyway, and also seemed not useful as all other PerfLogger calls are logged
with same method name and this one was the outlyer.

Diffs (updated)

  common/pom.xml 2292fdf 
  common/src/java/org/apache/hadoop/hive/common/metrics/ 52d99e4 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/ 49b2b32 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/ a5aa995

  common/src/java/org/apache/hadoop/hive/common/metrics/common/ PRE-CREATION

  common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/ 3db26af

  common/src/java/org/apache/hadoop/hive/ql/log/ 548d7db 
  common/src/test/org/apache/hadoop/hive/common/metrics/ PRE-CREATION

  common/src/test/org/apache/hadoop/hive/common/metrics/ c3e8282 
  itests/hive-unit/pom.xml c202001 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/ PRE-CREATION

  metastore/src/java/org/apache/hadoop/hive/metastore/ 3c40d6e 
  ql/src/java/org/apache/hadoop/hive/ql/ 9a5495b 
  service/pom.xml 22234d9 
  service/src/java/org/apache/hive/service/cli/operation/ a851936 
  service/src/java/org/apache/hive/service/cli/session/ a9b4334 
  service/src/test/org/apache/hive/service/cli/session/ PRE-CREATION



Added two unit tests:  TestHs2Metrics, TestSessionManagerMetrics, test all of these metric
categories in in-flight and post-operation metric counts.


Szehon Ho

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message