hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <sze...@cloudera.com>
Subject Re: Review Request 40318: HIVE-12271 : Add metrics around HS2 query execution and job submission for Hive
Date Mon, 16 Nov 2015 19:21:18 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40318/
-----------------------------------------------------------

(Updated Nov. 16, 2015, 7:21 p.m.)


Review request for hive.


Changes
-------

Rebase.


Bugs: HIVE-12271
    https://issues.apache.org/jira/browse/HIVE-12271


Repository: hive-git


Description
-------

Add following metrics:

1.  Size of async thread pool and queue. (in SessionManager)
2.  Number/Duration of all HS2 operations in each state, and count of what state they finished
(in Operation)
3.  Number/Duration of all calls recorded by PerfLogger (in PerfLogger)


In the metrics, cleaned up the definition of MetricsScope, extended it to also record number
of active calls instead of just time.  MetricsScope are stored by framework itself for metastore/PerfLogger
cases, and kept by the code in Operation (due to nature of operations being run on async threads).

Minor note: removed a PerfLogBegin call for task with specific task-id.  It is missing PerfLogEnd
so never recorded anyway, and also seemed not useful as all other PerfLogger calls are logged
with same method name and this one was the outlyer.


Diffs (updated)
-----

  common/pom.xml 2292fdf 
  common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 52d99e4 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 49b2b32 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java a5aa995

  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsScope.java PRE-CREATION

  common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java 3db26af

  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 548d7db 
  common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java PRE-CREATION

  common/src/test/org/apache/hadoop/hive/common/metrics/TestLegacyMetrics.java c3e8282 
  common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
a3aa549 
  itests/hive-unit/pom.xml c202001 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java PRE-CREATION

  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3c40d6e 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9a5495b 
  service/pom.xml 22234d9 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java a851936 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java a9b4334 
  service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java PRE-CREATION


Diff: https://reviews.apache.org/r/40318/diff/


Testing
-------

Added two unit tests:  TestHs2Metrics, TestSessionManagerMetrics, test all of these metric
categories in in-flight and post-operation metric counts.


Thanks,

Szehon Ho


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message