hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <sze...@cloudera.com>
Subject Review Request 40318: HIVE-12271 : Add metrics around HS2 query execution and job submission for Hive
Date Sat, 14 Nov 2015 01:10:50 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40318/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-12271
    https://issues.apache.org/jira/browse/HIVE-12271


Repository: hive-git


Description
-------

Add following metrics:

1.  Size of async thread pool and queue. (in SessionManager)
2.  Number/Duration of all HS2 operations in each state, and count of what state they finished
(in Operation)
3.  Number/Duration of all calls recorded by PerfLogger (in PerfLogger)


In the metrics, cleaned up the definition of MetricsScope, extended it to also record number
of active calls instead of just time.  MetricsScope are stored by framework itself for metastore/PerfLogger
cases, and kept by the code in Operation (due to nature of operations being run on async threads).

Minor note: removed a PerfLogBegin call for task with specific task-id.  It is missing PerfLogEnd
so never recorded anyway, and also seemed not useful as all other PerfLogger calls are logged
with same method name and this one was the outlyer.


Diffs
-----

  common/pom.xml 2292fdf 
  common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 52d99e4 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 49b2b32 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java a5aa995

  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsScope.java PRE-CREATION

  common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java 3db26af

  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 548d7db 
  common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java PRE-CREATION

  common/src/test/org/apache/hadoop/hive/common/metrics/TestLegacyMetrics.java c3e8282 
  common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
a3aa549 
  itests/hive-unit/pom.xml c202001 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java PRE-CREATION

  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3c40d6e 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 93c7a54 
  service/pom.xml 22234d9 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 4ca0561 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java a9b4334 
  service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java PRE-CREATION


Diff: https://reviews.apache.org/r/40318/diff/


Testing
-------

Added two unit tests:  TestHs2Metrics, TestSessionManagerMetrics, test all of these metric
categories in in-flight and post-operation metric counts.


Thanks,

Szehon Ho


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message