hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Barna Zsombor Klara <zsombor.kl...@cloudera.com>
Subject Re: Review Request 51193: HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark, tez)
Date Thu, 18 Aug 2016 23:07:07 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51193/
-----------------------------------------------------------

(Updated Aug. 18, 2016, 11:07 p.m.)


Review request for hive, Lenni Kuff, Mohit Sabharwal, and Sergio Pena.


Changes
-------

Removed unused import.


Repository: hive-git


Description
-------

HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark,
tez)


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java 9dc96f9c6412720a891b5c55e2074049c893d780

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 183ed829ef1742e48539f8928293d56b77bc43c8

  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java eeaa54320ffaa7ba5d6ebece80a0cb4aadc1dada

  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java ce1106d91db9ef75e7b425d5950f888bacbfb3e5

  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ac922ce486babe042984d87a7f7442cbfc11484f

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 0b494aa5548f8e6ae76e2d0eea9a7afb33961f97

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25c4514b34fb2ed4fc8b1238059bd9dc29d2741b

  ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 53672a9783b4d13c5eed4ef01f5c16af568a0a41


Diff: https://reviews.apache.org/r/51193/diff/


Testing
-------

Ran the new unit tests in the ql project, everything was green.
Checked that the metrics for map reduce and spark tasks were appearing and being incremented
correctly using JMX. 
Map reduce tasks were being created by a simple select statement containing a join.
Spark tasks were being created by the same query with the spark execution engine being used.
The metrics were correct across several beeline connections, and were reset once the HiveServer2
was restarted.
The metric collection can be turned on/off using the configuration variable "hive.server2.metrics.enabled".
No errors/exceptions encountered when the metrics were disabled.

NB only the root tasks are incrementing the counter since the original jira was about counting
the number of queries issued against each exeution engine, so a complex query resulting in
more than one task should only count as one as per my understanding.


Thanks,

Barna Zsombor Klara


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message