Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============5063278465871358013=="
MIME-Version: 1.0
Subject: Re: Review Request 51193: HIVE-14358: Add metrics for number of
 queries executed for each execution engine (mr, spark, tez)
From: Barna Zsombor Klara <zsombor.klara@cloudera.com>
To: Sergio Pena <sergio.pena@cloudera.com>, Lenni Kuff <lskuff@cloudera.com>,
 Mohit Sabharwal <mohit@cloudera.com>
Cc: Barna Zsombor Klara <zsombor.klara@cloudera.com>,
 Peter Vary <pvary@cloudera.com>, hive <dev@hive.apache.org>
Date: Thu, 18 Aug 2016 23:07:07 -0000
Message-ID: <20160818230707.26606.35564@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Barna Zsombor Klara <noreply@reviews.apache.org>
References: <20160818230017.25630.15017@reviews.apache.org>
In-Reply-To: <20160818230017.25630.15017@reviews.apache.org>
Reply-To: Barna Zsombor Klara <zsombor.klara@cloudera.com>
archived-at: Thu, 18 Aug 2016 23:07:11 -0000

--===============5063278465871358013==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51193/
-----------------------------------------------------------

(Updated Aug. 18, 2016, 11:07 p.m.)


Review request for hive, Lenni Kuff, Mohit Sabharwal, and Sergio Pena.


Changes
-------

Removed unused import.


Repository: hive-git


Description
-------

HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark, tez)


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java 9dc96f9c6412720a891b5c55e2074049c893d780 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 183ed829ef1742e48539f8928293d56b77bc43c8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java eeaa54320ffaa7ba5d6ebece80a0cb4aadc1dada 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java ce1106d91db9ef75e7b425d5950f888bacbfb3e5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ac922ce486babe042984d87a7f7442cbfc11484f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 0b494aa5548f8e6ae76e2d0eea9a7afb33961f97 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25c4514b34fb2ed4fc8b1238059bd9dc29d2741b 
  ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 53672a9783b4d13c5eed4ef01f5c16af568a0a41 

Diff: https://reviews.apache.org/r/51193/diff/


Testing
-------

Ran the new unit tests in the ql project, everything was green.
Checked that the metrics for map reduce and spark tasks were appearing and being incremented correctly using JMX. 
Map reduce tasks were being created by a simple select statement containing a join.
Spark tasks were being created by the same query with the spark execution engine being used.
The metrics were correct across several beeline connections, and were reset once the HiveServer2 was restarted.
The metric collection can be turned on/off using the configuration variable "hive.server2.metrics.enabled". No errors/exceptions encountered when the metrics were disabled.

NB only the root tasks are incrementing the counter since the original jira was about counting the number of queries issued against each exeution engine, so a complex query resulting in more than one task should only count as one as per my understanding.


Thanks,

Barna Zsombor Klara


--===============5063278465871358013==--