hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4318) OperatorHooks hit performance even when not used
Date Sat, 20 Apr 2013 19:11:16 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637323#comment-13637323
] 

Hudson commented on HIVE-4318:
------------------------------

Integrated in Hive-trunk-hadoop2 #167 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/167/])
    HIVE-4318 : OperatorHooks hit performance even when not used (Gunther Hagleitner via Ashutosh
Chauhan) (Revision 1470101)

     Result = FAILURE
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470101
Files : 
* /hive/trunk/build-common.xml
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecMapper.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecReducer.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorHook.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorHookContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorHookUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilePublisher.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilePublisherInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfiler.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilerAggregateStat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilerConnectionInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilerStats.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilerStatsAggregator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/profiler/HiveProfilerUtils.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TstOperatorHook.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TstOperatorHookUtils.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/HiveProfilerResultsHook.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/PostTestOperatorHook.java
* /hive/trunk/ql/src/test/queries/clientpositive/hiveprofiler0.q
* /hive/trunk/ql/src/test/queries/clientpositive/hiveprofiler_script0.q
* /hive/trunk/ql/src/test/queries/clientpositive/hiveprofiler_union0.q
* /hive/trunk/ql/src/test/queries/clientpositive/operatorhook.q
* /hive/trunk/ql/src/test/results/clientpositive/hiveprofiler0.q.out
* /hive/trunk/ql/src/test/results/clientpositive/hiveprofiler_script0.q.out
* /hive/trunk/ql/src/test/results/clientpositive/hiveprofiler_union0.q.out
* /hive/trunk/ql/src/test/results/clientpositive/operatorhook.q.out

                
> OperatorHooks hit performance even when not used
> ------------------------------------------------
>
>                 Key: HIVE-4318
>                 URL: https://issues.apache.org/jira/browse/HIVE-4318
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>         Environment: Ubuntu LXC (64 bit)
>            Reporter: Gopal V
>            Assignee: Gunther Hagleitner
>             Fix For: 0.12.0
>
>         Attachments: HIVE-4318.1.patch, HIVE-4318.2.patch, HIVE-4318.3.patch, HIVE-4318.patch.pam.txt
>
>
> Operator Hooks inserted into Operator.java cause a performance hit even when it is not
being used.
> For a count(1) query tested with & without the operator hook calls.
> {code:title=with}
> 2013-04-09 07:33:58,920 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 84.07 sec
> Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec
> OK
> 28800991
> Time taken: 40.407 seconds, Fetched: 1 row(s)
> {code}
> {code:title=without}
> 2013-04-09 07:33:02,355 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 68.48 sec
> ...
> Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec
> OK
> 28800991
> Time taken: 35.907 seconds, Fetched: 1 row(s)
> {code}
> The effect is multiplied by the number of operators in the pipeline that has to forward
the row - the more operators there are the, the slower the query.
> The modification made to test this was 
> {code:title=Operator.java}
> --- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> +++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> @@ -526,16 +526,16 @@ public void process(Object row, int tag) throws HiveException {
>        return;
>      }
>      OperatorHookContext opHookContext = new OperatorHookContext(this, row, tag);
> -    preProcessCounter();
> -    enterOperatorHooks(opHookContext);
> +    //preProcessCounter();
> +    //enterOperatorHooks(opHookContext);
>      processOp(row, tag);
> -    exitOperatorHooks(opHookContext);
> -    postProcessCounter();
> +    //exitOperatorHooks(opHookContext);
> +    //postProcessCounter();
>    }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message