hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14521) codahale metrics exceptions
Date Thu, 11 Aug 2016 22:02:21 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418015#comment-15418015
] 

Szehon Ho commented on HIVE-14521:
----------------------------------

So there is a method 'SessionState.getPerfLogger().cleanupPerfLogMetrics();' that i introduced
in the driver return path for success or failure case that should take care of the error handling
in theory.

As per multi threading, the scopes are Thread local, but they are opened at [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L348]
and closed [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L559]
so i believe this should be on the same thread.

I did a search and noticed that the PerfLogger.COMPILE scope is also used in TezJobMonitor,
that might cause a conflict as its different thread.  Is it possible to rename that one?

> codahale metrics exceptions
> ---------------------------
>
>                 Key: HIVE-14521
>                 URL: https://issues.apache.org/jira/browse/HIVE-14521
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>
> One some random setup, I see bazillions of errors like this in HS2 log, Gb-s of logs
worth:
> {noformat}
> 2016-08-08 04:52:18,619 WARN  [HiveServer2-Handler-Pool: Thread-101]: log.PerfLogger
(PerfLogger.java:beginMetrics(226)) - Error recording metrics
> java.io.IOException: Scope named api_Driver.run is not closed, cannot be opened.
>         at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope.open(CodahaleMetrics.java:133)
>         at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics.startStoredScope(CodahaleMetrics.java:220)
>         at org.apache.hadoop.hive.ql.log.PerfLogger.beginMetrics(PerfLogger.java:223)
>         at org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:143)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:378)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:320)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1214)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1208)
>         at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
>         at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:226)
>         at org.apache.hive.service.cli.operation.Operation.run(Operation.java:276)
>         at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:468)
>         at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:456)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {noformat}
> I suspect that either, just like the metastore deadline, this needs better error handling
when whatever the metrics surround fails; or, it is just not thread safe.
> But I actually haven't looked at the code yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message