hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20103) WM: Only Aggregate DAG counters if at least one is used
Date Fri, 06 Jul 2018 07:43:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gopal V updated HIVE-20103:
---------------------------
    Status: Patch Available  (was: Open)

> WM: Only Aggregate DAG counters if at least one is used
> -------------------------------------------------------
>
>                 Key: HIVE-20103
>                 URL: https://issues.apache.org/jira/browse/HIVE-20103
>             Project: Hive
>          Issue Type: Bug
>          Components: llap
>    Affects Versions: 3.0.0, 4.0.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>            Priority: Major
>         Attachments: HIVE-20103.1.patch
>
>
> {code}
>         status = dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS), checkInterval);
>         TezCounters dagCounters = status.getDAGCounters();
> ...
>         if (dagCounters != null && wmContext != null) {
>           Set<String> desiredCounters = wmContext.getSubscribedCounters();
>           if (desiredCounters != null && !desiredCounters.isEmpty()) {
>             Map<String, Long> currentCounters = getCounterValues(dagCounters, vertexNames,
vertexProgressMap,
>               desiredCounters, done);
> {code}
> Skip collecting DAG counters unless there at least one desired counter in wmContext.
> The AM has a hard-lock around the counters, so the current jstacks are full of 
> {code}
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.String.intern(Native Method)
>         at org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
>         at org.apache.tez.common.counters.GenericCounter.<init>(GenericCounter.java:50)
>         at org.apache.tez.common.counters.TezCounters$GenericGroup.newCounter(TezCounters.java:65)
>         at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
>         at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
>         - locked <0x00007efb3ac7af38> (a org.apache.tez.common.counters.TezCounters$GenericGroup)
>         at org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
>         at org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
>         - eliminated <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
>         at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:357)
>         - locked <0x00007efb3ac64ee8> (a org.apache.tez.common.counters.TezCounters)
>         at org.apache.tez.dag.app.dag.impl.TaskImpl.getCounters(TaskImpl.java:462)
>         at org.apache.tez.dag.app.dag.impl.VertexImpl.aggrTaskCounters(VertexImpl.java:1342)
>         at org.apache.tez.dag.app.dag.impl.VertexImpl.getAllCounters(VertexImpl.java:1202)
>         at org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:755)
>         at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:704)
>         at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
>         at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
>         at org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message