hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15139) HoS local mode fails with NumberFormatException
Date Tue, 08 Nov 2016 02:30:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646213#comment-15646213
] 

Rui Li commented on HIVE-15139:
-------------------------------

Hi [~ctang.ma], thanks for the review. Yeah your solution also works, like you said, the stageId
is not used when aggregating all the metrics. But it's also no harm to pass the actual stageId
which we have already stored in the map.
Why I remove the attempt number is because what we expect is just the stageId down the road,
e.g. in {{TaskInfo}}, {{MetricsCollection}} etc. That means currently we don't differentiate
multiple attempts for a single stage, in terms of metrics.
In remote mode, we also only collect the stageId. You can refer to the {{ClientListener}}
in RemoteDriver. So this will make local mode consistent with remote mode.

> HoS local mode fails with NumberFormatException
> -----------------------------------------------
>
>                 Key: HIVE-15139
>                 URL: https://issues.apache.org/jira/browse/HIVE-15139
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HIVE-15139.1.patch
>
>
> It's because we store {{stageId_attemptNum}} in JobMetricsListener but expect only {{stageId}}
in LocalSparkJobStatus.
> {noformat}
> java.lang.NumberFormatException: For input string: "0_0"
> 	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> 	at java.lang.Integer.parseInt(Integer.java:580)
> 	at java.lang.Integer.parseInt(Integer.java:615)
> 	at org.apache.hadoop.hive.ql.exec.spark.status.impl.LocalSparkJobStatus.getSparkStatistics(LocalSparkJobStatus.java:146)
> 	at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:104)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message