hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12466) SparkCounter not initialized error
Date Mon, 23 Nov 2015 06:48:10 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021611#comment-15021611
] 

Chengxiang Li commented on HIVE-12466:
--------------------------------------

[~xuefuz], due to the limitation of Spark accumulator, the `SparkCounter` has to register
the counter name before the job execution. The error message shows that specified counter
name is not registered before. In default, all the default spark counters are collected with
`SparkTask::getCounterPrefixes()`, `RECORDS_OUT_0`, `RECORDS_OUT_1_default.tmp_tmp` and `RECORDS_OUT_1_default.test_table`
are not included, seems the counter logic changes in `ReduceSinkOperator` and 'FileSinkOperator',
we need to update the logic of `SparkTask::getOperatorCounters`. 

> SparkCounter not initialized error
> ----------------------------------
>
>                 Key: HIVE-12466
>                 URL: https://issues.apache.org/jira/browse/HIVE-12466
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Xuefu Zhang
>
> During a query, lots of the following error found in executor's log:
> {noformat}
> 03:47:28.759 [Executor task launch worker-0] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_0] has not initialized before.
> 03:47:28.762 [Executor task launch worker-1] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_0] has not initialized before.
> 03:47:30.707 [Executor task launch worker-1] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_1_default.tmp_tmp] has not initialized before.
> 03:47:33.385 [Executor task launch worker-1] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:33.388 [Executor task launch worker-0] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:33.495 [Executor task launch worker-0] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:35.141 [Executor task launch worker-1] ERROR org.apache.hive.spark.counter.SparkCounters
- counter[HIVE, RECORDS_OUT_1_default.test_table] has not initialized before.
> ...........
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message