hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on
Date Thu, 18 Feb 2016 12:38:18 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152252#comment-15152252
] 

Rui Li commented on HIVE-13066:
-------------------------------

I'm not able to reproduce the issue. But I tried to make the task fail if the output file
already exists. And speculative task fails as expected, which means it's possible that speculative
tasks write to the same file as original tasks and corrupt the results. Therefore I think
we should append the real attempt ID for each task, if that doesn't cause too much trouble.
Meanwhile we still need a way to verify the root cause of the issue. Any ideas?

> Hive on Spark gives incorrect results when speculation is on
> ------------------------------------------------------------
>
>                 Key: HIVE-13066
>                 URL: https://issues.apache.org/jira/browse/HIVE-13066
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Rui Li
>
> The issue is reported by users. One possible reason is that we always append 0 as the
attempt ID for each task so that hive won't be able to distinguish between speculative tasks
and original ones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message