hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14636) pass information from FSOP/TezTask to MoveTask to take care of speculative execution and failed tasks
Date Fri, 26 Aug 2016 00:06:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438172#comment-15438172
] 

Sergey Shelukhin commented on HIVE-14636:
-----------------------------------------

After some discussions and looking at code, it looks like for now we will write a manifest
into the FS to replace the FSOp-specific commit logic.
MoveTask will go off these manifests and delete them
There's no way to pass the payload from FSOp running inside tasks back to the client where
MoveTask lives without involving some intermediate storage.
As a side note, it appears that Hive does not actually support speculative execution provided
by Tez, so we never call canCommit.
Retroactive failures are impossible for final outputs, and at any rate it doesn't seem like
they should happen against persistent FS  if Hive could write final outputs from intermediate
stages - otherwise it would mean HDFS/S3/... is losing data.
So \*waves hands vigorously\* manifests will not collide the same way the existing FSOP moves
do not collide.


> pass information from FSOP/TezTask to MoveTask to take care of speculative execution
and failed tasks
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-14636
>                 URL: https://issues.apache.org/jira/browse/HIVE-14636
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message