hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-587) Duplicate result from multiple TIPs of the same task
Date Mon, 29 Jun 2009 20:32:47 GMT

    [ https://issues.apache.org/jira/browse/HIVE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725333#action_12725333

Namit Jain commented on HIVE-587:

I did not understand the changes in ScriptOperator

> Duplicate result from multiple TIPs of the same task
> ----------------------------------------------------
>                 Key: HIVE-587
>                 URL: https://issues.apache.org/jira/browse/HIVE-587
>             Project: Hadoop Hive
>          Issue Type: Bug
>    Affects Versions: 0.3.0, 0.3.1
>            Reporter: Zheng Shao
>            Priority: Blocker
>         Attachments: HIVE-587.1.patch
> On our cluster we found a job committed with duplicate output from different TIPs of
the same Task (from FileSinkOperator).
> The reason is that FileSinkOperator.commit can be called at multiple TIPs of the same
> FileSinkOperator.jobClose() (which is called at the Hive Client side) should do either:
> A. Get all successful TIPs and only move the output files of those TIPs to the output
> B. Ignore TIPs from the JobInProgress, but only move one file out of potentially several
output files 
> B is preferred because A might be slow (if the job finished and immediately got moved
out of the JobTracker memory). Since we control the file name by ourselves, we know exactly
what the file names are.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message