hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1651) ScriptOperator should not forward any output to downstream operators if an exception is happened
Date Fri, 17 Sep 2010 22:46:32 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910834#action_12910834
] 

Ning Zhang commented on HIVE-1651:
----------------------------------

@joydeep, the output file will not be committed if an exception occurred and close(abort=true)
is called. This bug happened in a short time window after the exception occurred and before
the close(abort) is called. Although the file got deleted, the dynamic partition insert already
created a directory which later will be considered as an empty partition. 

> ScriptOperator should not forward any output to downstream operators if an exception
is happened
> ------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-1651
>                 URL: https://issues.apache.org/jira/browse/HIVE-1651
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
>         Attachments: HIVE-1651.patch
>
>
> ScriptOperator spawns 2 threads for getting the stdout and stderr from the script and
then forward the output from stdout to downstream operators. In case of any exceptions to
the script (e.g., got killed), the ScriptOperator got an exception and throw it to upstream
operators until MapOperator got it and call close(abort). Before the ScriptOperator.close()
is called the script output stream can still forward output to downstream operators. We should
terminate it immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message