hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-480) allow option to retry map-reduce tasks
Date Fri, 12 Jun 2009 21:33:07 GMT

    [ https://issues.apache.org/jira/browse/HIVE-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718990#action_12718990
] 

Joydeep Sen Sarma commented on HIVE-480:
----------------------------------------

unfortunately the hive server route would almost mean that this jira is dead. even FB hasn't
standardized on using it. other installs may never use it.

is there something short term we can do? for example - taking the current patch and adding
a user prompt (unless prompting is disabled and for the case of '-f' execution) would provide
a short term solution that may help some subset of users.

another practical solution could be to try and distinguish between communication failures
to the JT (which scream for sleep/retry) and failures of the job due to task failure (which
means we shouldn't retry automatically). is not possible to do this distinction at all? (if
not - perhaps we can do something on the hadoop side to enable this).

> allow option to retry map-reduce tasks
> --------------------------------------
>
>                 Key: HIVE-480
>                 URL: https://issues.apache.org/jira/browse/HIVE-480
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Joydeep Sen Sarma
>         Attachments: HIVE-480.1.patch
>
>
> for long running queries with multiple map-reduce jobs - this should help in dealing
with any transient cluster failures without having to re-running all the tasks.
> ideally - the entire plan can be serialized out and the actual process of executing the
workflow can be left to a pluggable workflow execution engine (since this is a problem that
has been solved many times already).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message