hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6387) Fix Task object to allow Jobs not submitted via distributed cache to survive rolling upgrade
Date Thu, 04 Jun 2015 16:51:38 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun Suresh updated MAPREDUCE-6387:
-----------------------------------
    Summary: Fix Task object to allow Jobs not submitted via distributed cache to survive
rolling upgrade  (was: Fix Task object to serialize encryptedSpillKey at the end to to allow
rolling upgrades without distributed cache to work)

> Fix Task object to allow Jobs not submitted via distributed cache to survive rolling
upgrade
> --------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6387
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>            Priority: Minor
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object. And when
serialized, this field was written out somewhere in the middle. This caused deployments that
do not use DistributedCache to push job jars before running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there might still
be deployments that use the node local classpath to pick up the MR framework classes (eg.
for efficiency purposes, since this does not require the jar being copied to hdfs and then
to all the nodes)
> Ensuring that it is the last field written and read when the Task object is serialized
would alleviate this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message