hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7118) Distributed cache conflicts breaks backwards compatability
Date Thu, 05 Jul 2018 21:24:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534176#comment-16534176

Rohini Palaniswamy commented on MAPREDUCE-7118:

bq. we actually took care of this in OOZIE-3227
   That is fixed in the newer release in which Oozie runs launcher jobs in its own AM instead
of MR AM. I am not sure how many would be migrating to it soon. Also this might affect other
schedulers and directly launched MR applications by users. It is not worth breaking the backward
compatibility for MR at this point as it's usage is already going down and make it a pain
for users. The jars have to be de-duplicated in any case. It is not much of a difference if
it is done in Oozie or MR AM as in both cases, it is not the user who is removing duplicates
and one jar randomly wins over another beating the actual intent of not allowing duplicates.
So I am all for going back to the old behavior of allowing duplicates in MR AM itself.

> Distributed cache conflicts breaks backwards compatability
> ----------------------------------------------------------
>                 Key: MAPREDUCE-7118
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7118
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 3.0.0, 3.1.0, 3.2.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Blocker
>         Attachments: MAPREDUCE-7118.001.patch
> MAPREDUCE-4503 made distributed cache conflicts break job submission, but this was quickly
downgraded to a warning in MAPREDUCE-4549.  Unfortunately the latter did not go into trunk,
so the fix is only in 0.23 and 2.x.  When Oozie, Pig, and other downstream projects that can
occasionally generate distributed cache conflicts move to Hadoop 3.x the workflows that used
to work on 0.23 and 2.x no longer function.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message