tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mustafa İman (Jira) <j...@apache.org>
Subject [jira] [Commented] (TEZ-4254) Don't unset the tez config if both mr and tez config have same value.
Date Wed, 06 Jan 2021 07:32:00 GMT

    [ https://issues.apache.org/jira/browse/TEZ-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259466#comment-17259466
] 

Mustafa İman commented on TEZ-4254:
-----------------------------------

[~rajesh.balamohan] it makes sense for tez value to reference mapreduce one to make sure
it always matches NM value. Otherwise, you have to make the value in tez-site and core-site
in sync manually. I remember I specifically did such thing for cloudera manager.

As to solution, what matters is not that mr config and tez config resolve to the same value.
What matters is tez config references the mr counterpart. The more general approach is just
not to unset mr key if tez key is a reference (${another.key} rather than  a concrete value).

> Don't unset the tez config if both mr and tez config have same value.
> ---------------------------------------------------------------------
>
>                 Key: TEZ-4254
>                 URL: https://issues.apache.org/jira/browse/TEZ-4254
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.9.1
>            Reporter: Rajkumar Singh
>            Priority: Major
>         Attachments: TEZ-4254.patch
>
>
> while translated mr config to the tez config we unset config here.
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRHelpers.java#L187
> There is no need to unset the config if both mr and tez have same value.
> we observed if equivalent of tez config for mapreduce.shuffle.ssl.enabled (tez.runtime.shuffle.ssl.enable)
set to true, we unset the config, this propagate config tez.runtime.shuffle.ssl.enable={mapreduce.shuffle.ssl.enabled}
at runtime and variable substitution does not work correctly in the absence of mrconfig runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message