hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-17781) Map MR settings to Tez settings via DeprecatedKeys
Date Thu, 12 Oct 2017 00:00:00 GMT
Mithun Radhakrishnan created HIVE-17781:
-------------------------------------------

             Summary: Map MR settings to Tez settings via DeprecatedKeys
                 Key: HIVE-17781
                 URL: https://issues.apache.org/jira/browse/HIVE-17781
             Project: Hive
          Issue Type: Bug
          Components: Configuration, Tez
    Affects Versions: 3.0.0
            Reporter: Mithun Radhakrishnan
            Assignee: Chris Drome


Here's one that [~cdrome] and [~thiruvel] worked on:

We found that certain Hadoop Map/Reduce settings that are set in site config files do not
take effect in Hive jobs, because the Tez site configs do not contain the same settings.

In Yahoo's case, the problem was that, at the time, there was no mapping between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}}
and {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where significant
capacity on production clusters were being used up doing nothing, while waiting for slow tasks
to complete. This would have been avoided, were the mappings in place.

Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to Tez settings.
Hive should use this to ensure that the mappings are in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message