hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-17781) Map MR settings to Tez settings via DeprecatedKeys
Date Thu, 12 Oct 2017 13:45:00 GMT


Hive QA commented on HIVE-17781:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 11211 tests executed
*Failed tests:*
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=239)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed

This message is automatically generated.

ATTACHMENT ID: 12891595 - PreCommit-HIVE-Build

> Map MR settings to Tez settings via DeprecatedKeys
> --------------------------------------------------
>                 Key: HIVE-17781
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Configuration, Tez
>    Affects Versions: 3.0.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Chris Drome
>         Attachments: HIVE-17781.1.patch
> Here's one that [~cdrome] and [~thiruvel] worked on:
> We found that certain Hadoop Map/Reduce settings that are set in site config files do
not take effect in Hive jobs, because the Tez site configs do not contain the same settings.
> In Yahoo's case, the problem was that, at the time, there was no mapping between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}}
and {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where significant
capacity on production clusters were being used up doing nothing, while waiting for slow tasks
to complete. This would have been avoided, were the mappings in place.
> Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to Tez settings.
Hive should use this to ensure that the mappings are in sync.
> (Note to self: YHIVE-883)

This message was sent by Atlassian JIRA

View raw message