hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7158) Use Tez auto-parallelism in Hive
Date Sun, 01 Jun 2014 23:28:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015118#comment-14015118
] 

Hive QA commented on HIVE-7158:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12647760/HIVE-7158.2.patch

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 5571 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats16
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/362/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/362/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-362/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12647760

> Use Tez auto-parallelism in Hive
> --------------------------------
>
>                 Key: HIVE-7158
>                 URL: https://issues.apache.org/jira/browse/HIVE-7158
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gunther Hagleitner
>            Assignee: Gunther Hagleitner
>         Attachments: HIVE-7158.1.patch, HIVE-7158.2.patch
>
>
> Tez can optionally sample data from a fraction of the tasks of a vertex and use that
information to choose the number of downstream tasks for any given scatter gather edge.
> Hive estimates the count of reducers by looking at stats and estimates for each operator
in the operator pipeline leading up to the reducer. However, if this estimate turns out to
be too large, Tez can reign in the resources used to compute the reducer.
> It does so by combining partitions of the upstream vertex. It cannot, however, add reducers
at this stage.
> I'm proposing to let users specify whether they want to use auto-parallelism or not.
If they do there will be scaling factors to determine max and min reducers Tez can choose
from. We will then partition by max reducers, letting Tez sample and reign in the count up
until the specified min.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message