hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14627) Improvements to MiniMr tests
Date Sat, 27 Aug 2016 18:25:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15442017#comment-15442017
] 

Hive QA commented on HIVE-14627:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825747/HIVE-14627.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10464 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1023/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1023/console
Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1023/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12825747 - PreCommit-HIVE-MASTER-Build

> Improvements to MiniMr tests
> ----------------------------
>
>                 Key: HIVE-14627
>                 URL: https://issues.apache.org/jira/browse/HIVE-14627
>             Project: Hive
>          Issue Type: Sub-task
>    Affects Versions: 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-14627.1.patch, HIVE-14627.2.patch, HIVE-14627.3.patch
>
>
> Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following are the
execution time breakdown
> Total time - 13m59s
> Junit reported time for testcase - 50s
> Most of the time is spent in creating/loading/analyzing initial tables - ~12m
> Cleanup - ~1m
> There is huge overhead for running MiniMr tests when compared to the actual test runtime.

> Ran the same test without init script.
> Total time - 2m17s
> Junit reported time for testcase - 52s
> Also I noticed some tests that doesn't have to run on MiniMr (like udf_using.q that does
not require MiniMr. It just reads/write to hdfs which we can do in MiniTez/MiniLlap which
are way faster). Most tests access only very few initial tables to read few rows from it.
We can fix those tests to load just the table that is required for the table instead of all
initial tables. Also we can remove q_init_script.sql initialization for MiniMr after rewriting
and moving over the unwanted tests which should cut down the runtime a lot.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message