hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Friedrich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7955) Investigate query failures (4)
Date Fri, 07 Nov 2014 23:52:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202958#comment-14202958
] 

Thomas Friedrich commented on HIVE-7955:
----------------------------------------

The test hook_context_cs fails while checking for expected value in VerifyContentSummaryCacheHook:
  TestSparkCliDriver.testCliDriver_hook_context_cs:120->runTest:146 Unexpected exception
junit.framework.AssertionFailedError: expected:<1> but was:<0>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook.run(VerifyContentSummaryCacheHook.java:34)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1512)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1216)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1033)


The problem is that the post-hook check in org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook
fails because inputToCS.size is 0 instead of 1:
Assert.assertEquals(1, inputToCS.size());

I ran the same test with TestCliDriver where Hive will create MapRedTask objects. For a MapRedTask,
the hookContext.getInputPathToContentSummary() is set in the setNumberOfReducers method, line
400: 
inputSummary =  Utilities.getInputSummary(driverContext.getCtx(), work.getMapWork(), null);
(the path is added in 	Utilities.getInputSummary(Context, MapWork, PathFilter) line: 2584).

Since we have a SparkTask, the array will be empty, hence the check for length==1 fails. 

It looks somewhat related to HIVE-8504 where the test case also configures a hive.exec.post.hooks,
but since it's a different post-hook class altogether, I opened a separate JIRA HIVE-8790.
Maybe something needs to be added to the SparkTask.

> Investigate query failures (4)
> ------------------------------
>
>                 Key: HIVE-7955
>                 URL: https://issues.apache.org/jira/browse/HIVE-7955
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Brock Noland
>            Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, but we should
understand why they failed with an exception. This JIRA is to look into these failures and
document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dynpart_sort_optimization
12 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_schemeAuthority2	0.23 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part8	10 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4	11
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_orc_analyze	8 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_tez_join_hash	0.98 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_hook_context_cs	2.1 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_overwrite_local_directory_1
3.7 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_archive_excludeHadoop20
27 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9	8.2
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_partition_metadataonly
0.77 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers2	7 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_bigdata	0.6 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_6
6.6 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25	2.6 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query3	0.48 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16	8.5
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_empty_dir_in_table	2.6 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input33	1.3 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_admin_almighty1
2.8 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_context_aware	0.23 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_view_sqlstd
4.1 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_12
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message