hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Friedrich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7953) Investigate query failures (2)
Date Thu, 09 Oct 2014 02:19:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164597#comment-14164597
] 

Thomas Friedrich commented on HIVE-7953:
----------------------------------------

The 4 tests
bucketsortoptimize_insert_2
bucketsortoptimize_insert_4
bucketsortoptimize_insert_7
bucketsortoptimize_insert_8
all fail with the same NPE related to SMB joins:

order object is null in SMBMapJoinOperator: 
      // fetch the first group for all small table aliases
      for (byte pos = 0; pos < order.length; pos++) {
        if (pos != posBigTable) {
          fetchNextGroup(pos);
        }

Daemon Thread [Executor task launch worker-3] (Suspended (exception NullPointerException))

	SMBMapJoinOperator.processOp(Object, int) line: 258	
	FilterOperator(Operator<T>).forward(Object, ObjectInspector) line: 799	
	FilterOperator.processOp(Object, int) line: 137	
	TableScanOperator(Operator<T>).forward(Object, ObjectInspector) line: 799	
	TableScanOperator.processOp(Object, int) line: 95	
	MapOperator(Operator<T>).forward(Object, ObjectInspector) line: 799	
	MapOperator.process(Writable) line: 536	
	SparkMapRecordHandler.processRow(Object, Object) line: 139	
	HiveMapFunctionResultList.processNextRecord(Tuple2<BytesWritable,BytesWritable>) line:
47	
	HiveMapFunctionResultList.processNextRecord(Object) line: 28	
	HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108	
	Wrappers$JIteratorWrapper<A>.hasNext() line: 41	
	Iterator$class.foreach(Iterator, Function1) line: 727	
	Wrappers$JIteratorWrapper<A>(AbstractIterator<A>).foreach(Function1<A,U>)
line: 1157	
	RDD$$anonfun$foreach$1.apply(Iterator<T>) line: 760	
	RDD$$anonfun$foreach$1.apply(Object) line: 760	
	SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator<T>) line: 1118	
	SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118	
	ResultTask<T,U>.runTask(TaskContext) line: 61	
	ResultTask<T,U>(Task<T>).run(long) line: 56	
	Executor$TaskRunner.run() line: 182	
	ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145	
	ThreadPoolExecutor$Worker.run() line: 615	
	Thread.run() line: 745	

There is also a NPE in the FileSinkOperator: the FileSystem object fs is null: 
      // in recent hadoop versions, use deleteOnExit to clean tmp files.
      if (isNativeTable) {
        autoDelete = fs.deleteOnExit(fsp.outPaths[0]);

Daemon Thread [Executor task launch worker-1] (Suspended (exception NullPointerException))

	FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495	
	FileSinkOperator.closeOp(boolean) line: 925	
	FileSinkOperator(Operator<T>).close(boolean) line: 582	
	SelectOperator(Operator<T>).close(boolean) line: 594	
	SMBMapJoinOperator(Operator<T>).close(boolean) line: 594	
	DummyStoreOperator(Operator<T>).close(boolean) line: 594	
	FilterOperator(Operator<T>).close(boolean) line: 594	
	TableScanOperator(Operator<T>).close(boolean) line: 594	
	MapOperator(Operator<T>).close(boolean) line: 594	
	SparkMapRecordHandler.close() line: 175	
	HiveMapFunctionResultList.closeRecordProcessor() line: 57	
	HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122	
	Wrappers$JIteratorWrapper<A>.hasNext() line: 41	
	Iterator$class.foreach(Iterator, Function1) line: 727	
	Wrappers$JIteratorWrapper<A>(AbstractIterator<A>).foreach(Function1<A,U>)
line: 1157	
	RDD$$anonfun$foreach$1.apply(Iterator<T>) line: 760	
	RDD$$anonfun$foreach$1.apply(Object) line: 760	
	SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator<T>) line: 1118	
	SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118	
	ResultTask<T,U>.runTask(TaskContext) line: 61	
	ResultTask<T,U>(Task<T>).run(long) line: 56	
	Executor$TaskRunner.run() line: 182	
	ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145	
	ThreadPoolExecutor$Worker.run() line: 615	
	Thread.run() line: 745	


> Investigate query failures (2)
> ------------------------------
>
>                 Key: HIVE-7953
>                 URL: https://issues.apache.org/jira/browse/HIVE-7953
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Brock Noland
>            Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, but we should
understand why they failed with an exception. This JIRA is to look into these failures and
document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external	0.33
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers	4.3
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2	11
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
0.65 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
4.7 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
2.8 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
5.5 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position	1.5 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external	2.4
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6	11
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11	5.1 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
10 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join	5.4 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part	0.81
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1	0.31 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1	0.26 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2	0.73 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3	8.5
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5	0.34 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata	0.93 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
6.3 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3	2.4 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2	0.56 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23	3.1
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2	4.3 sec
2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part	3.2
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external	2.8
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed	3.4
sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location
3.3 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location
6.9 sec	2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message