hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruben de Vries <ruben.devr...@hyves.nl>
Subject RE: subquery + lateral view fails without count
Date Wed, 25 Apr 2012 07:15:33 GMT
Awh poor pastie :(

Here's a gist: https://gist.github.com/2487666 
with the working and crashing query, create statements and the error attached too

-----Original Message-----
From: Mark Grover [mailto:mgrover@oanda.com] 
Sent: Wednesday, April 25, 2012 1:40 AM
To: user@hive.apache.org
Subject: Re: subquery + lateral view fails without count

Hi Ruben,
Looks like pastie is down (http://pastie.org/) because of recent DDOS attacks. Can you please
post your queries elsewhere?

Mark

Mark Grover, Business Intelligence Analyst OANDA Corporation 

www: oanda.com www: fxtrade.com
e: mgrover@oanda.com 

"Best Trading Platform" - World Finance's Forex Awards 2009. 
"The One to Watch" - Treasury Today's Adam Smith Awards 2009. 


----- Original Message -----
From: "Ruben de Vries" <ruben.devries@hyves.nl>
To: user@hive.apache.org
Sent: Monday, April 23, 2012 9:08:16 AM
Subject: subquery + lateral view fails without count




It’s a bit of a weird case but I thought I might share it and hopefully find someone who
can confirm this to be a bug or tell me I should do things differently! 



Here you can find a pastie with the full create and select queries: http://pastie.org/3838924




I’ve got two tables: 

`visit_stats` with cols date_int, memberId and parts 

`member_map` with cols member_id, gender 



I use LATERAL VIEW explode(parts) to explode the parts MAP and group on those and I want to
JOIN the member_map onto the visit_stats to group by gender. 

Since LATERAL VIEW and JOIN together aren’t supported I’m doing the join in a subquery
and then LATERAL VIEW onto that. 



It’s working as long as my query contains a COUNT ( memberId ), but if I removed that from
the result it will crash with the error below: 



java.lang.RuntimeException: Error in configuring object 

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:431) 

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416) 

at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 

at java.security.AccessController.doPrivileged(Native Method) 

at javax.security.auth.Subject.doAs(Subject.java:396) 

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) 

at org.apache.hadoop.mapred.Child.main(Child.java:262) 

Caused by: java.lang.reflect.InvocationTargetException 

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 

at java.lang.reflect.Method.invoke(Method.java:597) 

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 

... 9 more 

Caused by: java.lang.RuntimeException: Reduce operator initialization failed 

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157) 

... 14 more 

Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:_col4, 1:_col6, 2:_col10]


at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:345)


at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143)


at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)


at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896) 

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:922)


at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) 

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) 

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) 

at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) 

at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:60) 

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) 

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150) 

... 14 more 


Mime
View raw message