hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruben de Vries <>
Subject subquery + lateral view fails without count
Date Mon, 23 Apr 2012 13:08:16 GMT
It's a bit of a weird case but I thought I might share it and hopefully find someone who can
confirm this to be a bug or tell me I should do things differently!

Here you can find a pastie with the full create and select queries:

I've got two tables:
`visit_stats` with cols date_int, memberId and parts
`member_map` with cols member_id, gender

I use LATERAL VIEW explode(parts) to explode the parts MAP and group on those and I want to
JOIN the member_map onto the visit_stats to group by gender.
Since LATERAL VIEW and JOIN together aren't supported I'm doing the join in a subquery and
then LATERAL VIEW onto that.

It's working as long as my query contains a COUNT ( memberId ), but if I removed that from
the result it will crash with the error below:

java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(
        at org.apache.hadoop.util.ReflectionUtils.setConf(
        at org.apache.hadoop.util.ReflectionUtils.newInstance(
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(
        at org.apache.hadoop.mapred.Child$
        at Method)
        at org.apache.hadoop.mapred.Child.main(
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
        at java.lang.reflect.Method.invoke(
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(
        ... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
        at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(
        ... 14 more
Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:_col4, 1:_col6, 2:_col10]
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(
        at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(
        at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(
        at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(
        at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(
        at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(
        ... 14 more

View raw message