hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kumar V <kumarbuyonl...@yahoo.com>
Subject Re: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3]
Date Thu, 27 Feb 2014 19:05:53 GMT
Hi,
   I wrote a similar simple UDTF and a new table. This simple UDTF does work on hive 0.10. 

But my original one doesn't.  Still don't understand why.

Does the fact that the original query works with the setting 'set hive.optimize.ppd=true'
give any clue ?

Please let me know.




On Tuesday, February 25, 2014 3:28 PM, java8964 <java8964@hotmail.com> wrote:
  
Works for me on 0.10.

Yong



________________________________
Date: Tue, 25 Feb 2014 11:37:32 -0800
From: kumarbuyonline@yahoo.com
Subject: Re: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3]
To: user@hive.apache.org


Hi,
    Thanks for looking into it.
I am also trying this on hive 0.11 to see if it works there.  If you get a chance to reproduce
this problem on hive 0.10, please let me know.

Thanks.



On Monday, February 24, 2014 10:59 PM, java8964 <java8964@hotmail.com> wrote:
  
My guess is that your UDTF will return an array of struct. 

I don't have Hive 0.10 in handy right now, but I write a simple UDTF to return an array of
struct to test on Hive 0.12 release.

hive> desc test;
OK
id                  int                 None                
name                string              None                
Time taken: 0.074 seconds, Fetched: 2 row(s)
hive> select * from test;
OK
1Apples,Bananas,Carrots
Time taken: 0.08 seconds, Fetched: 1 row(s)

The pair UDTF will output "Apples,Bananas,Carrots"
to 
"Apples, Bananas"
"Apples, Carrots"
"Bananas, Carrots"
an array of 2 elements struct.

hive> select id, name, m1, m2 from test lateral view pair(name) p as m1, m2 where m1 is
not null;
OK
1Apples,Bananas,CarrotsApplesBananas
1Apples,Bananas,CarrotsApplesCarrots
1Apples,Bananas,CarrotsBananasCarrots
Time taken: 7.683 seconds, Fetched: 3 row(s)

hive> select id, name, m1, m2 from test lateral view pair(name) p as m1, m2 where m1 =
'Apples';
OK
1Apples,Bananas,CarrotsApplesBananas
1Apples,Bananas,CarrotsApplesCarrots
Time taken: 7.726 seconds, Fetched: 2 row(s)

hive> set hive.optimize.ppd=true;
hive> select id, name, m1, m2 from test lateral view pair(name) p as m1, m2 where m1 is
not null;
Total MapReduce jobs = 1
OK
1Apples,Bananas,CarrotsApplesBananas
1Apples,Bananas,CarrotsApplesCarrots
1Apples,Bananas,CarrotsBananasCarrots
Time taken: 7.716 seconds, Fetched: 3 row(s)
I cannot reproduce your error in Hive 0.12, as you can see. 

I can test on Hive 0.10 tomorrow when I have time, but can your test your case in Hive 0.12,
or review your UDTF again?

Yong



________________________________
Date: Mon, 24 Feb 2014 07:09:44 -0800
From: kumarbuyonline@yahoo.com
Subject: Re: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3]
To: user@hive.apache.org; kumarbuyonline@yahoo.com


As suggested, I changed the query like this:

select x.f1,x,f2,x,f3,x.f4
from (
select e.f1 as f1,e.f2 as f2,e.f3 as f3,e.f4 as f4 from mytable LATERAL VIEW myfunc(p1,p2,p3,p4)
e  as f1,f2,f3,f4 where lang=123) x 
where x.f3 is not null;

And it still doesn't work. I am getting the same error.  
If anyone has any ideas, please let me know.

Thanks.



On Friday, February 21, 2014 11:27 AM, Kumar V <kumarbuyonline@yahoo.com> wrote:
  
Line 316 in my UDTF where is shows the error is the line where I call forward().

The whole trace is :

Caused by: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col6,
3:_col7, 4:_col8, 5:_col9]
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:346)
at
 org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
at org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:55)
at org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:55)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128)
at
 org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:85)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
 org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:112)
at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:81)
at pb2CSVReadFile.FlatTableFileUDTFTx.process(FlatTableFileUDTFTx.java:316)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.processOp(UDTFOperator.java:98)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
 org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
 org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:132)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at
 org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:658)
... 9 more



On Friday, February 21, 2014 11:18 AM, java8964 <java8964@hotmail.com> wrote:
  
What is your stracktrace? Can you paste here?

It is maybe a different bug.

If you put e.f3 <> null at an outsider query? Does that work?

Or maybe you have to enhance your UDTF to push that filter into your UDTF. It is not perfect,
but maybe a solution for you as now.

You can create a new Jira if it is a new bug.

Yong



________________________________
Date: Fri, 21 Feb 2014 07:18:32 -0800
From:
 kumarbuyonline@yahoo.com
Subject: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3]
To: user@hive.apache.org


Hi,
       I have a UDTF which works fine except when I do a query like the following :

select e.* from mytable LATERAL VIEW myfunc(p1,p2,p3,p4) e  as f1,f2,f3,f4 where lang=123and
e.f3 <> null;

The error I see is:

java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3]



If i remove 'and e.f3 <> null' from the WHERE clause, it works fine.

Also, with e.f3 <> null in the WHERE clause, if I add the setting 

hive.optimize.ppd=false

 
it works fine, but now, instead of using 600 mappers, it uses about 10,000 mappers and runs
for more than 2 hours instead of a few minutes.

I am using hive 0.10.  I saw the jira HIVE-3226 which says that it has been fixed in hive
0.10.  Is this the bug that I am hitting now ?
Any other ideas of how to make it work ? I am actually on CDH 4.4 which has hive 0.10.

Please let me know.

Thanks,
Murali. 
Mime
View raw message