hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Alvaro <palv...@gmail.com>
Subject more ClassCastExceptions with joins, group by and non-strings
Date Sat, 25 Apr 2009 23:47:27 GMT
Hi,
It appears that queries with all three of (join, group by, non-string
datatype) cause a crash in the serde code run at the reducer:


java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to
org.apache.hadoop.io.Text
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:179)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.DoubleWritable cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:359)


the workaround mentioned in the FAQ(3) (and reported fixed by HIVE-405,
although this seems to be a different issue) does not seem to fix the
problem, which has existed since hive revision 764548.  I am using hadoop
v.0.19, though I get the same errors when I use the latest trunk.

this script (which includes the workaround) captures the issue:


-----------------------------------------
create table foo (
    bas string,
    bam double
)

ROW FORMAT DELIMITED FIELDS TERMINATED BY '\174';

create table bar (
    bas string--,
    --bat double
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\174';



load data local inpath '/PATH/TO/demo_foo.txt' overwrite into table foo;
load data local inpath '/PATH/TO/demo_bar.txt' overwrite into table bar;


select f.bas, cast(bam as string)
    from foo f
        join bar b on (f.bas = b.bas)
            group by f.bas, cast(bam as string);

-------------------------------

contents of demo_foo.txt:

11234325|0.123
221346|10.12
33463246|100.25
432462634|0.12
5346236|345.12


contents of demo_bar:

11234325|0.1222
221346|1.11
33463246|235.23
432462634|6.33
5346236|77.77

thanks,

Peter Alvaro
UC Berkeley

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message