hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Phillips (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-215) Map-side aggregate with DISTINCT generates bad intermediate table
Date Wed, 07 Jan 2009 14:12:44 GMT

    [ https://issues.apache.org/jira/browse/HIVE-215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661585#action_12661585
] 

David Phillips commented on HIVE-215:
-------------------------------------

I'm not really sure.  The summary "generates bad intermediate table" might be a bad diagnosis.
 Apply the patch and run these new tests:

ant -Dtestcase=TestCliDriver -Dqfile=aggr_dist1.q test
ant -Dtestcase=TestCliDriver -Dqfile=aggr_dist1_map.q test

Same query but it fails in the second reduce step with map side aggregates enabled:

{noformat}
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
    at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeDouble.serialize(DynamicSerDeTypeDouble.java:60)
    at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:236)
    at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
    at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:177)
    at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:188)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:279)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:174)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:71)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
{noformat}

> Map-side aggregate with DISTINCT generates bad intermediate table
> -----------------------------------------------------------------
>
>                 Key: HIVE-215
>                 URL: https://issues.apache.org/jira/browse/HIVE-215
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: David Phillips
>         Attachments: hive-aggrdist-testcase.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message