pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai" <dai...@gmail.com>
Subject Re: Bug in new logical optimizer framework?
Date Mon, 28 Jun 2010 04:32:21 GMT
New logical plan is half way done so it is not surprising to see exceptions at current stage.
We are actively developing it and will deliver patch shortly. Meanwhile, please attach the
problematic scripts (I didn't see it in your mail) so we can make sure those exceptions are


From: Swati Jain 
Sent: Sunday, June 27, 2010 7:07 PM
To: pig-dev@hadoop.apache.org 
Subject: Bug in new logical optimizer framework?


Posting on the dev since this is regarding the new logical plan optimization framework which
is not enabled yet. I was interested in playing around with the new optimization framework
and try adding some simple rules to it.

I have attached two simple programs which do not work when the new logical optimization framework
is enabled (they work when it is disabled). My changes to enable the new optimizer are pretty
straightforward and the diff on branch-0.7 are attached (I just set the appropriate property
to true). I have attached two very simple scripts both of which raise an exception (in local
mode of execution) "java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableIntWritable,
recieved org.apache.pig.impl.io.NullableBytesWritable" if there is atleast 1 row to be output.
The error goes away if I replace "DUMP" with "EXPLAIN" (presumably because the bug manifests
during plan execution). It would be great if someone could throw some light on this issue
or give pointers on workarounds or ways to fix this. I have not filed a JIRA for the above,
please let me know if I should.

Also, it would be great to get some guidance on the state of the new optimizer wrt testing
(I do understand it is not GA ready since it is disabled by default) and whether it is too
early to start playing around with adding new rules.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message