hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1191) POCast throws exception for certain sequences of LOAD, FILTER, FORACH
Date Sat, 16 Jan 2010 08:26:54 GMT

     [ https://issues.apache.org/jira/browse/PIG-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Pradeep Kamath updated PIG-1191:

    Attachment: PIG-1191-2.patch

Attached patch to fix the issues in the script reported. The cause for the issue is that in
the parser, getFieldSchema is called on Cast's expression which results in lineage information
(canonical map) being updated in the inner map look up operator in the case of two level map
look ups. However later the inner plans of foreach get cloned invalidating the previously
stored lineage information. In the TypeCheckingVisitor , while visiting LOForeach, the schemas
of all operators in the inner plans is supposed to be reset using SchemaRemover so that schemas
get re-calculated. However SchemaRemover did not have many of the visit() methods implemented
- this patch implements the required methods and fixes the issues reported here.

> POCast throws exception for certain sequences of LOAD, FILTER, FORACH
> ---------------------------------------------------------------------
>                 Key: PIG-1191
>                 URL: https://issues.apache.org/jira/browse/PIG-1191
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Ankur
>            Assignee: Pradeep Kamath
>            Priority: Blocker
>         Attachments: PIG-1191-1.patch, PIG-1191-2.patch
> When using a custom load/store function, one that returns complex data (map of maps,
list of maps), for certain sequences  of LOAD, FILTER, FOREACH pig script throws an exception
of the form -
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a bytearray
from the UDF. Cannot determine how to convert the bytearray to <actual-type>
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:639)
> ...
> Looking through the code of POCast, apparently the operator was unable to find the right
load function for doing the conversion and consequently bailed out with the exception failing
the entire pig script.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message