hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-767) Schema reported from DESCRIBE and actual schema of inner bags are different.
Date Sat, 10 Jul 2010 00:25:50 GMT

     [ https://issues.apache.org/jira/browse/PIG-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Olga Natkovich updated PIG-767:
-------------------------------

    Fix Version/s: 0.9.0
      Description: 
The following script:

urlContents = LOAD 'inputdir' USING BinStorage() AS (url:bytearray, pg:bytearray);
-- describe and dump are in-sync
DESCRIBE urlContents;
DUMP urlContents;

urlContentsG = GROUP urlContents BY url;
DESCRIBE urlContentsG;

urlContentsF = FOREACH urlContentsG GENERATE group,urlContents.pg;

DESCRIBE urlContentsF;
DUMP urlContentsF;


Prints for the DESCRIBE commands:

urlContents: {url: chararray,pg: chararray}
urlContentsG: {group: chararray,urlContents: {url: chararray,pg: chararray}}
urlContentsF: {group: chararray,pg: {pg: chararray}}

The reported schemas for urlContentsG and urlContentsF are wrong. They are also against the
section "Schemas for Complex Data Types" in http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.htm#_Schemas.

As expected, actual data observed from DUMP urlContentsG and DUMP urlContentsF do contain
the tuple inside the inner bags.

The correct schema for urlContentsG is:  {group: chararray,urlContents: {t1:(url: chararray,pg:
chararray)}}

This may sound like a technicality, but it isn't. For instance, a UDF that assumes an inner
bag of {chararray} will not work with {(chararray)}. 





  was:

The following script:

urlContents = LOAD 'inputdir' USING BinStorage() AS (url:bytearray, pg:bytearray);
-- describe and dump are in-sync
DESCRIBE urlContents;
DUMP urlContents;

urlContentsG = GROUP urlContents BY url;
DESCRIBE urlContentsG;

urlContentsF = FOREACH urlContentsG GENERATE group,urlContents.pg;

DESCRIBE urlContentsF;
DUMP urlContentsF;


Prints for the DESCRIBE commands:

urlContents: {url: chararray,pg: chararray}
urlContentsG: {group: chararray,urlContents: {url: chararray,pg: chararray}}
urlContentsF: {group: chararray,pg: {pg: chararray}}

The reported schemas for urlContentsG and urlContentsF are wrong. They are also against the
section "Schemas for Complex Data Types" in http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.htm#_Schemas.

As expected, actual data observed from DUMP urlContentsG and DUMP urlContentsF do contain
the tuple inside the inner bags.

The correct schema for urlContentsG is:  {group: chararray,urlContents: {t1:(url: chararray,pg:
chararray)}}

This may sound like a technicality, but it isn't. For instance, a UDF that assumes an inner
bag of {chararray} will not work with {(chararray)}. 






> Schema reported from DESCRIBE and actual schema of inner bags are different.
> ----------------------------------------------------------------------------
>
>                 Key: PIG-767
>                 URL: https://issues.apache.org/jira/browse/PIG-767
>             Project: Pig
>          Issue Type: Bug
>            Reporter: George Mavromatis
>             Fix For: 0.9.0
>
>
> The following script:
> urlContents = LOAD 'inputdir' USING BinStorage() AS (url:bytearray, pg:bytearray);
> -- describe and dump are in-sync
> DESCRIBE urlContents;
> DUMP urlContents;
> urlContentsG = GROUP urlContents BY url;
> DESCRIBE urlContentsG;
> urlContentsF = FOREACH urlContentsG GENERATE group,urlContents.pg;
> DESCRIBE urlContentsF;
> DUMP urlContentsF;
> Prints for the DESCRIBE commands:
> urlContents: {url: chararray,pg: chararray}
> urlContentsG: {group: chararray,urlContents: {url: chararray,pg: chararray}}
> urlContentsF: {group: chararray,pg: {pg: chararray}}
> The reported schemas for urlContentsG and urlContentsF are wrong. They are also against
the section "Schemas for Complex Data Types" in http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.htm#_Schemas.
> As expected, actual data observed from DUMP urlContentsG and DUMP urlContentsF do contain
the tuple inside the inner bags.
> The correct schema for urlContentsG is:  {group: chararray,urlContents: {t1:(url: chararray,pg:
chararray)}}
> This may sound like a technicality, but it isn't. For instance, a UDF that assumes an
inner bag of {chararray} will not work with {(chararray)}. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message