pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-5201) Null handling on FLATTEN
Date Sat, 28 Oct 2017 04:08:00 GMT

     [ https://issues.apache.org/jira/browse/PIG-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Koji Noguchi updated PIG-5201:
    Attachment: pig-5201-v05.patch

Thanks for the review Rohini! 

bq. 1) Can we rename flattenNumFieldsForNull to flattenNumFields? 

bq. 2) POForeach clone should copy flattenNumFieldsForNull
Completely missed that. Done.

bq. 3) isToBeFlattenedArray[ i ] is checked in 3 if statements. Can we create a outer block
with that condition to simplify?
Took a look but I'm already afraid of introducing a regression with this current change. 
Prefer not to touch other places if not necessary. 

bq. 4) TestFlatten.java - license is modified. Unintended change?
Ouch. Fixed.

bq. System.err.println("KOJIKOJI");
Another ouch. Wiped. 

7) testFlattenOnNullBagWithColumnPrune is redundant. ColumPrune does not affect Flatten. Currently
we don’t column prune nested data structures - tuple, bag (PIG-1324). 
Actually this was testing what would happen when a simple field is pruned. In the test with
schema {noformat}
(a0:int, bag1:bag {(a1_1:int, a1_2:chararray)}, bag2:bag{(a2_1:chararray, a2_2:chararray)})
this test is pruning a0:int and checking if {{flattenNumFields}} needs to be shifted.

Lastly, I added a document for this new behavior.

> Null handling on FLATTEN
> ------------------------
>                 Key: PIG-5201
>                 URL: https://issues.apache.org/jira/browse/PIG-5201
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>             Fix For: 0.18.0
>         Attachments: pig-5201-v00-testonly.patch, pig-5201-v01.patch, pig-5201-v02.patch,
pig-5201-v03.patch, pig-5201-v04.patch, pig-5201-v05.patch
> Sometimes, FLATTEN(null) or FLATTEN(bag-with-null) seem to produce incorrect results.
> Test code/script to follow.

This message was sent by Atlassian JIRA

View raw message