hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-161) Rework physical plan
Date Fri, 18 Apr 2008 16:56:28 GMT

    [ https://issues.apache.org/jira/browse/PIG-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590519#action_12590519

Alan Gates commented on PIG-161:

Comments on posort patch:

1) In the comparator function you provide to the sorted data bag, you are including functionality
for both user defined sort functions and standard data type aware sorting.  This forces an
if statement as the first action of every compare, yet that if will give the same answer for
every compare in a given sort.  So I think this should be broken into two inner classes, StandardSortComparator
(which just handles standard sorting) and UserDefinedSortComparator which handles wrapping
POUserFunc.  Then when you instantiate the SortedDataBag you can pass the appropriate comparator.

2) How does the user provided comparator know which columns to sort on?  I don't see where
that is being communicated to it.

3) Just a clarification.  This operator seems appropriate only for local operation.  Are you
assuming that in the MapReduce case the POSort operator will be replaced by a POCogroup by
the MapReduce compiler?

> Rework physical plan
> --------------------
>                 Key: PIG-161
>                 URL: https://issues.apache.org/jira/browse/PIG-161
>             Project: Pig
>          Issue Type: Sub-task
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: arithmeticOperators.patch, incr2.patch, incr3.patch, incr4.patch,
Phy_AbsClass.patch, pogenerate.patch, pogenerate.patch, pogenerate.patch, posort.patch
> This bug tracks work to rework all of the physical operators as described in http://wiki.apache.org/pig/PigTypesFunctionalSpec

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message