hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shubham Chopra (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-161) Rework physical plan
Date Thu, 17 Apr 2008 16:39:21 GMT

     [ https://issues.apache.org/jira/browse/PIG-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shubham Chopra updated PIG-161:
-------------------------------

    Attachment: posort.patch

Attached is the POSort implementation. It was made such that it can be used for the local
backend. The PhysicalToMapReduce compiler should take care of implementing the jobs associated
with sorting. This is because the distribution of the input data is not known and some reducers
might get most of the tuples if the distribution is particularly skewed. 

This way, the PhysicalToMapReduce compiler would be aware of the sort operator in the plan
and should be able to effectively convert it into 2 jobs. One to find out the data distribution
and the second to do the final sorting based on the information obtained. This was also the
reason converting Sort to other basic physical operators didn't seem like a good idea. Comments??

> Rework physical plan
> --------------------
>
>                 Key: PIG-161
>                 URL: https://issues.apache.org/jira/browse/PIG-161
>             Project: Pig
>          Issue Type: Sub-task
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: arithmeticOperators.patch, incr2.patch, incr3.patch, incr4.patch,
Phy_AbsClass.patch, pogenerate.patch, pogenerate.patch, pogenerate.patch, posort.patch
>
>
> This bug tracks work to rework all of the physical operators as described in http://wiki.apache.org/pig/PigTypesFunctionalSpec

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message