hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-161) Rework physical plan
Date Mon, 21 Apr 2008 20:53:21 GMT

    [ https://issues.apache.org/jira/browse/PIG-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591051#action_12591051
] 

Alan Gates commented on PIG-161:
--------------------------------

I haven't gotten fully through this, but I have a couple of comments:

1) We need to be using the combiner aggressively out of the gate.  I expect MRCompiler is
the place you'll want to use it.  It should be handling placing things in the compile stage.
  If you want to do it in a second pass over the MROpPlan after MRCompiler is done, I think
that's fine.  But we need to have this as part of our initial work.  I'd like to resolve whether
we're going to do the combiner in MRCompiler or a separate pass before I commit this code.

2) In several places in MRCompiler you have unexpected if/else chains ending in 'log.warn("didn't
expect this")'.  These should be runtime exceptions I suspect.

3) Rather than the MergeException you created, let's create a general PlanException.  I can
then change OperatorPlan to throw that instead of an IOException, and you can use it in your
merge stuff.  I'll make that change.

> Rework physical plan
> --------------------
>
>                 Key: PIG-161
>                 URL: https://issues.apache.org/jira/browse/PIG-161
>             Project: Pig
>          Issue Type: Sub-task
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: arithmeticOperators.patch, incr2.patch, incr3.patch, incr4.patch,
incr5.patch, MRCompilerTests_PlansAndOutputs.txt, Phy_AbsClass.patch, podistinct.patch, pogenerate.patch,
pogenerate.patch, pogenerate.patch, posort.patch
>
>
> This bug tracks work to rework all of the physical operators as described in http://wiki.apache.org/pig/PigTypesFunctionalSpec

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message