pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4079) Parallel clause is not honored in local mode
Date Tue, 29 Jul 2014 16:37:40 GMT

     [ https://issues.apache.org/jira/browse/PIG-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-4079:
----------------------------

    Attachment: PIG-4079-2.patch

TestLogicalPlanBuilder.testQuery18 after the patch. Attach PIG-4079-2.patch.

> Parallel clause is not honored in local mode
> --------------------------------------------
>
>                 Key: PIG-4079
>                 URL: https://issues.apache.org/jira/browse/PIG-4079
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: 0.14.0
>
>         Attachments: PIG-4079-1.patch, PIG-4079-2.patch
>
>
> I want to debug my custom partitioner in local mode. But it's not possible with Pig because:
> # MR does not invoke partitioner if # of reducers is equal to 1.
> https://issues.apache.org/jira/browse/MAPREDUCE-1287
> # Pig always forces a single reducers in local mode.
> {code:title=LogicalPlanBuilder.java}
>      void setParallel(LogicalRelationalOperator op, Integer parallel) {
>          if( parallel != null ) {
>             op.setRequestedParallelism( pigContext.getExecType() == ExecType.LOCAL ?
1 : parallel );
>          }
>      }
> {code}
> After I change Pig to honor the parallel clause, my custom partitioner gets invoked in
local mode.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message