pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PIG-3799) TestCustomPartitioner is broken in tez branch
Date Fri, 07 Mar 2014 01:10:46 GMT

    [ https://issues.apache.org/jira/browse/PIG-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922937#comment-13922937
] 

Cheolsoo Park edited comment on PIG-3799 at 3/7/14 1:10 AM:
------------------------------------------------------------

The parallelism is set in POForeach, and it used to overwrite the parallelism of TezOperator
before PIG-3795. But since I got rid of the overwriting logic, the parallelism of vertex is
no longer set.

Attached is a patch that explicitly sets the parallelism of vertex.


was (Author: cheolsoo):
The parallelism used to be set in POForeach, and that value used to overwrite that of TezOperator
before PIG-3795. But since I got rid of the overwriting logic, the parallelism of vertex is
no longer set.

Attached is a patch that explicitly sets the parallelism of vertex.

> TestCustomPartitioner is broken in tez branch
> ---------------------------------------------
>
>                 Key: PIG-3799
>                 URL: https://issues.apache.org/jira/browse/PIG-3799
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3799-1.patch
>
>
> This is a regression of PIG-3795. In TezCompiler, visitDistinct() doesn't set the requested
parallelism of TezOperator, resulting that only one reducer runs for the following query-
> {code}
> A = LOAD 'table_testCustomPartitionerDistinct' as (a0:int, a1:int);
> B = distinct A PARTITION BY org.apache.pig.test.utils.SimpleCustomPartitioner3 parallel
2;
> {code}
> The test fails because it sees a single output file while it expects two.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message