Mailing-List: contact issues-help@tez.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tez.apache.org
Date: Tue, 23 Feb 2016 17:58:20 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: issues@tez.apache.org
Message-ID: <JIRA.12940198.1455814451000.124317.1456250300118@Atlassian.JIRA>
In-Reply-To: <JIRA.12940198.1455814451000@Atlassian.JIRA>
References: <JIRA.12940198.1455814451000@Atlassian.JIRA>
 <JIRA.12940198.1455814451675@arcas>
Subject: [jira] [Commented] (TEZ-3126) Log reason for not reducing
 parallelism
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/TEZ-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159312#comment-15159312 ] 

Jonathan Eagles commented on TEZ-3126:
--------------------------------------

[~rohini], 	TEZ-2962 was filed before to make the change you are suggesting and in the above comment. Let's track that work in that JIRA. Linking these two together to better track this issue.

> Log reason for not reducing parallelism
> ---------------------------------------
>
>                 Key: TEZ-3126
>                 URL: https://issues.apache.org/jira/browse/TEZ-3126
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Critical
>             Fix For: 0.7.1, 0.8.3
>
>         Attachments: TEZ-3126.1.patch, TEZ-3126.2.patch
>
>
> For example, when reducing parallelism from 36 to 22. The basePartitionRange will be 1 and will not re-configure the vertex.
> {code:java|title=ShuffleVertexManager#determineParallelismAndApply|borderStyle=dashed|bgColor=lightgrey}
>     int desiredTaskParallelism = 
>         (int)(
>             (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
>             desiredTaskInputDataSize);
>     if(desiredTaskParallelism < minTaskParallelism) {
>       desiredTaskParallelism = minTaskParallelism;
>     }
>     
>     if(desiredTaskParallelism >= currentParallelism) {
>       return true;
>     }
>     
>     // most shufflers will be assigned this range
>     basePartitionRange = currentParallelism/desiredTaskParallelism;
>     
>     if (basePartitionRange <= 1) {
>       // nothing to do if range is equal 1 partition. shuffler does it by default
>       return true;
>     }
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)