Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 7 Nov 2016 16:34:58 +0000 (UTC)
From: "Varun Saxena (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12642353.1365813263000.210988.1478536498767@Atlassian.JIRA>
In-Reply-To: <JIRA.12642353.1365813263000@Atlassian.JIRA>
References: <JIRA.12642353.1365813263000@Atlassian.JIRA> <JIRA.12642353.1365813263770@arcas>
Subject: [jira] [Commented] (YARN-574) PrivateLocalizer does not support
 parallel resource download via ContainerLocalizer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 07 Nov 2016 16:35:00 -0000


    [ https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644642#comment-15644642 ] 

Varun Saxena commented on YARN-574:
-----------------------------------

[~ajithshetty], thanks for the patch.
Parallel downloads should speed up container localization phase.
Coming to the patch, should the configuration value be 4 ? Or should we keep it as 1 i.e. current behavior. And then it can be tuned based on size and visibility of resources to be downloaded, if somebody wants to use it.
Localizers are not treated as containers which means resources used by them are not accounted for, so it should not be that they all together end up eating up quite a bit of resources on the node with default value of 4.
Thoughts ?


> PrivateLocalizer does not support parallel resource download via ContainerLocalizer
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-574
>                 URL: https://issues.apache.org/jira/browse/YARN-574
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.6.0, 2.8.0, 2.7.1
>            Reporter: Omkar Vinit Joshi
>            Assignee: Ajith S
>         Attachments: YARN-574.03.patch, YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple containers request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer and ContainerLocalizer supports multiple downloads however it is not used and only one resource is sent for downloading at a time.
> I think we can increase / assure parallelism (even for single container requesting resource) for private/application resources by making multiple downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number of containers[private and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number of containers * max downloads per container [private and application resource]


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org