hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-574) PrivateLocalizer does not support parallel resource download via ContainerLocalizer
Date Mon, 07 Nov 2016 17:11:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644745#comment-15644745
] 

Naganarasimha G R commented on YARN-574:
----------------------------------------

Thanks [~ajithshetty], for the patch. 
As discussed offline, I think existing approach of sending only one resource as HB response
is better as there can be some control at NM end to determine (if req in future) whether to
give multiple resources to be localized or one at a time(or even none based on the NM's load).
Further to it i would to add further points
# Agree to [~varun_saxena]'s point having fixed 4 threads (core and max pool size ) is not
ideal. But IMHO i would like to keep atleast 2(default max pool size), as users might not
be able make use of the benifit if not.
# I think instead of using {{Executors.newFixedThreadPool(nThreads, tf)}} we can try to use
{{ThreadPoolExecutor}} so that we can we can make use of {{ThreadPoolExecutor.getActiveCount()}}
and compare it with max pool size to determine whether to request Heartbeat immediately or
to wait 1000s.
# If NM sends *LIVE* and if no ResourceLocalizationSpecs are shared across then no need to
check the current load on the executor, we can wait for the defined poll period and then do
the HB.
# TestContainerLocalizer.java, ln no 301-306, Please add some proper message which can be
shown on failure.
# {{TestYarnConfigurationFields.testCompareConfigurationClassAgainstXml}} is related to the
patch

> PrivateLocalizer does not support parallel resource download via ContainerLocalizer
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-574
>                 URL: https://issues.apache.org/jira/browse/YARN-574
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.6.0, 2.8.0, 2.7.1
>            Reporter: Omkar Vinit Joshi
>            Assignee: Ajith S
>         Attachments: YARN-574.03.patch, YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple containers
request the same resource. However otherwise it will be serial. The protocol between PrivateLocalizer
and ContainerLocalizer supports multiple downloads however it is not used and only one resource
is sent for downloading at a time.
> I think we can increase / assure parallelism (even for single container requesting resource)
for private/application resources by making multiple downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number of containers[private
and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number of containers
* max downloads per container [private and application resource]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message