hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3491) Improve the public resource localization to do both FSDownload submission to the thread pool and completed localization handling in one thread (PublicLocalizer).
Date Thu, 16 Apr 2015 01:14:59 GMT

     [ https://issues.apache.org/jira/browse/YARN-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zhihai xu updated YARN-3491:
----------------------------
    Description: 
Improve the public resource localization to do both FSDownload submission to the thread pool
and completed localization handling in one thread (PublicLocalizer).
Currently FSDownload submission to the thread pool is done in PublicLocalizer#addResource
which is running in Dispatcher thread and completed localization handling is done in PublicLocalizer#run
which is running in PublicLocalizer thread.
Because PublicLocalizer#addResource is time consuming, the thread pool can't be fully utilized.
Instead of doing public resource localization in parallel(multithreading), public resource
localization is serialized most of the time.

Also there are two more benefits with this change:
1. The Dispatcher thread won't be blocked by above FSDownload submission. Dispatcher thread
handles most of time critical events at Node manager.
2. don't need synchronization on HashMap (pending).
Because pending will be only accessed in PublicLocalizer thread.

  was:
Improve the public resource localization to do both FSDownload submission to the thread pool
and completed localization handling in one thread (PublicLocalizer).
Currently FSDownload submission to the thread pool is done in PublicLocalizer#addResource
which is running in Dispatcher thread and completed localization handling is done in PublicLocalizer#run
which is running in PublicLocalizer thread.
Because FSDownload submission to the thread pool at the following code is time consuming,
the thread pool can't be fully utilized. Instead of doing public resource localization in
parallel(multithreading), public resource localization is serialized most of the time.
{code}
            synchronized (pending) {
              pending.put(queue.submit(new FSDownload(lfs, null, conf,
                  publicDirDestPath, resource, request.getContext().getStatCache())),
                  request);
            }
{code}

Also there are two more benefits with this change:
1. The Dispatcher thread won't be blocked by above FSDownload submission. Dispatcher thread
handles most of time critical events at Node manager.
2. don't need synchronization on HashMap (pending).
Because pending will be only accessed in PublicLocalizer thread.


> Improve the public resource localization to do both FSDownload submission to the thread
pool and completed localization handling in one thread (PublicLocalizer).
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3491
>                 URL: https://issues.apache.org/jira/browse/YARN-3491
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>            Priority: Critical
>
> Improve the public resource localization to do both FSDownload submission to the thread
pool and completed localization handling in one thread (PublicLocalizer).
> Currently FSDownload submission to the thread pool is done in PublicLocalizer#addResource
which is running in Dispatcher thread and completed localization handling is done in PublicLocalizer#run
which is running in PublicLocalizer thread.
> Because PublicLocalizer#addResource is time consuming, the thread pool can't be fully
utilized. Instead of doing public resource localization in parallel(multithreading), public
resource localization is serialized most of the time.
> Also there are two more benefits with this change:
> 1. The Dispatcher thread won't be blocked by above FSDownload submission. Dispatcher
thread handles most of time critical events at Node manager.
> 2. don't need synchronization on HashMap (pending).
> Because pending will be only accessed in PublicLocalizer thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message