hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
Date Wed, 21 Sep 2016 04:41:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508689#comment-15508689
] 

Jian He edited comment on YARN-5621 at 9/21/16 4:40 AM:
--------------------------------------------------------

Thanks for your explanation,  some parts I'm missing: 
bq. Both will start ContainerLocalizer instances
Is it starting both instances now?  Not sure if I read the code wrong...  It seems not the
case. Based on the code, if it's an already existing resource,  it will NOT start the ContainerLocalizer.
e.g. below comments in ResourceLocalizationService.  Could you point me the right cod... maybe
I'm missing something.
{code}
         /*
          * Multiple containers will try to download the same resource. So the
          * resource download should start only if
          * 1) We can acquire a non blocking semaphore lock on resource
          * 2) Resource is still in DOWNLOADING state
          */
{code}
bq.  it may be used by both, concurrently.
This approach may not be easily worked for the new containers without structural change, because
for new containers, when localizer is started, the work-dirs are not setup yet. It cannot
create symlinks on localization for new containers. The work-dirs are created later when launching
the container.
bq.  for services that upgrade over minutes/hours,
Not only for upgrades, it's also used by Tez for localizing resources on demand. 

I think I understand your approach now,  basically,
To create the symlinks,
1. we start the localizer process,
2. send the symlinks over on localizer heartbeat, 
3. localizer process create symlinks. 
right ?


was (Author: jianhe):
Thanks for your explanation,  some parts I'm missing: 
bq. Both will start ContainerLocalizer instances
Is it starting both instances now?  Not sure if I read the code wrong...  It seems not the
case. Based on the code, if it's an already existing resource,  it will NOT start the ContainerLocalizer.
e.g. below comments in ResourceLocalizationService.  Could you point me the right cod... maybe
I'm missing something.
{code}
         /*
          * Multiple containers will try to download the same resource. So the
          * resource download should start only if
          * 1) We can acquire a non blocking semaphore lock on resource
          * 2) Resource is still in DOWNLOADING state
          */
{code}
bq.  it may be used by both, concurrently.
This approach may not be easily worked for the new containers without structural change, because
for new containers, when localizer is started, the work-dirs are not setup yet. It cannot
create symlinks on localization for new containers. The work-dirs are created later when launching
the container.
bq.  for services that upgrade over minutes/hours,
Not only for upgrades, it's also used by Tez for localizing resources on demand. 

I think I understand your approach now,  basically,
To create the symlinks,
1. we start the localizer process,
2. send the symlinks over on localizer heartbeat, 
3. localizer process create symlinks. 

One question is, should we keep the localizer process around or terminates it immediately
after symlinks created. If we keep it around for certain time, we need to add some life timeout
for the localizer process. If it's immediately terminated,  that means every localize request
by AM would spawn/kill a new process, and I think that could churn over the machine resources.

> Support LinuxContainerExecutor to create symlinks for continuously localized resources
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, YARN-5621.4.patch,
YARN-5621.5.patch
>
>
> When new resources are localized, new symlink needs to be created for the localized resource.
This is the change for the LinuxContainerExecutor to create the symlinks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message