hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map
Date Tue, 13 Dec 2016 18:45:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745900#comment-15745900
] 

Miklos Szegedi commented on YARN-5714:
--------------------------------------

Thank you, [~rcatherinot] for the reply!
1.
{quote}
echo %%A%% in windows does output A's value and not %%A%%...
{quote}
The double %% sign is the escape for %, so a batch program like the one below will print %A%.
I tested it on Windows 10 within a batch file. (http://www.robvanderwoude.com/escapechars.php)
a.bat:
{code}
set A=foo
echo %%A%%
{code}
The output running a.bat:
{code}
%A%
{code}
2.
Yes, it is not a bug now, but it may lead to errors, if someone changes the code later. I
think it is a better practice to use the pattern that you also use below and the index never
exceeds the limit by 2:
{code}
1005	          if ((++i) >= l) {
1006	            break;
1007	          }
{code}
3.
I think you are missing an 's' at the *:
{code}
855	      // 2 - we need a map implementation that support* entries to be
{code}
4.
We cannot rule out that someone will want to run YARN with very short queries in the future.
In this case however your algorithm will be on a critical path. It might help to prove the
efficiency to add a unit test with 100 environment variables with dependencies in reverse
order, to verify a limit on the runtime.
5.
Understood.

> ContainerExecutor does not order environment map
> ------------------------------------------------
>
>                 Key: YARN-5714
>                 URL: https://issues.apache.org/jira/browse/YARN-5714
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
>         Environment: all (linux and windows alike)
>            Reporter: Remi Catherinot
>            Assignee: Remi Catherinot
>            Priority: Trivial
>              Labels: oct16-medium
>         Attachments: YARN-5714.001.patch, YARN-5714.002.patch, YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped based on the
order internally used by the map implementation (hash based). It does not take into consideration
that some env varibales may refer each other, and so that some env variables must be declared
before those referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on HADOOP_COMMON_HOME
being dumped before HADOOP_COMMON_HOME. Thus it had a wrong value and so native libraries
weren't loaded. jobs were running but not at their best efficiency. This is just a use case
falling into that bug, but i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate to 5 days
for packaging the patch in the right fashion for JIRA + try my best to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will only implement
the unix version on a 1st release. I'm not used to windows env variables syntax so it will
take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message