hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2495) Allow admin specify labels in each NM (Distributed configuration)
Date Mon, 13 Oct 2014 16:52:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169528#comment-14169528

Allen Wittenauer commented on YARN-2495:

I don't fully understand your question, but I'll admit I've been distracted with other JIRAs
lately.  Hopefully I'm guessing correctly though... :)

The idea that there are 'valid' labels and that one turns them on/off seems to conflict with
the goals that I want to see fulfilled with this feature.  Let me give a more concrete example,
since I know a lot of people are confused as to why labels should be dynamic anyway:

Here's node A's configuration:

$ hadoop version
Hadoop 2.6.0
Subversion http://github.com/apache/hadoop -r 1e6d81a8869ceeb2f0f81f2ee4b89833f2b22cd4
Compiled by aw on 2014-10-10T19:31Z
Compiled with protoc 2.5.0
>From source with checksum 201dad5b10939faa6e5841378c8c94
This command was run using /sw/hadoop/hadoop-2.6.0-SNAPSHOT/share/hadoop/common/hadoop-common-2.6.0-SNAPSHOT.jar
$ java -version
java version "1.6.0_32"
OpenJDK Runtime Environment (IcedTea6 1.13.4) (rhel-
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Here's node B's configuration:
$ ~/HADOOP/hadoop-3.0.0-SNAPSHOT/bin/hadoop version
Hadoop 3.0.0
Source code repository https://git-wip-us.apache.org/repos/asf/hadoop.git -r 7b29f99ad23b2a87eac17fdcc7b5b29cd6c9b0c0
Compiled by aw on 2014-10-08T21:12Z
Compiled with protoc 2.5.0
>From source with checksum ae703b1a38a35d19f4584495dc31944
This command was run using /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

(Ignore the fact that they probably can't be in the same cluster. That's not really relevant.)

I want to provide a script similar to this one:
#!/usr/bin/env bash

HADOOPVERSION=$(${HADOOP_PREFIX}/bin/hadoop version 2>/dev/null | /usr/bin/head -1 |/usr/bin/cut
-f2 -d\ )
JAVAVERSION=$(${JAVA_HOME}/bin/java -version 2>&1 |/usr/bin/head -1| /usr/bin/cut -f2
-d\" )
JAVAMMM=$(echo ${JAVAVERSION}| /usr/bin/cut -f1 -d_)


This script, when run, should tell the RM that:

Node A has labels 1.6.0_32, jdk1.6.0, and hadoop_2.6.0 .
Node B has labels LABELS=jdk1.7.0_67, jdk1.7.0, and hadoop_3.0.0

Users should be able to submit a job that specifies that it only runs with 'hadoop_3.0.0'.
 Or that it requires 'jdk_1.7.0'.

Making labels either valid or invalid based upon a central configuration defeats the purpose
of having a living, breathing cluster able to adapt to change without operations intervention.
 If we are rolling out a new version of the JDK, I shouldn't have to tell the system that
it's ok to broadcast that JDK version first.

> Allow admin specify labels in each NM (Distributed configuration)
> -----------------------------------------------------------------
>                 Key: YARN-2495
>                 URL: https://issues.apache.org/jira/browse/YARN-2495
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
> Target of this JIRA is to allow admin specify labels in each NM, this covers
> - User can set labels in each NM (by setting yarn-site.xml or using script suggested
by [~aw])
> - NM will send labels to RM via ResourceTracker API
> - RM will set labels in NodeLabelManager when NM register/update labels

This message was sent by Atlassian JIRA

View raw message