Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 15 Nov 2016 21:52:59 +0000 (UTC)
From: "Sangjin Lee (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12688637.1389656759000.285897.1479246779533@Atlassian.JIRA>
In-Reply-To: <JIRA.12688637.1389656759000@Atlassian.JIRA>
References: <JIRA.12688637.1389656759000@Atlassian.JIRA> <JIRA.12688637.1389656759965@arcas>
Subject: [jira] [Commented] (YARN-1593) support out-of-proc
 AuxiliaryServices
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 15 Nov 2016 21:53:02 -0000


    [ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668499#comment-15668499 ] 

Sangjin Lee commented on YARN-1593:
-----------------------------------

Thanks for starting the proposal!

I took a quick look at it, and here are some of my initial thoughts (maybe more later).

One use case that is not mentioned is the timeline service v.2 collector (writer). We can think of it in two possible approaches: (1) another system container/service that needs to be launched on every node before NM can serve containers, or (2) system container that can be started on demand when an app is started (one container per app). I think (1) fits nicely with the system container you're envisioning. (2) is much more dynamic than any of the approaches discussed in the doc. FYI.

I think discovery is going to be one major piece that needs to be addressed from the beginning. Even in the most basic use cases (e.g. MR shuffle handler, timeline reader, etc.), the discoverability of the containers and their endpoints is hugely important. It would be great if it is addressed in the first design.

I also agree that localization is going to be a problem, and I think it's going to be an issue no matter which option you take. If the system container needs to run as long as the node is up, it's hard to avoid the issue of localization unless you pre-deliver the bits as part of setting up the nodes.

In terms of the approaches, I lean slightly towards (3). It feels awkward to treat it as "just another app" as they have different semantics from any other app. If we're elevating the notion of the system containers to first class, we might as well be explicit while still trying to reuse a lot of the pieces for implementation. That's my 2 cents.

One question: what do we do with the resource utilization of these system containers? Should they be reported just like any container (I'm thinking of {{ContainersMonitorImpl}}, {{NMTimelinePublisher}} and so on)? Or should they be considered outside the monitoring scope, like a YARN daemon today? Have you thought about that?

> support out-of-proc AuxiliaryServices
> -------------------------------------
>
>                 Key: YARN-1593
>                 URL: https://issues.apache.org/jira/browse/YARN-1593
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, rolling upgrade
>            Reporter: Ming Ma
>            Assignee: Varun Vasudev
>         Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the ShuffleHandler restart. If ShuffleHandler runs as a separate process, ShuffleHandler can continue to run during NM restart. NM can reconnect the the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will be implemented. AuxiliaryServices are considered YARN application specific and could consume lots of resources. Running AuxiliaryServices in separate processes allow easier resource management. NM could potentially stop a specific AuxiliaryServices process from running if it consumes resource way above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will continue to run. NM could reconnect to the running AuxiliaryService processes upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have immediate need for this. AuxiliaryService could run inside a container and its resource utilization could be taken into account by RM and RM could consider a specific type of applications overutilize cluster resource.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org