Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7E30118207 for ; Wed, 6 Jan 2016 15:07:40 +0000 (UTC) Received: (qmail 35746 invoked by uid 500); 6 Jan 2016 15:07:40 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 35697 invoked by uid 500); 6 Jan 2016 15:07:40 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 35655 invoked by uid 99); 6 Jan 2016 15:07:40 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2016 15:07:40 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id CF4252C1F56 for ; Wed, 6 Jan 2016 15:07:39 +0000 (UTC) Date: Wed, 6 Jan 2016 15:07:39 +0000 (UTC) From: "Junping Du (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (YARN-1593) support out-of-proc AuxiliaryServices MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reassigned YARN-1593: -------------------------------- Assignee: Junping Du > support out-of-proc AuxiliaryServices > ------------------------------------- > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade > Reporter: Ming Ma > Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the ShuffleHandler restart. If ShuffleHandler runs as a separate process, ShuffleHandler can continue to run during NM restart. NM can reconnect the the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will be implemented. AuxiliaryServices are considered YARN application specific and could consume lots of resources. Running AuxiliaryServices in separate processes allow easier resource management. NM could potentially stop a specific AuxiliaryServices process from running if it consumes resource way above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will continue to run. NM could reconnect to the running AuxiliaryService processes upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have immediate need for this. AuxiliaryService could run inside a container and its resource utilization could be taken into account by RM and RM could consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332)