Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0439A200CC9 for ; Mon, 3 Jul 2017 04:56:13 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 02EDA160BFE; Mon, 3 Jul 2017 02:56:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 47FD2160BEE for ; Mon, 3 Jul 2017 04:56:12 +0200 (CEST) Received: (qmail 77084 invoked by uid 500); 3 Jul 2017 02:56:06 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 77072 invoked by uid 99); 3 Jul 2017 02:56:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2017 02:56:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E3ECF185EB6 for ; Mon, 3 Jul 2017 02:56:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Qx5v5ayYndOn for ; Mon, 3 Jul 2017 02:56:05 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id D7C6E5F3A1 for ; Mon, 3 Jul 2017 02:56:04 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 655FEE0D3C for ; Mon, 3 Jul 2017 02:56:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0267C245E2 for ; Mon, 3 Jul 2017 02:56:01 +0000 (UTC) Date: Mon, 3 Jul 2017 02:56:01 +0000 (UTC) From: "zhengchenyu (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 03 Jul 2017 02:56:13 -0000 [ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071869#comment-16071869 ] zhengchenyu commented on YARN-6396: ----------------------------------- The invocation 'createAppDir' in initAppAggregator will also visit the defaultNs! In our experience, it decreases the speed of application. > Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node > ------------------------------------------------------------------------------------------------------------------------------- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation > Affects Versions: 3.0.0-alpha2 > Reporter: zhihai xu > Assignee: zhihai xu > Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir will be called before doing log aggregation, This will be a non trivial overhead for name node in a large cluster since verifyAndCreateRemoteLogDir calls getFileStatus. Once the remote log directory is created successfully, it is not necessary to call it again. It will be better to call verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org