Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5538E200C5B for ; Thu, 13 Apr 2017 01:48:46 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 53C06160BA8; Wed, 12 Apr 2017 23:48:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9BCF7160B95 for ; Thu, 13 Apr 2017 01:48:45 +0200 (CEST) Received: (qmail 81245 invoked by uid 500); 12 Apr 2017 23:48:44 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 81234 invoked by uid 99); 12 Apr 2017 23:48:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Apr 2017 23:48:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 60FF6C04DD for ; Wed, 12 Apr 2017 23:48:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id bVQWSrpYve_r for ; Wed, 12 Apr 2017 23:48:43 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 8A5375FD81 for ; Wed, 12 Apr 2017 23:48:43 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id EB500E0069 for ; Wed, 12 Apr 2017 23:48:42 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id D59462406D for ; Wed, 12 Apr 2017 23:48:41 +0000 (UTC) Date: Wed, 12 Apr 2017 23:48:41 +0000 (UTC) From: "zhihai xu (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 12 Apr 2017 23:48:46 -0000 [ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961891#comment-15961891 ] zhihai xu edited comment on YARN-6396 at 4/12/17 11:48 PM: ----------------------------------------------------------- Thanks for the review [~haibochen], [~rkanter][~xgong] Could you also help review the patch? thanks was (Author: zxu): Thanks for the review [~haibochen], [~xgong] Could you also help review the patch? thanks > Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node > ------------------------------------------------------------------------------------------------------------------------------- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation > Affects Versions: 3.0.0-alpha2 > Reporter: zhihai xu > Assignee: zhihai xu > Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir will be called before doing log aggregation, This will be a non trivial overhead for name node in a large cluster since verifyAndCreateRemoteLogDir calls getFileStatus. Once the remote log directory is created successfully, it is not necessary to call it again. It will be better to call verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org