From yarn-issues-return-136757-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Jan 31 17:09:33 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 944F9180662 for ; Wed, 31 Jan 2018 17:09:33 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 844FE160C55; Wed, 31 Jan 2018 16:09:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CB2A6160C25 for ; Wed, 31 Jan 2018 17:09:32 +0100 (CET) Received: (qmail 61605 invoked by uid 500); 31 Jan 2018 16:09:31 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 61594 invoked by uid 99); 31 Jan 2018 16:09:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Jan 2018 16:09:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 468AE198BDE for ; Wed, 31 Jan 2018 16:09:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id DZvual6MbI9N for ; Wed, 31 Jan 2018 16:09:30 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 7FF4A61280 for ; Wed, 31 Jan 2018 16:09:12 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5A05FE0231 for ; Wed, 31 Jan 2018 16:09:10 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4148224122 for ; Wed, 31 Jan 2018 16:09:06 +0000 (UTC) Date: Wed, 31 Jan 2018 16:09:06 +0000 (UTC) From: "Hudson (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2185) Use pipes when localizing archives MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347067#comment-16347067 ] Hudson commented on YARN-2185: ------------------------------ SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13588 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13588/]) YARN-2185. Use pipes when localizing archives. Contributed by Miklos (jlowe: rev 7288b08330001102f944124d18e02bf0585d3e96) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java > Use pipes when localizing archives > ---------------------------------- > > Key: YARN-2185 > URL: https://issues.apache.org/jira/browse/YARN-2185 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.4.0 > Reporter: Jason Lowe > Assignee: Miklos Szegedi > Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-2185.000.patch, YARN-2185.001.patch, YARN-2185.002.patch, YARN-2185.003.patch, YARN-2185.004.patch, YARN-2185.005.patch, YARN-2185.006.patch, YARN-2185.007.patch, YARN-2185.008.patch, YARN-2185.009.patch, YARN-2185.010.patch, YARN-2185.011.patch, YARN-2185.012.patch, YARN-2185.012.patch, YARN-2185.013.patch, YARN-2185.014.patch > > > Currently the nodemanager downloads an archive to a local file, unpacks it, and then removes it. It would be more efficient to stream the data as it's being unpacked to avoid both the extra disk space requirements and the additional disk activity from storing the archive. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org