Return-Path: X-Original-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5BA1462B1 for ; Mon, 16 May 2011 09:24:30 +0000 (UTC) Received: (qmail 54860 invoked by uid 500); 16 May 2011 09:24:30 -0000 Delivered-To: apmail-hadoop-mapreduce-commits-archive@hadoop.apache.org Received: (qmail 54811 invoked by uid 500); 16 May 2011 09:24:29 -0000 Mailing-List: contact mapreduce-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-commits@hadoop.apache.org Received: (qmail 54798 invoked by uid 99); 16 May 2011 09:24:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 09:24:29 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 09:24:26 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 4B49C2388980; Mon, 16 May 2011 09:24:05 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1103657 - in /hadoop/mapreduce/branches/MR-279: CHANGES.txt mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java Date: Mon, 16 May 2011 09:24:05 -0000 To: mapreduce-commits@hadoop.apache.org From: vinodkv@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110516092405.4B49C2388980@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: vinodkv Date: Mon May 16 09:24:04 2011 New Revision: 1103657 URL: http://svn.apache.org/viewvc?rev=1103657&view=rev Log: Launching bin/yarn and bin/mapred only *once* in AM for constructing classpaths to avoid multiple forks and huge vmem usage by AM. Contributed by Vinod Kumar Vavilapalli. Modified: hadoop/mapreduce/branches/MR-279/CHANGES.txt hadoop/mapreduce/branches/MR-279/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java Modified: hadoop/mapreduce/branches/MR-279/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/mapreduce/branches/MR-279/CHANGES.txt?rev=1103657&r1=1103656&r2=1103657&view=diff ============================================================================== --- hadoop/mapreduce/branches/MR-279/CHANGES.txt (original) +++ hadoop/mapreduce/branches/MR-279/CHANGES.txt Mon May 16 09:24:04 2011 @@ -4,6 +4,9 @@ Trunk (unreleased changes) MAPREDUCE-279 + Launching bin/yarn and bin/mapred only *once* in AM for constructing + classpaths to avoid multiple forks and huge vmem usage by AM. (vinodkv) + Fix calculation of maximum capacity in ParentQueue to use its parent's absolute-capacity rather than its own absolute-capacity. (acmurthy) Modified: hadoop/mapreduce/branches/MR-279/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java URL: http://svn.apache.org/viewvc/hadoop/mapreduce/branches/MR-279/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java?rev=1103657&r1=1103656&r2=1103657&view=diff ============================================================================== --- hadoop/mapreduce/branches/MR-279/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java (original) +++ hadoop/mapreduce/branches/MR-279/mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java Mon May 16 09:24:04 2011 @@ -29,6 +29,7 @@ import java.util.EnumSet; import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; @@ -143,6 +144,8 @@ public abstract class TaskAttemptImpl im private final Lock writeLock; private Collection> fsTokens; private Token jobToken; + private AtomicBoolean initialEnvFlag = new AtomicBoolean(); + private Map initialEnv = null; private long launchTime; private long finishTime; @@ -451,6 +454,23 @@ public abstract class TaskAttemptImpl im } /** + * Lock this on initialEnv so that there is only one fork in the AM for + * getting the initial class-path. TODO: This should go away once we construct + * a parent CLC and use it for all the containers. + */ + private Map getInitialEnv() throws IOException { + synchronized (initialEnvFlag) { + if (initialEnvFlag.get()) { + return initialEnv; + } + initialEnv = new HashMap(); + MRApps.setInitialClasspath(initialEnv); + initialEnvFlag.set(true); + return initialEnv; + } + } + + /** * Create the {@link ContainerLaunchContext} for this attempt. */ private ContainerLaunchContext createContainerLaunchContext() { @@ -524,9 +544,7 @@ public abstract class TaskAttemptImpl im ByteBuffer.wrap(jobToken_dob.getData(), 0, jobToken_dob.getLength())); - Map env = new HashMap(); - MRApps.setInitialClasspath(env); - container.addAllEnv(env); + container.addAllEnv(getInitialEnv()); } catch (IOException e) { throw new YarnException(e); }