Return-Path: X-Original-To: apmail-incubator-mesos-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-mesos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF45FE398 for ; Fri, 25 Jan 2013 19:20:02 +0000 (UTC) Received: (qmail 16569 invoked by uid 500); 25 Jan 2013 19:20:02 -0000 Delivered-To: apmail-incubator-mesos-dev-archive@incubator.apache.org Received: (qmail 16536 invoked by uid 500); 25 Jan 2013 19:20:02 -0000 Mailing-List: contact mesos-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mesos-dev@incubator.apache.org Delivered-To: mailing list mesos-dev@incubator.apache.org Received: (qmail 16512 invoked by uid 99); 25 Jan 2013 19:20:02 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jan 2013 19:20:02 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id D23041C6AB0; Fri, 25 Jan 2013 19:19:55 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2146245547904957785==" MIME-Version: 1.0 Subject: Re: Review Request: Improved Hadoop port for mesos From: "Vinod Kone" To: "Benjamin Hindman" , "Ben Mahler" Cc: "mesos" , "Vinod Kone" Date: Fri, 25 Jan 2013 19:19:55 -0000 Message-ID: <20130125191955.5634.32105@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Vinod Kone" X-ReviewGroup: mesos X-ReviewRequest-URL: https://reviews.apache.org/r/8965/ X-Sender: "Vinod Kone" References: <20130125040435.5634.68912@reviews.apache.org> In-Reply-To: <20130125040435.5634.68912@reviews.apache.org> Reply-To: "Vinod Kone" --===============2146245547904957785== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable > On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lin= e 236 > > > > > > Could also be expired. if its expired, this call wouldn't be made by the jobtracker! > On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lin= e 237 > > > > > > so null is handled? > > = > > seems like it would expect an empty list returned intead.. no, null is handled correctly by the jobtracker. > On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lin= e 587 > > > > > > I think a switch (with all the states) is clearer here, but up to y= ou. 'if' seems more appropriate here. > On Jan. 25, 2013, 4:04 a.m., Ben Mahler wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lin= e 304 > > > > > > I think the logic here is incorrect on this latest review. > > = > > We want to know about the "available" slots. That is, slots that ar= e not currently in use. (status.getAvailableMapSlots()). > > = > > Whereas here we consider all launched slots as available, regardles= s of whether or not they are in use. > > = > > So.. to fix this: > > kill the launched vars here > > = > > and the loop above becomes: > > = > > idleMapSlots =3D 0; > > idleReduceSlots =3D 0; > > for (TaskTrackerStatus status : jobTracker.taskTrackers()) { > > HttpHost host =3D new HttpHost(status.getHost(), status.getHt= tpPort()); > > if (mesosTrackers.containsKey(host)) { > > mesosTrackers.get(host).active =3D true; > > idleMapSlots +=3D status.getAvailableMapSlots(); > > idleReduceSlots +=3D status.getAvailableReduceSlots(); > > } = > > } > > = > > make sense? makes total sense. thank you. - Vinod ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8965/#review15674 ----------------------------------------------------------- On Jan. 25, 2013, 2:32 a.m., Vinod Kone wrote: > = > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8965/ > ----------------------------------------------------------- > = > (Updated Jan. 25, 2013, 2:32 a.m.) > = > = > Review request for mesos, Benjamin Hindman and Ben Mahler. > = > = > Description > ------- > = > Features: > = > --> Mesos scheduler is a thin shim that delegates the actual scheduling o= f task trackers to the underlying scheduler. > --> Mesos executor is a thin shim that starts/stops task tracker properly. > --> Doesn't depend on mesos specific patches (status updates, task tracke= r instrumentation) for hadoop. > --> Builds mesos executor bundle that can be used when running on a real = mesos cluster. > = > = > Diffs > ----- > = > hadoop/Makefile.am 46d16f7c03cda217d1851d78515cad96f60cba2b = > hadoop/NOTES PRE-CREATION = > hadoop/TUTORIAL.sh fca22c5b7bd485a5d18c99cb6666fab6bb0a06bb = > hadoop/hadoop-0.20.2-cdh3u3.patch 84a948f23122a25fb41dfe497a73bff0c8e08= 67d = > hadoop/hadoop-0.20.2-cdh3u3_hadoop-env.sh.patch b1731e27b1f59d8dc913d25= 9d8f32a69f46ac0b0 = > hadoop/hadoop-0.20.2-cdh3u3_mesos.patch f93e53e286525829bdc2a3da71eefab= 8f093e5b8 = > hadoop/hadoop-0.20.205.0.patch cab488e402343bdbd5d0c4101785b9a48331f85e = > hadoop/hadoop-0.20.205.0_hadoop-env.sh.patch 1b8f4d06323a3ca3f624d93740= 66e29be5a18acd = > hadoop/hadoop-7698-1.patch PRE-CREATION = > hadoop/mapred-site.xml.patch c2d6397d8fc1d662afd606e1069ff469c3556e19 = > hadoop/mesos-executor ce8dcccf7eaecd35dfa041a78aa9760dc2ce451c = > hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkExecutor.java 9= 238fe53a718c47ff4bbe2d555d1c072fa8bd6e2 = > hadoop/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java = 754b198a5c7baacc92ce2adc96da15aa97c47ad1 = > hadoop/mesos/src/java/org/apache/hadoop/mapred/HadoopFrameworkMessage.j= ava 1e9510873bab962250afbaf81a5c63a6b191e299 = > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosExecutor.java PRE-C= REATION = > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java 1408= f300a43eab043b40e4d313e6632cb21fd7b2 = > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrume= ntation.java 8c517d80e54555e12f906a219b12eb572ed7e3a3 = > = > Diff: https://reviews.apache.org/r/8965/diff/ > = > = > Testing > ------- > = > make hadoop-0.20.2-cdh3u3 > make hadoop-0.20.205.0 > = > Also, verified that the generator executor bundle is self-contained and w= orks as expected for non-local runs. > = > = > Thanks, > = > Vinod Kone > = > --===============2146245547904957785==--