Return-Path: X-Original-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0E94B24A for ; Tue, 10 Jan 2012 22:28:46 +0000 (UTC) Received: (qmail 44581 invoked by uid 500); 10 Jan 2012 22:28:46 -0000 Delivered-To: apmail-hadoop-mapreduce-commits-archive@hadoop.apache.org Received: (qmail 44532 invoked by uid 500); 10 Jan 2012 22:28:46 -0000 Mailing-List: contact mapreduce-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-commits@hadoop.apache.org Received: (qmail 44524 invoked by uid 99); 10 Jan 2012 22:28:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jan 2012 22:28:45 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jan 2012 22:28:43 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 953DA23888E4; Tue, 10 Jan 2012 22:28:23 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1229768 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ hadoop-mapreduce-client/hadoop-mapreduce-client-app/... Date: Tue, 10 Jan 2012 22:28:23 -0000 To: mapreduce-commits@hadoop.apache.org From: vinodkv@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20120110222823.953DA23888E4@eris.apache.org> Author: vinodkv Date: Tue Jan 10 22:28:22 2012 New Revision: 1229768 URL: http://svn.apache.org/viewvc?rev=1229768&view=rev Log: MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv) svn merge --ignore-ancestry -c 1229766 ../../trunk/ Added: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java - copied unchanged from r1229766, hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptsInfo.java - copied unchanged from r1229766, hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptsInfo.java Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptsInfo.java hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Tue Jan 10 22:28:22 2012 @@ -105,6 +105,9 @@ Release 0.23.1 - Unreleased MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end notification URL. (Ravi Prakash via vinodkv) + MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the + job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv) + OPTIMIZATIONS MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java Tue Jan 10 22:28:22 2012 @@ -33,6 +33,7 @@ import javax.ws.rs.core.Response.Status; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapreduce.JobACL; +import org.apache.hadoop.mapreduce.v2.api.records.AMInfo; import org.apache.hadoop.mapreduce.v2.api.records.JobId; import org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId; import org.apache.hadoop.mapreduce.v2.api.records.TaskId; @@ -42,6 +43,8 @@ import org.apache.hadoop.mapreduce.v2.ap import org.apache.hadoop.mapreduce.v2.app.job.Task; import org.apache.hadoop.mapreduce.v2.app.job.TaskAttempt; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AppInfo; +import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AMAttemptInfo; +import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AMAttemptsInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.ConfInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.JobCounterInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.JobInfo; @@ -211,6 +214,21 @@ public class AMWebServices { } @GET + @Path("/jobs/{jobid}/jobattempts") + @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) + public AMAttemptsInfo getJobAttempts(@PathParam("jobid") String jid) { + + Job job = getJobFromJobIdString(jid, appCtx); + AMAttemptsInfo amAttempts = new AMAttemptsInfo(); + for (AMInfo amInfo : job.getAMInfos()) { + AMAttemptInfo attempt = new AMAttemptInfo(amInfo, MRApps.toString( + job.getID()), job.getUserName()); + amAttempts.add(attempt); + } + return amAttempts; + } + + @GET @Path("/jobs/{jobid}/counters") @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) public JobCounterInfo getJobCounters(@Context HttpServletRequest hsr, Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java Tue Jan 10 22:28:22 2012 @@ -30,6 +30,8 @@ import javax.ws.rs.ext.ContextResolver; import javax.ws.rs.ext.Provider; import javax.xml.bind.JAXBContext; +import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AMAttemptInfo; +import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AMAttemptsInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AppInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.ConfInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.ConfEntryInfo; @@ -54,22 +56,22 @@ public class JAXBContextResolver impleme private JAXBContext context; private final Set types; - + // you have to specify all the dao classes here - private final Class[] cTypes = {AppInfo.class, CounterInfo.class, - JobTaskAttemptCounterInfo.class, JobTaskCounterInfo.class, - TaskCounterGroupInfo.class, ConfInfo.class, JobCounterInfo.class, - TaskCounterInfo.class, CounterGroupInfo.class, JobInfo.class, - JobsInfo.class, ReduceTaskAttemptInfo.class, TaskAttemptInfo.class, - TaskInfo.class, TasksInfo.class, TaskAttemptsInfo.class, - ConfEntryInfo.class}; - + private final Class[] cTypes = {AMAttemptInfo.class, AMAttemptsInfo.class, + AppInfo.class, CounterInfo.class, JobTaskAttemptCounterInfo.class, + JobTaskCounterInfo.class, TaskCounterGroupInfo.class, ConfInfo.class, + JobCounterInfo.class, TaskCounterInfo.class, CounterGroupInfo.class, + JobInfo.class, JobsInfo.class, ReduceTaskAttemptInfo.class, + TaskAttemptInfo.class, TaskInfo.class, TasksInfo.class, + TaskAttemptsInfo.class, ConfEntryInfo.class}; + public JAXBContextResolver() throws Exception { this.types = new HashSet(Arrays.asList(cTypes)); this.context = new JSONJAXBContext(JSONConfiguration.natural(). rootUnwrapping(false).build(), cTypes); } - + @Override public JAXBContext getContext(Class objectType) { return (types.contains(objectType)) ? context : null; Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java Tue Jan 10 22:28:22 2012 @@ -20,6 +20,7 @@ package org.apache.hadoop.mapreduce.v2.a import static org.apache.hadoop.mapreduce.v2.app.webapp.AMParams.JOB_ID; import static org.apache.hadoop.yarn.util.StringHelper.join; +import static org.apache.hadoop.yarn.util.StringHelper.ujoin; import static org.apache.hadoop.yarn.webapp.view.JQueryUI._EVEN; import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; import static org.apache.hadoop.yarn.webapp.view.JQueryUI._ODD; @@ -28,14 +29,22 @@ import static org.apache.hadoop.yarn.web import static org.apache.hadoop.yarn.webapp.view.JQueryUI._TH; import java.util.Date; +import java.util.List; +import org.apache.hadoop.mapreduce.v2.api.records.AMInfo; import org.apache.hadoop.mapreduce.v2.api.records.JobId; import org.apache.hadoop.mapreduce.v2.app.AppContext; import org.apache.hadoop.mapreduce.v2.app.job.Job; +import org.apache.hadoop.mapreduce.v2.app.webapp.dao.AMAttemptInfo; import org.apache.hadoop.mapreduce.v2.app.webapp.dao.JobInfo; import org.apache.hadoop.mapreduce.v2.util.MRApps; import org.apache.hadoop.mapreduce.v2.util.MRApps.TaskAttemptStateUI; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.util.BuilderUtils; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; import org.apache.hadoop.yarn.webapp.view.HtmlBlock; import org.apache.hadoop.yarn.webapp.view.InfoBlock; @@ -62,6 +71,11 @@ public class JobBlock extends HtmlBlock p()._("Sorry, ", jid, " not found.")._(); return; } + + List amInfos = job.getAMInfos(); + String amString = + amInfos.size() == 1 ? "ApplicationMaster" : "ApplicationMasters"; + JobInfo jinfo = new JobInfo(job, true); info("Job Overview"). _("Job Name:", jinfo.getName()). @@ -69,10 +83,40 @@ public class JobBlock extends HtmlBlock _("Uberized:", jinfo.isUberized()). _("Started:", new Date(jinfo.getStartTime())). _("Elapsed:", StringUtils.formatTime(jinfo.getElapsedTime())); - html. + DIV div = html. _(InfoBlock.class). - div(_INFO_WRAP). + div(_INFO_WRAP); + + // MRAppMasters Table + TABLE> table = div.table("#job"); + table. + tr(). + th(amString). + _(). + tr(). + th(_TH, "Attempt Number"). + th(_TH, "Start Time"). + th(_TH, "Node"). + th(_TH, "Logs"). + _(); + for (AMInfo amInfo : amInfos) { + AMAttemptInfo attempt = new AMAttemptInfo(amInfo, + jinfo.getId(), jinfo.getUserName()); + + table.tr(). + td(String.valueOf(attempt.getAttemptId())). + td(new Date(attempt.getStartTime()).toString()). + td().a(".nodelink", url("http://", attempt.getNodeHttpAddress()), + attempt.getNodeHttpAddress())._(). + td().a(".logslink", url(attempt.getLogsLink()), + "logs")._(). + _(); + } + + table._(); + div._(); + html.div(_INFO_WRAP). // Tasks table table("#job"). tr(). Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java Tue Jan 10 22:28:22 2012 @@ -214,7 +214,7 @@ public class JobInfo { return this.state.toString(); } - public String getUser() { + public String getUserName() { return this.user; } Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java Tue Jan 10 22:28:22 2012 @@ -18,6 +18,7 @@ package org.apache.hadoop.mapreduce.v2.app.webapp; +import static org.apache.hadoop.yarn.util.StringHelper.ujoin; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; @@ -33,6 +34,7 @@ import javax.xml.parsers.DocumentBuilder import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapreduce.JobACL; +import org.apache.hadoop.mapreduce.v2.api.records.AMInfo; import org.apache.hadoop.mapreduce.v2.api.records.JobId; import org.apache.hadoop.mapreduce.v2.api.records.JobReport; import org.apache.hadoop.mapreduce.v2.app.AppContext; @@ -44,6 +46,7 @@ import org.apache.hadoop.yarn.Clock; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.util.BuilderUtils; import org.apache.hadoop.yarn.util.Times; import org.apache.hadoop.yarn.webapp.GenericExceptionHandler; import org.apache.hadoop.yarn.webapp.WebServicesTestUtils; @@ -76,6 +79,7 @@ import com.sun.jersey.test.framework.Web * /ws/v1/mapreduce/jobs * /ws/v1/mapreduce/jobs/{jobid} * /ws/v1/mapreduce/jobs/{jobid}/counters + * /ws/v1/mapreduce/jobs/{jobid}/jobattempts */ public class TestAMWebServicesJobs extends JerseyTest { @@ -777,4 +781,136 @@ public class TestAMWebServicesJobs exten } } + @Test + public void testJobAttempts() throws JSONException, Exception { + WebResource r = resource(); + Map jobsMap = appContext.getAllJobs(); + for (JobId id : jobsMap.keySet()) { + String jobId = MRApps.toString(id); + + ClientResponse response = r.path("ws").path("v1") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") + .accept(MediaType.APPLICATION_JSON).get(ClientResponse.class); + assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); + JSONObject json = response.getEntity(JSONObject.class); + assertEquals("incorrect number of elements", 1, json.length()); + JSONObject info = json.getJSONObject("jobAttempts"); + verifyJobAttempts(info, jobsMap.get(id)); + } + } + + @Test + public void testJobAttemptsSlash() throws JSONException, Exception { + WebResource r = resource(); + Map jobsMap = appContext.getAllJobs(); + for (JobId id : jobsMap.keySet()) { + String jobId = MRApps.toString(id); + + ClientResponse response = r.path("ws").path("v1") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts/") + .accept(MediaType.APPLICATION_JSON).get(ClientResponse.class); + assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); + JSONObject json = response.getEntity(JSONObject.class); + assertEquals("incorrect number of elements", 1, json.length()); + JSONObject info = json.getJSONObject("jobAttempts"); + verifyJobAttempts(info, jobsMap.get(id)); + } + } + + @Test + public void testJobAttemptsDefault() throws JSONException, Exception { + WebResource r = resource(); + Map jobsMap = appContext.getAllJobs(); + for (JobId id : jobsMap.keySet()) { + String jobId = MRApps.toString(id); + + ClientResponse response = r.path("ws").path("v1") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") + .get(ClientResponse.class); + assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); + JSONObject json = response.getEntity(JSONObject.class); + assertEquals("incorrect number of elements", 1, json.length()); + JSONObject info = json.getJSONObject("jobAttempts"); + verifyJobAttempts(info, jobsMap.get(id)); + } + } + + @Test + public void testJobAttemptsXML() throws Exception { + WebResource r = resource(); + Map jobsMap = appContext.getAllJobs(); + for (JobId id : jobsMap.keySet()) { + String jobId = MRApps.toString(id); + + ClientResponse response = r.path("ws").path("v1") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") + .accept(MediaType.APPLICATION_XML).get(ClientResponse.class); + assertEquals(MediaType.APPLICATION_XML_TYPE, response.getType()); + String xml = response.getEntity(String.class); + DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); + DocumentBuilder db = dbf.newDocumentBuilder(); + InputSource is = new InputSource(); + is.setCharacterStream(new StringReader(xml)); + Document dom = db.parse(is); + NodeList attempts = dom.getElementsByTagName("jobAttempts"); + assertEquals("incorrect number of elements", 1, attempts.getLength()); + NodeList info = dom.getElementsByTagName("jobAttempt"); + verifyJobAttemptsXML(info, jobsMap.get(id)); + } + } + + public void verifyJobAttempts(JSONObject info, Job job) + throws JSONException { + + JSONArray attempts = info.getJSONArray("jobAttempt"); + assertEquals("incorrect number of elements", 2, attempts.length()); + for (int i = 0; i < attempts.length(); i++) { + JSONObject attempt = attempts.getJSONObject(i); + verifyJobAttemptsGeneric(job, attempt.getString("nodeHttpAddress"), + attempt.getString("nodeId"), attempt.getInt("id"), + attempt.getLong("startTime"), attempt.getString("containerId"), + attempt.getString("logsLink")); + } + } + + public void verifyJobAttemptsXML(NodeList nodes, Job job) { + + assertEquals("incorrect number of elements", 2, nodes.getLength()); + for (int i = 0; i < nodes.getLength(); i++) { + Element element = (Element) nodes.item(i); + verifyJobAttemptsGeneric(job, + WebServicesTestUtils.getXmlString(element, "nodeHttpAddress"), + WebServicesTestUtils.getXmlString(element, "nodeId"), + WebServicesTestUtils.getXmlInt(element, "id"), + WebServicesTestUtils.getXmlLong(element, "startTime"), + WebServicesTestUtils.getXmlString(element, "containerId"), + WebServicesTestUtils.getXmlString(element, "logsLink")); + } + } + + public void verifyJobAttemptsGeneric(Job job, String nodeHttpAddress, + String nodeId, int id, long startTime, String containerId, String logsLink) { + boolean attemptFound = false; + for (AMInfo amInfo : job.getAMInfos()) { + if (amInfo.getAppAttemptId().getAttemptId() == id) { + attemptFound = true; + String nmHost = amInfo.getNodeManagerHost(); + int nmHttpPort = amInfo.getNodeManagerHttpPort(); + int nmPort = amInfo.getNodeManagerPort(); + WebServicesTestUtils.checkStringMatch("nodeHttpAddress", nmHost + ":" + + nmHttpPort, nodeHttpAddress); + WebServicesTestUtils.checkStringMatch("nodeId", + BuilderUtils.newNodeId(nmHost, nmPort).toString(), nodeId); + assertTrue("startime not greater than 0", startTime > 0); + WebServicesTestUtils.checkStringMatch("containerId", amInfo + .getContainerId().toString(), containerId); + + String localLogsLink = ujoin("node", "containerlogs", containerId); + + assertTrue("logsLink", logsLink.contains(localLogsLink)); + } + } + assertTrue("attempt: " + id + " was not found", attemptFound); + } + } Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java Tue Jan 10 22:28:22 2012 @@ -229,7 +229,7 @@ public class HsWebServices { } @GET - @Path("/mapreduce/jobs/{jobid}/attempts") + @Path("/mapreduce/jobs/{jobid}/jobattempts") @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) public AMAttemptsInfo getJobAttempts(@PathParam("jobid") String jid) { Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java Tue Jan 10 22:28:22 2012 @@ -30,7 +30,7 @@ import org.apache.hadoop.yarn.api.record import org.apache.hadoop.yarn.api.records.NodeId; import org.apache.hadoop.yarn.util.BuilderUtils; -@XmlRootElement(name = "amAttempt") +@XmlRootElement(name = "jobAttempt") @XmlAccessorType(XmlAccessType.FIELD) public class AMAttemptInfo { @@ -52,12 +52,14 @@ public class AMAttemptInfo { this.nodeHttpAddress = ""; this.nodeId = ""; String nmHost = amInfo.getNodeManagerHost(); - int nmPort = amInfo.getNodeManagerHttpPort(); + int nmHttpPort = amInfo.getNodeManagerHttpPort(); + int nmPort = amInfo.getNodeManagerPort(); if (nmHost != null) { - this.nodeHttpAddress = nmHost + ":" + nmPort; + this.nodeHttpAddress = nmHost + ":" + nmHttpPort; NodeId nodeId = BuilderUtils.newNodeId(nmHost, nmPort); this.nodeId = nodeId.toString(); } + this.id = amInfo.getAppAttemptId().getAttemptId(); this.startTime = amInfo.getStartTime(); this.containerId = ""; Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptsInfo.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptsInfo.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptsInfo.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptsInfo.java Tue Jan 10 22:28:22 2012 @@ -21,12 +21,14 @@ import java.util.ArrayList; import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; +import javax.xml.bind.annotation.XmlElement; import javax.xml.bind.annotation.XmlRootElement; -@XmlRootElement(name = "attempts") +@XmlRootElement(name = "jobAttempts") @XmlAccessorType(XmlAccessType.FIELD) public class AMAttemptsInfo { + @XmlElement(name = "jobAttempt") protected ArrayList attempt = new ArrayList(); public AMAttemptsInfo() { Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java?rev=1229768&r1=1229767&r2=1229768&view=diff ============================================================================== --- hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java Tue Jan 10 22:28:22 2012 @@ -77,7 +77,7 @@ import com.sun.jersey.test.framework.Web * * /ws/v1/history/mapreduce/jobs /ws/v1/history/mapreduce/jobs/{jobid} * /ws/v1/history/mapreduce/jobs/{jobid}/counters - * /ws/v1/history/mapreduce/jobs/{jobid}/attempts + * /ws/v1/history/mapreduce/jobs/{jobid}/jobattempts */ public class TestHsWebServicesJobs extends JerseyTest { @@ -626,12 +626,12 @@ public class TestHsWebServicesJobs exten String jobId = MRApps.toString(id); ClientResponse response = r.path("ws").path("v1").path("history") - .path("mapreduce").path("jobs").path(jobId).path("attempts") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") .accept(MediaType.APPLICATION_JSON).get(ClientResponse.class); assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); JSONObject json = response.getEntity(JSONObject.class); assertEquals("incorrect number of elements", 1, json.length()); - JSONObject info = json.getJSONObject("attempts"); + JSONObject info = json.getJSONObject("jobAttempts"); verifyHsJobAttempts(info, jobsMap.get(id)); } } @@ -644,12 +644,12 @@ public class TestHsWebServicesJobs exten String jobId = MRApps.toString(id); ClientResponse response = r.path("ws").path("v1").path("history") - .path("mapreduce").path("jobs").path(jobId).path("attempts/") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts/") .accept(MediaType.APPLICATION_JSON).get(ClientResponse.class); assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); JSONObject json = response.getEntity(JSONObject.class); assertEquals("incorrect number of elements", 1, json.length()); - JSONObject info = json.getJSONObject("attempts"); + JSONObject info = json.getJSONObject("jobAttempts"); verifyHsJobAttempts(info, jobsMap.get(id)); } } @@ -662,12 +662,12 @@ public class TestHsWebServicesJobs exten String jobId = MRApps.toString(id); ClientResponse response = r.path("ws").path("v1").path("history") - .path("mapreduce").path("jobs").path(jobId).path("attempts") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") .get(ClientResponse.class); assertEquals(MediaType.APPLICATION_JSON_TYPE, response.getType()); JSONObject json = response.getEntity(JSONObject.class); assertEquals("incorrect number of elements", 1, json.length()); - JSONObject info = json.getJSONObject("attempts"); + JSONObject info = json.getJSONObject("jobAttempts"); verifyHsJobAttempts(info, jobsMap.get(id)); } } @@ -680,7 +680,7 @@ public class TestHsWebServicesJobs exten String jobId = MRApps.toString(id); ClientResponse response = r.path("ws").path("v1").path("history") - .path("mapreduce").path("jobs").path(jobId).path("attempts") + .path("mapreduce").path("jobs").path(jobId).path("jobattempts") .accept(MediaType.APPLICATION_XML).get(ClientResponse.class); assertEquals(MediaType.APPLICATION_XML_TYPE, response.getType()); String xml = response.getEntity(String.class); @@ -689,9 +689,9 @@ public class TestHsWebServicesJobs exten InputSource is = new InputSource(); is.setCharacterStream(new StringReader(xml)); Document dom = db.parse(is); - NodeList attempts = dom.getElementsByTagName("attempts"); + NodeList attempts = dom.getElementsByTagName("jobAttempts"); assertEquals("incorrect number of elements", 1, attempts.getLength()); - NodeList info = dom.getElementsByTagName("attempt"); + NodeList info = dom.getElementsByTagName("jobAttempt"); verifyHsJobAttemptsXML(info, jobsMap.get(id)); } } @@ -699,7 +699,7 @@ public class TestHsWebServicesJobs exten public void verifyHsJobAttempts(JSONObject info, Job job) throws JSONException { - JSONArray attempts = info.getJSONArray("attempt"); + JSONArray attempts = info.getJSONArray("jobAttempt"); assertEquals("incorrect number of elements", 2, attempts.length()); for (int i = 0; i < attempts.length(); i++) { JSONObject attempt = attempts.getJSONObject(i); @@ -732,9 +732,10 @@ public class TestHsWebServicesJobs exten if (amInfo.getAppAttemptId().getAttemptId() == id) { attemptFound = true; String nmHost = amInfo.getNodeManagerHost(); - int nmPort = amInfo.getNodeManagerHttpPort(); + int nmHttpPort = amInfo.getNodeManagerHttpPort(); + int nmPort = amInfo.getNodeManagerPort(); WebServicesTestUtils.checkStringMatch("nodeHttpAddress", nmHost + ":" - + nmPort, nodeHttpAddress); + + nmHttpPort, nodeHttpAddress); WebServicesTestUtils.checkStringMatch("nodeId", BuilderUtils.newNodeId(nmHost, nmPort).toString(), nodeId); assertTrue("startime not greater than 0", startTime > 0);