Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Tue, 12 Nov 2013 22:17:24 +0000 (UTC)
From: "Kousuke Saruta (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12676497.1383083074409.60484.1384294644713@arcas>
In-Reply-To: <JIRA.12676497.1383083074409@arcas>
References: <JIRA.12676497.1383083074409@arcas>
Subject: [jira] [Commented] (MAPREDUCE-5600) ConcurrentModificationException
 on /tasktracker.jsp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820550#comment-13820550 ] 

Kousuke Saruta commented on MAPREDUCE-5600:
-------------------------------------------

I think branch-1 has the same issue.

When TaskTracker#getTasksFromRunningJobs is called, we'll get a iterator of "runningJobs"

{code}
  /**
   * Get the list of tasks from running jobs on this task tracker.
   * @return a copy of the list of TaskStatus objects
   */
  synchronized List<TaskStatus> getTasksFromRunningJobs() {
    List<TaskStatus> result = new ArrayList<TaskStatus>(tasks.size());
    for (Map.Entry <JobID, RunningJob> item : runningJobs.entrySet()) {
          ...
      }
    }
    return result;
  }
{code}

On the other hand, TaskTracker#addTaskToJob can be called during iteration.
addTaskToJob method modifies "runningJob" so it can cause ConcurrentModificationException.


{code}
  private RunningJob addTaskToJob(JobID jobId,
                                  TaskInProgress tip) {
    synchronized (runningJobs) {
       ...
        runningJobs.put(jobId, rJob);
       ...
    }     
  }
{code}

When we call getTasksFromRunningJobs, we get a synchronized monitor for the instance of TaskTracker, but, when we call addTaskToJob,
we get a synchronized monitor for "runningJob". So, we may need to modify TaskTracker to get an appropriate monitor when calling getTasksFromRunningJobs. 

> ConcurrentModificationException on /tasktracker.jsp
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-5600
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5600
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>         Environment: Found this in the MR1 bundled with hadoop-2.0.0-mr1-cdh4.1.3 (which I think is based on some 0.20 version).
>            Reporter: Benoit Sigoure
>
> If you request {{/tasktracker.jsp}} frequently on a TaskTracker that's busy, every once in a while you'll get this:
> {code}
> 2013-10-29 13:25:55,524 ERROR org.mortbay.log: /tasktracker.jsp
> java.util.ConcurrentModificationException
>         at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117)
>         at java.util.TreeMap$EntryIterator.next(TreeMap.java:1153)
>         at java.util.TreeMap$EntryIterator.next(TreeMap.java:1148)
>         at org.apache.hadoop.mapred.TaskTracker.getTasksFromRunningJobs(TaskTracker.java:3991)
>         at org.apache.hadoop.mapred.tasktracker_jsp._jspService(tasktracker_jsp.java:98)
>         at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>         at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1056)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}


--
This message was sent by Atlassian JIRA
(v6.1#6144)