hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
Date Mon, 15 Feb 2010 15:13:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833834#action_12833834

Rajesh Balamohan commented on MAPREDUCE-1495:

As of now, its implemented as follows in JobTracker

public synchronized TaskCompletionEvent[] getTaskCompletionEvents(
      JobID jobid, int fromEventId, int maxEvents) throws IOException{
    synchronized (this) {
      JobInProgress job = this.jobs.get(jobid);
      if (null != job) {
        if (job.inited()) {
          return job.getTaskCompletionEvents(fromEventId, maxEvents);
        } else {
          return EMPTY_EVENTS;
    return completedJobStatusStore.readJobTaskCompletionEvents(jobid, fromEventId, maxEvents);

Where, jobs is TreeMap<JobID, JobInProgress>(). 

It is possible to reduce to contention in 2 ways.

1. Reduce the synch to only JobInProgress job = this.jobs.get(jobid); Rest of the code is
independent of the synch block (asaik).
2. Change datastructure of jobs to ConcurrentHashMap<JobID, JobInProgress>(). This way,
we can jobs.get(jobid) automatically becomes threadsafe and the total synchornization itself
can be eliminated. If its mandatory to maintain the order, I have to try the 1st  one.  

> Reduce locking contention on JobTracker.getTaskCompletionEvents()
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-1495
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Rajesh Balamohan
> While profiling JT for slow performance with small-jobs, it was observed that JobTracker.getTaskCompletionEvents()
is attributing to 40% of lock contention on JT.
> This JIRA ticket is created to explore the possibilities of reducing the sychronized
code block in this method. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message