hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Binglin Chang (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3612) Task.TaskReporter.done method blocked for some time when task is finishing
Date Tue, 03 Jan 2012 12:14:39 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Binglin Chang updated MAPREDUCE-3612:
-------------------------------------

         Description: 
We recently have done some tests to evaluate performances of different Hadoop versions(1.0,
0.23, Baidu internal version), and found some weird results. One of them is in 1.0 Task.TaskReporter.done()
takes too much time, about 2s, this is bad for small tasks. After reviewing source code and
add some log, the following code block Task.TaskReporter.done


{code:title=src/mapred/org/apache/hadoop/mapred/Task.java}
 658           try {
 659             Thread.sleep(PROGRESS_INTERVAL);
 660           }


 723     public void stopCommunicationThread() throws InterruptedException {
 724       // Updating resources specified in ResourceCalculatorPlugin
 725       if (pingThread != null) {
 726         synchronized(lock) {
 727           while(!done) {
 728             lock.wait();
 729           }
 730         }
 731         pingThread.interrupt();
 732         pingThread.join();
 733       }
 734     }
{code}
Originally line 724-730 don't exists, I don't know why it is added. If it is needed, we can
replace Thread.sleep with Object.wait(timeout) and Object.notify instead, so it won't block.




  was:
We recently have done some tests to evaluate performances of different Hadoop versions(1.0,
0.23, Baidu internal version), and found some weird results. One of them is in 1.0 Task.TaskReporter.done()
takes too much time, about 2s, this is bad for small tasks. After reviewing source code and
add some log, the following code block Task.TaskReporter.done

src/mapred/org/apache/hadoop/mapred/Task.java

 658           try {
 659             Thread.sleep(PROGRESS_INTERVAL);
 660           }


 723     public void stopCommunicationThread() throws InterruptedException {
 724       // Updating resources specified in ResourceCalculatorPlugin
 725       if (pingThread != null) {
 726         synchronized(lock) {
 727           while(!done) {
 728             lock.wait();
 729           }
 730         }
 731         pingThread.interrupt();
 732         pingThread.join();
 733       }
 734     }

Originally line 724-730 don't exists, I don't know why it is added. If it is needed, we can
replace Thread.sleep with Object.wait(timeout) and Object.notify instead, so it won't block.




    Target Version/s: 1.0.0, 0.20.205.0  (was: 0.20.205.0, 1.0.0)
    
> Task.TaskReporter.done method blocked for some time when task is finishing
> --------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3612
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3612
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Binglin Chang
>
> We recently have done some tests to evaluate performances of different Hadoop versions(1.0,
0.23, Baidu internal version), and found some weird results. One of them is in 1.0 Task.TaskReporter.done()
takes too much time, about 2s, this is bad for small tasks. After reviewing source code and
add some log, the following code block Task.TaskReporter.done
> {code:title=src/mapred/org/apache/hadoop/mapred/Task.java}
>  658           try {
>  659             Thread.sleep(PROGRESS_INTERVAL);
>  660           }
>  723     public void stopCommunicationThread() throws InterruptedException {
>  724       // Updating resources specified in ResourceCalculatorPlugin
>  725       if (pingThread != null) {
>  726         synchronized(lock) {
>  727           while(!done) {
>  728             lock.wait();
>  729           }
>  730         }
>  731         pingThread.interrupt();
>  732         pingThread.join();
>  733       }
>  734     }
> {code}
> Originally line 724-730 don't exists, I don't know why it is added. If it is needed,
we can replace Thread.sleep with Object.wait(timeout) and Object.notify instead, so it won't
block.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message