hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-195) transfer map output transfer with http instead of rpc
Date Tue, 09 May 2006 17:41:05 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-195?page=comments#action_12378670 ] 

Owen O'Malley commented on HADOOP-195:
--------------------------------------

btw, here is the timeline:

060508 145202 Running job: job_0002
060508 145203  map 0%  reduce 0%
060508 145316  map 1%  reduce 0%
060508 145435  map 2%  reduce 0%
060508 145553  map 3%  reduce 0%
060508 145704  map 4%  reduce 0%
060508 145822  map 5%  reduce 0%
060508 145934  map 6%  reduce 0%
060508 150045  map 7%  reduce 0%
060508 150203  map 8%  reduce 0%
060508 150323  map 9%  reduce 0%
060508 150435  map 10%  reduce 0%
060508 150551  map 11%  reduce 0%
060508 150704  map 12%  reduce 0%
060508 150824  map 13%  reduce 0%
060508 150942  map 14%  reduce 0%
060508 151055  map 15%  reduce 0%
060508 151211  map 16%  reduce 0%
060508 151326  map 17%  reduce 0%
060508 151446  map 18%  reduce 0%
060508 151606  map 19%  reduce 0%
060508 151731  map 20%  reduce 0%
060508 151847  map 21%  reduce 0%
060508 152006  map 22%  reduce 0%
060508 152127  map 23%  reduce 0%
060508 152246  map 24%  reduce 0%
060508 152407  map 25%  reduce 0%
060508 152526  map 26%  reduce 0%
060508 152645  map 27%  reduce 0%
060508 152806  map 28%  reduce 0%
060508 152926  map 29%  reduce 0%
060508 153054  map 30%  reduce 0%
060508 153112  map 30%  reduce 1%
060508 153214  map 31%  reduce 1%
060508 153336  map 32%  reduce 1%
060508 153457  map 33%  reduce 1%
060508 153616  map 34%  reduce 1%
060508 153736  map 35%  reduce 1%
060508 153903  map 36%  reduce 1%
060508 154023  map 37%  reduce 1%
060508 154143  map 38%  reduce 1%
060508 154303  map 39%  reduce 1%
060508 154424  map 40%  reduce 1%
060508 154544  map 41%  reduce 1%
060508 154705  map 42%  reduce 1%
060508 154825  map 43%  reduce 1%
060508 154944  map 44%  reduce 1%
060508 155104  map 45%  reduce 1%
060508 155223  map 46%  reduce 1%
060508 155345  map 47%  reduce 1%
060508 155505  map 48%  reduce 1%
060508 155624  map 49%  reduce 1%
060508 155745  map 50%  reduce 1%
060508 155906  map 51%  reduce 1%
060508 160027  map 52%  reduce 1%
060508 160146  map 53%  reduce 1%
060508 160305  map 54%  reduce 1%
060508 160427  map 55%  reduce 1%
060508 160546  map 56%  reduce 1%
060508 160712  map 57%  reduce 1%
060508 160833  map 58%  reduce 1%
060508 160953  map 59%  reduce 1%
060508 161102  map 59%  reduce 2%
060508 161115  map 60%  reduce 2%
060508 161236  map 61%  reduce 2%
060508 161403  map 62%  reduce 2%
060508 161523  map 63%  reduce 2%
060508 161642  map 64%  reduce 2%
060508 161804  map 65%  reduce 2%
060508 161925  map 66%  reduce 2%
060508 162044  map 67%  reduce 2%
060508 162203  map 68%  reduce 2%
060508 162323  map 69%  reduce 2%
060508 162445  map 70%  reduce 2%
060508 162607  map 71%  reduce 2%
060508 162725  map 72%  reduce 2%
060508 162844  map 73%  reduce 2%
060508 163005  map 74%  reduce 2%
060508 163124  map 75%  reduce 2%
060508 163246  map 76%  reduce 2%
060508 163406  map 77%  reduce 2%
060508 163527  map 78%  reduce 2%
060508 163646  map 79%  reduce 2%
060508 163814  map 80%  reduce 2%
060508 163936  map 81%  reduce 2%
060508 164055  map 82%  reduce 2%
060508 164215  map 83%  reduce 2%
060508 164336  map 84%  reduce 2%
060508 164456  map 85%  reduce 2%
060508 164617  map 86%  reduce 2%
060508 164736  map 87%  reduce 2%
060508 164857  map 88%  reduce 2%
060508 165017  map 89%  reduce 2%
060508 165144  map 90%  reduce 2%
060508 165303  map 91%  reduce 2%
060508 165355  map 91%  reduce 3%
060508 165424  map 92%  reduce 3%
060508 165544  map 93%  reduce 3%
060508 165704  map 94%  reduce 3%
060508 165825  map 95%  reduce 3%
060508 165947  map 96%  reduce 3%
060508 170106  map 97%  reduce 3%
060508 170224  map 98%  reduce 3%
060508 170336  map 99%  reduce 3%
060508 170454  map 100%  reduce 3%
060508 171720  map 100%  reduce 4%
060508 173342  map 100%  reduce 5%
060508 175008  map 100%  reduce 6%
060508 180613  map 100%  reduce 7%
060508 182222  map 100%  reduce 8%
060508 183822  map 100%  reduce 9%
060508 185446  map 100%  reduce 10%
060508 191056  map 100%  reduce 11%
060508 192715  map 100%  reduce 12%
060508 194330  map 100%  reduce 13%
060508 195923  map 100%  reduce 14%
060508 201521  map 100%  reduce 15%
060508 203111  map 100%  reduce 16%
060508 204638  map 100%  reduce 17%
060508 210211  map 100%  reduce 18%
060508 211742  map 100%  reduce 19%
060508 213258  map 100%  reduce 20%
060508 214824  map 100%  reduce 21%
060508 220344  map 100%  reduce 22%
060508 221856  map 100%  reduce 23%
060508 223424  map 100%  reduce 24%
060508 224715  map 100%  reduce 25%
060508 225618  map 100%  reduce 26%
060508 230309  map 100%  reduce 27%
060508 230842  map 100%  reduce 28%
060508 231336  map 100%  reduce 29%
060508 231800  map 100%  reduce 30%
060508 232215  map 100%  reduce 31%
060508 232613  map 100%  reduce 32%
060508 233012  map 100%  reduce 33%
060508 233410  map 100%  reduce 34%
060508 233806  map 100%  reduce 35%
060508 234202  map 100%  reduce 36%
060508 234557  map 100%  reduce 37%
060508 234948  map 100%  reduce 38%
060508 235344  map 100%  reduce 39%
060508 235744  map 100%  reduce 40%
060509 000132  map 100%  reduce 41%
060509 000533  map 100%  reduce 42%
060509 000932  map 100%  reduce 43%
060509 001347  map 100%  reduce 44%
060509 001822  map 100%  reduce 45%
060509 002355  map 100%  reduce 46%
060509 003007  map 100%  reduce 47%
060509 003723  map 100%  reduce 48%
060509 004651  map 100%  reduce 49%
060509 010213  map 100%  reduce 50%
060509 011506  map 100%  reduce 51%
060509 012217  map 100%  reduce 52%
060509 012730  map 100%  reduce 53%
060509 013027  map 100%  reduce 54%
060509 013259  map 100%  reduce 55%
060509 013606  map 100%  reduce 56%
060509 013913  map 100%  reduce 57%
060509 014153  map 100%  reduce 58%
060509 014350  map 100%  reduce 59%
060509 014557  map 100%  reduce 60%
060509 014854  map 100%  reduce 61%
060509 015056  map 100%  reduce 62%
060509 015356  map 100%  reduce 63%
060509 015638  map 100%  reduce 64%
060509 015947  map 100%  reduce 65%
060509 020204  map 100%  reduce 66%
060509 020543  map 100%  reduce 67%
060509 020728  map 100%  reduce 68%
060509 021001  map 100%  reduce 69%
060509 021200  map 100%  reduce 70%
060509 021330  map 100%  reduce 71%
060509 021502  map 100%  reduce 72%
060509 021703  map 100%  reduce 73%
060509 022007  map 100%  reduce 74%
060509 022301  map 100%  reduce 75%
060509 022609  map 100%  reduce 76%
060509 022940  map 100%  reduce 77%
060509 023324  map 100%  reduce 78%
060509 023642  map 100%  reduce 77%
060509 023723  map 100%  reduce 76%
060509 023746  map 100%  reduce 77%
060509 023800  map 100%  reduce 76%
060509 023820  map 100%  reduce 77%
060509 024342  map 100%  reduce 78%
060509 024559  map 100%  reduce 77%
060509 024602  map 100%  reduce 78%
060509 024845  map 100%  reduce 79%
060509 025201  map 100%  reduce 80%
060509 025453  map 99%  reduce 80%
060509 025523  map 100%  reduce 80%
060509 025645  map 100%  reduce 81%
060509 030103  map 100%  reduce 82%
060509 030454  map 100%  reduce 83%
060509 030844  map 100%  reduce 84%
060509 031401  map 100%  reduce 85%
060509 031924  map 100%  reduce 86%
060509 032352  map 100%  reduce 87%
060509 032927  map 100%  reduce 88%
060509 033737  map 100%  reduce 89%
060509 034300  map 100%  reduce 90%
060509 035312  map 100%  reduce 91%
060509 040142  map 100%  reduce 92%
060509 041755  map 100%  reduce 93%
060509 043246  map 100%  reduce 94%
060509 050725  map 100%  reduce 95%
060509 060826  map 100%  reduce 96%
060509 065234  map 100%  reduce 97%
060509 074655  map 100%  reduce 98%
060509 075706  map 100%  reduce 99%
060509 081125  map 100%  reduce 100%
060509 092825 Job complete: job_0002
Job ended: Tue May 09 09:28:25 PDT 2006
The job took 66983 seconds.


> transfer map output transfer with http instead of rpc
> -----------------------------------------------------
>
>          Key: HADOOP-195
>          URL: http://issues.apache.org/jira/browse/HADOOP-195
>      Project: Hadoop
>         Type: Improvement

>   Components: mapred
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3
>  Attachments: netstat.log, netstat.xls
>
> The data transfer of the map output should be transfered via http instead rpc, because
rpc is very slow for this application and the timeout behavior is suboptimal. (server sends
data and client ignores it because it took more than 10 seconds to be received.)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message