hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Noguchi <knogu...@yahoo-inc.com>
Subject Re: two servers nutch hadoop
Date Mon, 30 Apr 2007 17:00:53 GMT
Hi.

Could you check if you have a 'space' in the mapred.local.dir config?

<value>[space]/home/nutch/fs/mapred/local</value>

has resulted in java.io.FileNotFoundException: in hadoop-0.12.3.

Error message is a little different from yours, so it might not be related.

2007-04-30 16:42:23,441 WARN org.apache.hadoop.mapred.TaskRunner: 
java.io.FileNotFoundException: 
http://aaa.bbb.com:11111/mapOutput?map=task_0001_m_000000_2&reduce=0
  at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1243)
  at 
org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:200)
  at 
org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.copyOutput(ReduceTaskRunner.java:311)
  at 
org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.run(ReduceTaskRunner.java:274)

Filed a Jira.
https://issues.apache.org/jira/browse/HADOOP-1305


Koji

derevo wrote:
> hi, when i try
>
> [nutch@master ~/search]$ bin/nutch crawl urls -dir crawled -depth 3
>
> i have this error in log file 
>
> master server
>
> ==> hadoop-nutch-tasktracker-master.log <==
> 2007-04-30 08:53:52,318 WARN  mapred.TaskRunner - task_0002_r_000000_0 copy
> failed: task_0002_m_000004_0 from slave1
> 2007-04-30 08:53:52,319 WARN  mapred.TaskRunner - java.io.IOException: File
> /home/nutch/fs/mapred/local/task_0002_r_000000_0/map_4.out-6 not created
>         at
> org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.copyOutput(ReduceTaskRunner.java:322)
>         at
> org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.run(ReduceTaskRunner.java:274)
>
> 2007-04-30 08:53:52,319 WARN  mapred.TaskRunner - task_0002_r_000000_0
> adding host slave1 to penalty box, next contact in 73 seconds
>
>
> ==> hadoop-nutch-secondarynamenode-master.log <==
> 2007-04-30 08:47:31,766 WARN  NameNode.Secondary - Checkpoint done. Image
> Size:16 Edit Size:1055 New Image Size:941
>
>  slave1 server 
> 2007-04-30 08:55:02,493 WARN  mapred.TaskRunner - task_0001_r_000000_0 copy
> failed: task_0001_m_000015_0 from master
> 2007-04-30 08:55:02,494 WARN  mapred.TaskRunner - java.io.IOException: File
> /home/nutch/fs/mapred/local/task_0001_r_000000_0/map_15.out-1 not created
>         at
> org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.copyOutput(ReduceTaskRunner.java:322)
>         at
> org.apache.hadoop.mapred.ReduceTaskRunner$MapOutputCopier.run(ReduceTaskRunner.java:274)
>
> 2007-04-30 08:55:02,494 WARN  mapred.TaskRunner - task_0001_r_000000_0
> adding host master to penalty box, next contact in 84 seconds
>
> master# netstat -an | grep LIST
> tcp4       0      0  *.50050                *.*                    LISTEN
> tcp4       0      0  *.50060                *.*                    LISTEN
> tcp4       0      0  *.50030                *.*                    LISTEN
> tcp4       0      0  *.50090                *.*                    LISTEN
> tcp4       0      0  216.32.77.34.9001      *.*                    LISTEN
> tcp4       0      0  *.50075                *.*                    LISTEN
> tcp4       0      0  *.50010                *.*                    LISTEN
> tcp4       0      0  216.32.77.34.9000      *.*                    LISTEN
> tcp4       0      0  *.50070                *.*                    LISTEN
> tcp4       0      0  *.21                   *.*                    LISTEN
> tcp4       0      0  127.0.0.1.25           *.*                    LISTEN
> tcp4       0      0  *.22                   *.*                    LISTEN
> tcp6       0      0  *.22                   *.*                    LISTEN
>
>
>
> slave1# netstat -an | grep LIST
> tcp4       0      0  *.50050                *.*                    LISTEN
> tcp4       0      0  *.50060                *.*                    LISTEN
> tcp4       0      0  *.50075                *.*                    LISTEN
> tcp4       0      0  *.50010                *.*                    LISTEN
> tcp4       0      0  *.21                   *.*                    LISTEN
> tcp4       0      0  127.0.0.1.25           *.*                    LISTEN
> tcp4       0      0  *.22                   *.*                    LISTEN
> tcp6       0      0  *.22                   *.*                    LISTEN
>
>
> [nutch@master ~/search]$ bin/slaves.sh uptime
> xxx.32.77.xxx:  8:56AM  up 2 days,  2:50, 2 users, load averages: 0.08,
> 0.14, 0.11
> xxx.21.41.xxx:  8:57AM  up 2 days,  2:50, 2 users, load averages: 0.04,
> 0.08, 0.04
>
> OS: freebsd 6.2
>
>
>
>   


Mime
View raw message