hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Fail to start mapreduce tasks across nodes
Date Sat, 21 Jul 2012 03:52:54 GMT
A 2-node cluster is a fully-distributed cluster and cannot use a
file:/// FileSystem as thats not a distributed filesystem (unless its
an NFS mount). This explains why some of your tasks aren't able to
locate an earlier written file on the /tmp dir thats probably
available on the JT node alone, not the TT nodes.

Use hdfs:// FS for fully-distributed operation.

On Fri, Jul 20, 2012 at 10:06 PM, Steve Sonnenberg <steveisoft@gmail.com> wrote:
> I have a 2-node Fedora system and in cluster mode, I have the following
> issue that I can't resolve.
>
> Hadoop 1.0.3
> I'm running with filesystem, file:/// and invoking the simple 'grep' example
>
> hadoop jar hadoop-examples-1.0.3.jar grep inputdir outputdir simple-pattern
>
> The initiator displays
>
> Error initializing attempt_201207201103_0003_m_000004_0:
>    java.io.FileNotFoundException: File
> file:/tmp/hadoop-hadoop/mapred/system/job_201207201103_0003/jobToken does
> not exist.
>      getFileStatus(RawLocalFileSystem.java)
>      localizeJobTokenFile(TaskTracker.java:4268)
>      initializeJob(TaskTracker.java:1177)
>      localizeJob
>      run
>
> The /tmp/hadoop-hadoop/mapred/system directory only contains a
> 'jobtracker.info' file (on all systems)
>
> On the target system, in the tasktracker log file, I get the following:
>
> 2012-07-20 11:35:59,954 DEBUG org.apache.hadoop.mapred.TaskTracker: Got
> heartbeatResponse from JobTracker with responseId: 641 and 1 actions
> 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201207201103_0003_m_000006_0 task's
> state:UNASSIGNED
> 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
> launch : attempt_201207201103_0003_m_000006_0 which needs 1 slots
> 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201207201103_0003_m_000006_0 which needs 1 slots
> 2012-07-20 11:35:59,955 WARN org.apache.hadoop.mapred.TaskTracker: Error
> initializing attempt_201207201103_0003_m_000006_0:
> java.io.FileNotFoundException: File
> file:/tmp/hadoop-hadoop/mapred/system/job_201207201103_0003/jobToken does
> not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>         at
> org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4268)
>         at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1177)
>         at
> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 2012-07-20 11:35:59,955 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to
> set finish time for task attempt_201207201103_0003_m_000006_0 when no start
> time is set, stackTrace is : java.lang.Exception
>         at
> org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145)
>         at
> org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3142)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2440)
>         at java.lang.Thread.run(Thread.java:636)
>
> On both systems, ownership of all files directories under /tmp/hadoop-hadoop
> is the user/group hadoop/hadoop.
>
>
> Any ideas?
>
> Thanks
>
>
> --
> Steve Sonnenberg
>



-- 
Harsh J

Mime
View raw message