hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumanth V <vsumant...@gmail.com>
Subject ENOENT: No such file or directory
Date Fri, 17 Feb 2012 02:25:06 GMT
Hi,

We have a 20 node hadoop cluster running CDH3 U2. Some of our jobs
are failing with the following errors. We noticed that we are
consistently hitting this error condition when the total number of map
tasks in a particular job exceeds the total map task capacity of the
cluster.
Other jobs where the number of map tasks are lower than the total map task
capacity fares well.

Here are the lines from Job Tracker log file -

2012-02-16 15:05:28,695 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from attempt_201202161408_0004_m_000169_0: ENOENT: No such file or
directory
        at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)

at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java: 172)

at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:215)
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:288)
        at org.apache.hadoop.mapred.Child.main(Child.java:245)

Here is the task tracker log -

2012-02-16 15:05:22,126 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201202161408_0004_m_1467721896 exited with exit code 0. Number of tasks
it ran: 1
2012-02-16 15:05:22,127 WARN org.apache.hadoop.mapred.TaskLogsTruncater:
Exception in truncateLogs while getting allLogsFileDetails(). Ignoring the
truncation of logs of this process.
java.io.FileNotFoundException:
/usr/lib/hadoop-0.20/logs/userlogs/
job_201202161408_0004/attempt_201202161408_0004_m_000112_1/log.index
(No
such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:120)
        at java.io.FileReader.<init>(FileReader.java:55)

at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java: 110)

at org.apache.hadoop.mapred.TaskLogsTruncater.getAllLogsFileDetails(TaskLogsTr
uncater.java: 353)

at org.apache.hadoop.mapred.TaskLogsTruncater.shouldTruncateLogs(TaskLogsTrunc
ater.java: 98)

at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.doJv
mFinishedAction(UserLogManager.java: 163)

at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.proc
essEvent(UserLogManager.java: 137)

at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.moni
tor(UserLogManager.java: 132)

at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager
$1.run(UserLogManager.java:66)
2012-02-16 15:05:22,228 INFO
org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000006_0
0.0%
2012-02-16 15:05:22,228 INFO
org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000053_0
0.0%
2012-02-16 15:05:22,329 INFO
org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000057_0
0.0%

Any help in resolving this issue would be highly appreciated! Let me
know if any other config info is needed.

Thanks,
Sumanth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message