hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinivas Surasani <vas...@gmail.com>
Subject Re: ENOENT: No such file or directory
Date Fri, 17 Feb 2012 03:11:55 GMT
Sumanth, I think Sreedhar is pointing to "dfs.datanode.max.xceivers"
property in hdfs-site.xml. Try setting this property to higher value.



On Thu, Feb 16, 2012 at 9:51 PM, Sumanth V <vsumanth10@gmail.com> wrote:
> ulimit values are set to much higher values than the default values
> Here is the /etc/security/limits.conf contents -
> *       -       nofile  64000
> hdfs    -       nproc   32768
> hdfs    -       stack   10240
> hbase   -       nproc   32768
> hbase   -       stack   10240
> mapred  -       nproc   32768
> mapred  -       stack   10240
>
>
> Sumanth
>
>
>
> On Thu, Feb 16, 2012 at 6:48 PM, Sree K <quikreply@yahoo.com> wrote:
>
>> Sumanth,
>>
>> You may want to check ulimit setting for open files.
>>
>>
>> Set it to a higher value if it is at default value of 1024.
>>
>> Regards,
>> Sreedhar
>>
>>
>>
>>
>> ________________________________
>>  From: Sumanth V <vsumanth10@gmail.com>
>> To: common-user@hadoop.apache.org
>> Sent: Thursday, February 16, 2012 6:25 PM
>> Subject: ENOENT: No such file or directory
>>
>> Hi,
>>
>> We have a 20 node hadoop cluster running CDH3 U2. Some of our jobs
>> are failing with the following errors. We noticed that we are
>> consistently hitting this error condition when the total number of map
>> tasks in a particular job exceeds the total map task capacity of the
>> cluster.
>> Other jobs where the number of map tasks are lower than the total map task
>> capacity fares well.
>>
>> Here are the lines from Job Tracker log file -
>>
>> 2012-02-16 15:05:28,695 INFO org.apache.hadoop.mapred.TaskInProgress:
>> Error from attempt_201202161408_0004_m_000169_0: ENOENT: No such file or
>> directory
>>         at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
>>
>> at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:
>> 172)
>>
>> at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:215)
>>         at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:288)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:245)
>>
>> Here is the task tracker log -
>>
>> 2012-02-16 15:05:22,126 INFO org.apache.hadoop.mapred.JvmManager: JVM :
>> jvm_201202161408_0004_m_1467721896 exited with exit code 0. Number of tasks
>> it ran: 1
>> 2012-02-16 15:05:22,127 WARN org.apache.hadoop.mapred.TaskLogsTruncater:
>> Exception in truncateLogs while getting allLogsFileDetails(). Ignoring the
>> truncation of logs of this process.
>> java.io.FileNotFoundException:
>> /usr/lib/hadoop-0.20/logs/userlogs/
>> job_201202161408_0004/attempt_201202161408_0004_m_000112_1/log.index
>> (No
>> such file or directory)
>>         at java.io.FileInputStream.open(Native Method)
>>         at java.io.FileInputStream.<init>(FileInputStream.java:120)
>>         at java.io.FileReader.<init>(FileReader.java:55)
>>
>> at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:
>> 110)
>>
>> at
>> org.apache.hadoop.mapred.TaskLogsTruncater.getAllLogsFileDetails(TaskLogsTr
>> uncater.java: 353)
>>
>> at
>> org.apache.hadoop.mapred.TaskLogsTruncater.shouldTruncateLogs(TaskLogsTrunc
>> ater.java: 98)
>>
>> at
>> org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.doJv
>> mFinishedAction(UserLogManager.java: 163)
>>
>> at
>> org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.proc
>> essEvent(UserLogManager.java: 137)
>>
>> at
>> org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.moni
>> tor(UserLogManager.java: 132)
>>
>> at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager
>> $1.run(UserLogManager.java:66)
>> 2012-02-16 15:05:22,228 INFO
>> org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000006_0
>> 0.0%
>> 2012-02-16 15:05:22,228 INFO
>> org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000053_0
>> 0.0%
>> 2012-02-16 15:05:22,329 INFO
>> org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000057_0
>> 0.0%
>>
>> Any help in resolving this issue would be highly appreciated! Let me
>> know if any other config info is needed.
>>
>> Thanks,
>> Sumanth
>>



-- 
-- Srinivas
Srinivas@Cloudwick.com

Mime
View raw message