hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4003) log.index (No such file or directory) AND Task process exit with nonzero status of 126
Date Thu, 15 Mar 2012 01:50:39 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Koji Noguchi updated MAPREDUCE-4003:
------------------------------------

    Affects Version/s: 0.20.205.0
                       1.0.1

This also happens on our 0.20.205 clusters when users' jvm crashes at start up due to invalid
jvm param or JNI crashes. stderr/stdout files exist but log.index does not.

WebUI shows
{noformat}
HTTP ERROR 410

Problem accessing /tasklog. Reason:

    Failed to retrieve stdout log for task: attempt_201202130706_50000_m_000004_0

{noformat}
TaskTracker log shows
{noformat}
2012-03-15 01:35:18,431 WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stdout log
for task: attempt_201202130706_50000_m_000004_0
java.io.FileNotFoundException: /grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/log.index
(No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
        at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187)
        at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:422)
        at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
        at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:269)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at yjava.servlet.filter.BouncerFilter.doFilter(BouncerFilter.java:411)
        at com.yahoo.hadoop.HadoopBouncerFilter.doFilter(HadoopBouncerFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-03-15 01:35:18,431 WARN org.mortbay.log: /tasklog: java.io.IOException: Closed
{noformat}

Stdout/Stderr still exist with information that would help the users.

{noformat}
-bash-3.2$ ls -l /grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/
total 4
-rw-r----- 1 knoguchi hadoop 137 Mar 14 22:43 stderr
-rw-r----- 1 knoguchi hadoop   0 Mar 14 22:43 stdout
-bash-3.2$ cat /grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/stderr
Invalid maximum heap size: -Xmx10g
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
-bash-3.2$ 
{noformat}

I thought MAPREDUCE-2366 fixed this but I guess it didn't cover this case.  Fixing this would
save users' time as well as support(my) time.


                
> log.index (No such file or directory) AND Task process exit with nonzero status of 126
> --------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4003
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4003
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task-controller, tasktracker
>    Affects Versions: 0.20.205.0, 1.0.1
>         Environment: hadoop version -------Hadoop 0.20.2-cdh3u3
> uname -a: Linux xxxx 2.6.18-194.17.4.0.1.el5PAE #1 SMP Tue Oct 26 20:15:18 EDT 2010 i686
i686 i386 GNU/Linux
> core-site.xml:<configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://xxxxx:8020</value>
> </property>
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/home/hadoop/tmp20/</value>
>  </property>
> </configuration>
> mapred-site.xml:
> <configuration>
>     <property>
> <name>mapred.job.tracker</name>
> <value>192.168.9.60:9001</value>
> </property>
> <property>  <name>mapred.local.dir</name>  <value>/var/tmp/mapred/local</value>
</property>
> <property>  <name>mapred.system.dir</name>  <value>/mapred/system</value>
</property>
> </configuration>
>            Reporter: toughman
>
> hello´╝îI have dwelled on this hadoop(cdhu3) problem for 2 days,I have tried every google
method.This is the issue: when ran hadoop example "wordcount" ,the tasktracker's log in one
slave node presented such errors
>  1.WARN org.apache.hadoop.mapred.DefaultTaskController: Task wrapper stderr: bash: /var/tmp/mapred/local/ttprivate/taskTracker/hdfs/jobcache/job_201203131751_0003/attempt_201203131751_0003_m_000006_0/taskjvm.sh:
Permission denied
> 2.WARN org.apache.hadoop.mapred.TaskRunner: attempt_201203131751_0003_m_000006_0 : Child
Error java.io.IOException: Task process exit with nonzero status of 126.
> 3.WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stdout log for task: attempt_201203131751_0003_m_000003_0
java.io.FileNotFoundException: /usr/lib/hadoop-0.20/logs/userlogs/job_201203131751_0003/attempt_201203131751_0003_m_000003_0/log.index
(No such file or directory)
> I could not find similar issues in google,just got some posts seem a little relevant
,which suggest: A. the ulimit of hadoop user----but my ulimit is set large enough for this
bundled example;B. the memory used by jvm,but my jvm only use Xmx200m,too small to exceed
the limit of my machine ;C.the privilege of the mapred.local.dir and logs dir----I set them
by "chmod 777";D .the disk space is full----there are enough space for hadoop in my log directory
and mapred.local.dir.
> Thanks for you all,I am really at my wit's end,I have spend days on it. I really appreciate
any light!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message