hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@gmail.com>
Subject Re: Hadoop 0.20.2: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
Date Fri, 13 Aug 2010 10:23:10 GMT
Aside from usual checks regarding connectivity between master and
slaves, one quick observation is that the hadoop.tmp.dir says 'tmp'.
Shouldn't this be '/tmp' ?

Thanks
Hemanth
2010/8/13 Kevin Chen <bestgembler@hotmail.com>:
>
> Hello,
>
> I've confused the problem for a week already, Please sharing if you know what  could be causing this, Thinks in advance!
>
> Hadoop version: 0.20.2
>
> Machines:
>
> machine 1 - NameNode,JobTracker,SecondNameNode:
> OS: ubuntu
> Hostname: master
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
> machine 2 - DataNode, TaskTracker:
>
> OS: ubuntu
>
> Hostname: slave1
>
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
> machine 3 - DataNode, TaskTracker:
>
> OS: ubuntu
>
> Hostname: slave2
>
> ulimit -n: 10240
> disk space: enough
> ${mapred.local.dir}'s permissions: 777
>
>
> Configurations:
>
> core-site.xml:
> <configuration>
>  <property>
>    <name>hadoop.tmp.dir</name>
>    <value>tmp</value>
>  </property>
>  <property>
>    <name>fs.default.name</name>
>    <value>hdfs://master:9000</value>
>  </property>
> </configuration>
>
>
> hdfs-site.xml:
> <configuration>
> </configuration>
>
>
> mapred-site.xml:
> <configuration>
>  <property>
>    <name>mapred.job.tracker</name>
>    <value>master:9001</value>
>  </property>
> </configuration>
>
>
> masters:
> bestgembler@master
>
>
> slaves:
> hadoop@slave1
> hadoop@slave2
>
> Parts of error:
>
>
> Terminal:
> 10/08/13 17:35:03 INFO mapred.FileInputFormat: Total input paths to process : 1
> 10/08/13 17:35:03 INFO mapred.JobClient: Running job: job_201008131730_0001
> 10/08/13 17:35:04 INFO mapred.JobClient:  map 0% reduce 0%
> 10/08/13 17:35:14 INFO mapred.JobClient:  map 50% reduce 0%
> 10/08/13 17:35:17 INFO mapred.JobClient:  map 100% reduce 0%
> 10/08/13 17:35:23 INFO mapred.JobClient: Task Id : attempt_201008131730_0001_m_000001_0, Status : FAILED
> Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> hadoop-bestgembler-jobtracker-master.log:
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
> 2010-08-13 17:32:54,127 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:32:54,128 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
> 2010-08-13 17:32:54,152 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
> 2010-08-13 17:32:54,153 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
> 2010-08-13 17:32:54,471 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave1
> 2010-08-13 17:32:57,603 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave2
> 2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201008131730_0001
> 2010-08-13 17:35:03,905 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201008131730_0001
> 2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201008131730_0001 = 1366. Number of splits = 2
> 2010-08-13 17:35:04,370 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave2
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000000 has split on node:/default-rack/slave1
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave2
> 2010-08-13 17:35:04,371 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201008131730_0001_m_000001 has split on node:/default-rack/slave1
> 2010-08-13 17:35:06,945 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000003_0' to tip task_201008131730_0001_m_000003, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:10,062 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000003_0' has completed task_201008131730_0001_m_000003 successfully.
> 2010-08-13 17:35:10,068 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_0' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:10,071 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
> 2010-08-13 17:35:12,958 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_0' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:12,959 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
> 2010-08-13 17:35:13,084 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_0' has completed task_201008131730_0001_m_000000 successfully.
> 2010-08-13 17:35:13,109 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_r_000000_0' to tip task_201008131730_0001_r_000000, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:16,016 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_0' has completed task_201008131730_0001_m_000001 successfully.
> 2010-08-13 17:35:22,027 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000001_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:22,029 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000001_0' from 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:22,138 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_0: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:22,140 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000001_1' to tip task_201008131730_0001_m_000001, for tracker 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000001
> 2010-08-13 17:35:22,141 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201008131730_0001_m_000000_0' from 'tracker_slave2:localhost/127.0.0.1:42915'
> 2010-08-13 17:35:25,032 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201008131730_0001_m_000000_1' to tip task_201008131730_0001_m_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:56514'
> 2010-08-13 17:35:25,033 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201008131730_0001_m_000000
> 2010-08-13 17:35:25,150 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000001_1' has completed task_201008131730_0001_m_000001 successfully.
> 2010-08-13 17:35:28,043 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201008131730_0001_m_000000_1' has completed task_201008131730_0001_m_000000 successfully.
> 2010-08-13 17:35:31,049 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201008131730_0001_m_000000_1: Map output lost, rescheduling: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
>
> hadoop-hadoop-tasktracker-slave1.log:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host = slave1/192.168.0.186
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-08-13 17:31:46,410 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2010-08-13 17:31:46,499 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-08-13 17:31:46,505 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2010-08-13 17:31:46,506 INFO org.mortbay.log: jetty-6.1.14
> 2010-08-13 17:31:46,714 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2010-08-13 17:31:46,721 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-08-13 17:31:46,731 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=56514
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 56514: starting
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:56514
> 2010-08-13 17:31:46,760 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 56514: starting
> 2010-08-13 17:31:46,761 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave1:localhost/127.0.0.1:56514
> 2010-08-13 17:33:54,121 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave1:localhost/127.0.0.1:56514
> 2010-08-13 17:33:54,124 INFO org.apache.hadoop.mapred.TaskTracker:  Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@102f729e
> 2010-08-13 17:33:54,133 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2010-08-13 17:33:54,136 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2010-08-13 17:36:12,961 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_0 task's state:UNASSIGNED
> 2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:12,969 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:13,481 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-841871629
> 2010-08-13 17:36:13,482 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-841871629 spawned.
> 2010-08-13 17:36:13,918 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-841871629 given task: attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:14,455 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_0 is done.
> 2010-08-13 17:36:14,474 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_0  was 0
> 2010-08-13 17:36:14,497 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:14,651 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-841871629 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:15,961 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out in any of the configured local directories
> 2010-08-13 17:36:19,657 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:19,657 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:19,666 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:45984, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_0
> 2010-08-13 17:36:19,666 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_0/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:24,987 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_1 task's state:UNASSIGNED
> 2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:24,988 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:25,095 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-673950863
> 2010-08-13 17:36:25,096 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-673950863 spawned.
> 2010-08-13 17:36:25,475 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-673950863 given task: attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:25,999 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_1 is done.
> 2010-08-13 17:36:26,018 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_1  was 0
> 2010-08-13 17:36:26,019 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:26,181 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-673950863 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:27,988 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out in any of the configured local directories
> 2010-08-13 17:36:29,740 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:29,740 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:29,746 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34390, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_1
> 2010-08-13 17:36:29,746 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_1/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:31,005 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_2 task's state:UNASSIGNED
> 2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:31,006 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1281957734
> 2010-08-13 17:36:31,143 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1281957734 spawned.
> 2010-08-13 17:36:31,557 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1281957734 given task: attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:32,079 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_2 is done.
> 2010-08-13 17:36:32,082 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_2  was 0
> 2010-08-13 17:36:32,086 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:32,271 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1281957734 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:34,006 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out in any of the configured local directories
> 2010-08-13 17:36:39,785 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_2,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:39,785 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:39,790 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34399, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_2
> 2010-08-13 17:36:39,790 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_2/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_3 task's state:UNASSIGNED
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:40,026 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:40,157 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-485455839
> 2010-08-13 17:36:40,182 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-485455839 spawned.
> 2010-08-13 17:36:40,584 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-485455839 given task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:41,072 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_3 is done.
> 2010-08-13 17:36:41,094 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_3  was 0
> 2010-08-13 17:36:41,096 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:41,300 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-485455839 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:43,027 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out in any of the configured local directories
> 2010-08-13 17:36:44,812 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_3,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:36:44,813 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:44,825 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.186:50060, dest: 192.168.0.187:34406, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:44,825 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_3/output/file.out.index in any of the configured local directories
> 2010-08-13 17:36:46,044 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000002_0 task's state:UNASSIGNED
> 2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,045 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:46,046 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000000_3
> 2010-08-13 17:36:46,047 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_3 done; removing files.
> 2010-08-13 17:36:46,049 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_3 not found in cache
> 2010-08-13 17:36:46,151 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1831604016
> 2010-08-13 17:36:46,152 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1831604016 spawned.
> 2010-08-13 17:36:46,538 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1831604016 given task: attempt_201008131730_0001_m_000002_0
> 2010-08-13 17:36:46,766 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0%
> 2010-08-13 17:36:46,769 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000002_0 0.0% cleanup
> 2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000002_0 is done.
> 2010-08-13 17:36:46,787 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000002_0  was 0
> 2010-08-13 17:36:46,790 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:36:46,978 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1831604016 exited. Number of tasks it ran: 1
> 2010-08-13 17:36:49,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000002_0/output/file.out in any of the configured local directories
> 2010-08-13 17:36:49,096 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
> 2010-08-13 17:36:49,097 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_2 done; removing files.
> 2010-08-13 17:36:49,099 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_2 not found in cache
> 2010-08-13 17:36:49,100 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_1 done; removing files.
> 2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_1 not found in cache
> 2010-08-13 17:36:49,102 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000002_0 done; removing files.
> 2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000002_0 not found in cache
> 2010-08-13 17:36:49,105 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_0 done; removing files.
> 2010-08-13 17:36:49,108 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_0 not found in cache
> 2010-08-13 17:37:01,309 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
>    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
>    at org.apache.hadoop.ipc.Client.call(Client.java:743)
>    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>    at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
>    at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
>    at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
>    at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
>    at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
> Caused by: java.io.IOException: Connection reset by peer
>    at sun.nio.ch.FileDispatcher.read0(Native Method)
>    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>    at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>    at java.io.FilterInputStream.read(FilterInputStream.java:116)
>    at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
>    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>    at java.io.DataInputStream.readInt(DataInputStream.java:370)
>    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> 2010-08-13 17:37:01,309 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '61
> 2010-08-13 17:37:01,388 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at slave1/192.168.0.186
> ************************************************************/
>
>
> hadoop-hadoop-tasktracker-slave2.log:
> 2010-08-13 17:30:35,572 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host = slave2/192.168.0.187
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-08-13 17:30:35,847 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2010-08-13 17:30:36,025 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
> 2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-08-13 17:30:36,036 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2010-08-13 17:30:36,037 INFO org.mortbay.log: jetty-6.1.14
> 2010-08-13 17:32:45,915 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
> 2010-08-13 17:32:45,930 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2010-08-13 17:32:45,952 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=TaskTracker, port=42915
> 2010-08-13 17:32:46,023 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:42915
> 2010-08-13 17:32:46,026 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_slave2:localhost/127.0.0.1:42915
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 42915: starting
> 2010-08-13 17:32:46,025 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 42915: starting
> 2010-08-13 17:32:46,034 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 42915: starting
> 2010-08-13 17:32:46,121 INFO org.apache.hadoop.mapred.TaskTracker:  Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@13a78071
> 2010-08-13 17:32:46,126 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
> 2010-08-13 17:32:46,128 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
> 2010-08-13 17:32:46,130 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_slave2:localhost/127.0.0.1:42915
> 2010-08-13 17:34:55,817 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000003_0 task's state:UNASSIGNED
> 2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:55,825 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:56,531 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-1485663808
> 2010-08-13 17:34:56,532 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-1485663808 spawned.
> 2010-08-13 17:34:57,324 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-1485663808 given task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:57,740 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000003_0 0.0% setup
> 2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000003_0 is done.
> 2010-08-13 17:34:57,742 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000003_0  was 0
> 2010-08-13 17:34:57,746 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:34:57,914 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-1485663808 exited. Number of tasks it ran: 1
> 2010-08-13 17:34:58,817 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000003_0/output/file.out in any of the configured local directories
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000000_0 task's state:UNASSIGNED
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:34:58,842 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000003_0
> 2010-08-13 17:34:58,843 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000003_0 done; removing files.
> 2010-08-13 17:34:58,844 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000003_0 not found in cache
> 2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_2093675286
> 2010-08-13 17:34:58,914 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_2093675286 spawned.
> 2010-08-13 17:34:59,647 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_2093675286 given task: attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:00,527 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000000_0 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:0+683
> 2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000000_0 is done.
> 2010-08-13 17:35:00,529 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000000_0  was 0
> 2010-08-13 17:35:00,530 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:00,635 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_2093675286 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:01,840 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out in any of the configured local directories
> 2010-08-13 17:35:01,891 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_r_000000_0 task's state:UNASSIGNED
> 2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:01,892 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:02,003 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_r_-1485663808
> 2010-08-13 17:35:02,004 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_r_-1485663808 spawned.
> 2010-08-13 17:35:02,866 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_r_-1485663808 given task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:08,507 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000000_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:08,508 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:08,533 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:45820, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000000_0
> 2010-08-13 17:35:08,534 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000000_0/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:09,311 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:10,908 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_1 task's state:UNASSIGNED
> 2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:10,909 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:11,067 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-83208921
> 2010-08-13 17:35:11,068 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-83208921 spawned.
> 2010-08-13 17:35:11,863 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-83208921 given task: attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:12,316 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:12,733 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_1 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_1 is done.
> 2010-08-13 17:35:12,735 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_1  was 0
> 2010-08-13 17:35:12,736 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:12,876 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-83208921 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:13,909 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out in any of the configured local directories
> 2010-08-13 17:35:18,323 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:18,548 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_1,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:18,551 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:18,573 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34209, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_1
> 2010-08-13 17:35:18,573 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_1/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:19,931 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_2 task's state:UNASSIGNED
> 2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:19,932 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:20,103 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_-916850178
> 2010-08-13 17:35:20,106 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_-916850178 spawned.
> 2010-08-13 17:35:20,947 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_-916850178 given task: attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:21,768 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_2 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_2 is done.
> 2010-08-13 17:35:21,770 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_2  was 0
> 2010-08-13 17:35:21,771 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:22,000 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_-916850178 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:22,931 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out in any of the configured local directories
> 2010-08-13 17:35:24,329 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:27,333 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:28,597 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_2,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:28,597 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:28,616 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34216, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_2
> 2010-08-13 17:35:28,616 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_2/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:28,952 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201008131730_0001_m_000001_3 task's state:UNASSIGNED
> 2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:28,953 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:29,108 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201008131730_0001_m_302025266
> 2010-08-13 17:35:29,109 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201008131730_0001_m_302025266 spawned.
> 2010-08-13 17:35:29,845 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201008131730_0001_m_302025266 given task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:30,701 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_m_000001_3 1.0% hdfs://master:9000/user/bestgembler/wordcount/input/README.txt:683+683
> 2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201008131730_0001_m_000001_3 is done.
> 2010-08-13 17:35:30,703 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201008131730_0001_m_000001_3  was 0
> 2010-08-13 17:35:30,704 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:30,828 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_m_302025266 exited. Number of tasks it ran: 1
> 2010-08-13 17:35:31,953 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out in any of the configured local directories
> 2010-08-13 17:35:33,339 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201008131730_0001_r_000000_0 0.0% reduce > copy >
> 2010-08-13 17:35:33,626 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201008131730_0001_m_000001_3,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
>    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
>    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
>    at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:324)
>    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> 2010-08-13 17:35:33,627 INFO org.apache.hadoop.mapred.TaskTracker: Reporting output lost:attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:33,638 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 192.168.0.187:50060, dest: 192.168.0.187:34225, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:33,638 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201008131730_0001/attempt_201008131730_0001_m_000001_3/output/file.out.index in any of the configured local directories
> 2010-08-13 17:35:34,969 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:34,970 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_m_000001_3
> 2010-08-13 17:35:34,971 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_3 done; removing files.
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_3 not found in cache
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:34,973 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201008131730_0001_r_000000_0
> 2010-08-13 17:35:34,975 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
> 2010-08-13 17:35:34,976 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
> 2010-08-13 17:35:35,073 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201008131730_0001_r_-1485663808 exited. Number of tasks it ran: 0
> 2010-08-13 17:35:37,975 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_r_000000_0 done; removing files.
> 2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201008131730_0001
> 2010-08-13 17:35:37,980 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000000_0 done; removing files.
> 2010-08-13 17:35:37,982 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000000_0 not found in cache
> 2010-08-13 17:35:37,983 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_1 done; removing files.
> 2010-08-13 17:35:37,984 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_1 not found in cache
> 2010-08-13 17:35:37,985 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201008131730_0001_m_000001_2 done; removing files.
> 2010-08-13 17:35:37,987 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201008131730_0001_m_000001_2 not found in cache
> 2010-08-13 17:35:50,122 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.io.IOException: Call to master/192.168.0.253:9001 failed on local exception: java.io.IOException: Connection reset by peer
>    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
>    at org.apache.hadoop.ipc.Client.call(Client.java:743)
>    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>    at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
>    at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1215)
>    at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1037)
>    at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1720)
>    at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2833)
> Caused by: java.io.IOException: Connection reset by peer
>    at sun.nio.ch.FileDispatcher.read0(Native Method)
>    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>    at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>    at java.io.FilterInputStream.read(FilterInputStream.java:116)
>    at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
>    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>    at java.io.DataInputStream.readInt(DataInputStream.java:370)
>    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> 2010-08-13 17:35:50,123 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'master' with reponseId '60
> 2010-08-13 17:35:50,513 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at slave2/192.168.0.187
> ************************************************************/
>

Mime
View raw message