hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yaotian <yaot...@gmail.com>
Subject Re: Re: I am running MapReduce on a 30G data on 1master/2 slave, but failed.
Date Wed, 16 Jan 2013 03:23:36 GMT
Thanks. I will read that document. I am new on this. So i attach the log
here for help. I can't understand this information.

These log from tasktrack log file:

2013-01-15 08:00:44,731 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:00:47,757 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:10:49,083 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0: Task
attempt_201301150318_0001_r_000000_0 failed to report status for 601
seconds. Killing!
2013-01-15 08:10:49,092 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
24 active threads
Thread 219 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 218 (JVM Runner jvm_201301150318_0001_r_-641939786 spawned.):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.lang.UNIXProcess@1195929
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
    org.apache.hadoop.util.Shell.runCommand(Shell.java:244)
    org.apache.hadoop.util.Shell.run(Shell.java:182)

org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)

org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:131)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
Thread 214 (Thread-135):
  State: WAITING
  Blocked count: 2
  Waited count: 3
  Waiting on java.lang.Object@1f35647
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:296)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:251)
Thread 38 (IPC Client (47) connection to master/10.120.253.32:9001 from
hadoop):
  State: TIMED_WAITING
  Blocked count: 11210
  Waited count: 11199
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 10 (taskCleanup):
  State: WAITING
  Blocked count: 4
  Waited count: 7
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@104109e
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:422)
    java.lang.Thread.run(Thread.java:662)
Thread 13 (Thread-5):
  State: WAITING
  Blocked count: 475
  Waited count: 646
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61bb9b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
Thread 14 (Thread-6):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 5
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
Thread 37 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 618
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 36 (23978087@qtp-7433399-0 - Acceptor0
SelectChannelConnector@0.0.0.0:50060):
  State: RUNNABLE
  Blocked count: 29509
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)

org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 35 (TaskLauncher for REDUCE tasks):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.util.LinkedList@ca1038
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 34 (TaskLauncher for MAP tasks):
  State: WAITING
  Blocked count: 170
  Waited count: 171
  Waiting on java.util.LinkedList@1ddec9e
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 27 (Map-events fetcher for all reduce tasks on
tracker_ip-10-87-145-204.ec2.internal:localhost/127.0.0.1:60497):
  State: TIMED_WAITING
  Blocked count: 16184
  Waited count: 17896
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:975)
Thread 25 (Thread-14):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 293
  Stack:
    java.lang.Thread.sleep(Native Method)

org.apache.hadoop.filecache.TrackerDistributedCacheManager$CleanupThread.run(TrackerDistributedCacheManager.java:926)
Thread 24 (IPC Server handler 1 on 60497):
  State: WAITING
  Blocked count: 1
  Waited count: 10501
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@fe99b6
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 0 on 60497):
  State: WAITING
  Blocked count: 1
  Waited count: 10502
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@fe99b6
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 20 (IPC Server listener on 60497):
  State: RUNNABLE
  Blocked count: 171
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 22 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (pool-1-thread-1):
  State: RUNNABLE
  Blocked count: 171
  Waited count: 171
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 15 (Directory/File cleanup thread):
  State: WAITING
  Blocked count: 0
  Waited count: 343
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@94af2f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

org.apache.hadoop.mapred.CleanupQueue$PathCleanupThread.run(CleanupQueue.java:130)
Thread 9 (Timer for 'TaskTracker' metrics system):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1796
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 186
  Waited count: 187
  Waiting on java.lang.ref.ReferenceQueue$Lock@19a0203
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 186
  Waited count: 187
  Waiting on java.lang.ref.Reference$Lock@1fa39bb
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: RUNNABLE
  Blocked count: 5850
  Waited count: 11663
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)

org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)

org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)

org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:1970)
    org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1652)
    org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2434)
    org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3675)

2013-01-15 08:10:49,093 INFO org.apache.hadoop.mapred.TaskTracker: About to
purge task: attempt_201301150318_0001_r_000000_0
2013-01-15 08:10:49,114 INFO org.apache.hadoop.util.ProcessTree: Killing
process group21719 with signal TERM. Exit code 0
2013-01-15 08:10:49,115 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2013-01-15 08:10:52,095 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201301150318_0001_r_000000_0
task's state:FAILED_UNCLEAN
2013-01-15 08:10:52,096 INFO org.apache.hadoop.mapred.TaskTracker: Trying
to launch : attempt_201301150318_0001_r_000000_0 which needs 1 slots
2013-01-15 08:10:52,096 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201301150318_0001_r_000000_0 which needs 1 slots
2013-01-15 08:10:52,112 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201301150318_0001_r_-56086075
2013-01-15 08:10:52,113 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201301150318_0001_r_-56086075 spawned.
2013-01-15 08:10:52,122 INFO org.apache.hadoop.mapred.TaskController:
Writing commands to
/data/hadoop/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201301150318_0001/attempt_201301150318_0001_r_000000_0.cleanup/taskjvm.sh
2013-01-15 08:10:52,780 WARN
org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 143
2013-01-15 08:10:52,780 INFO
org.apache.hadoop.mapred.DefaultTaskController: Output from
DefaultTaskController's launchTask follows:
2013-01-15 08:10:52,780 INFO org.apache.hadoop.mapred.TaskController:
2013-01-15 08:10:52,780 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201301150318_0001_r_-641939786 exited with exit code 143. Number of
tasks it ran: 0
2013-01-15 08:10:52,794 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
UserName hadoop for UID 1006 from the native implementation
2013-01-15 08:10:53,177 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201301150318_0001_r_-56086075 given task:
attempt_201301150318_0001_r_000000_0
2013-01-15 08:10:53,652 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.0%
2013-01-15 08:10:56,620 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.0% cleanup
2013-01-15 08:10:56,621 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201301150318_0001_r_000000_0 is done.
2013-01-15 08:10:56,621 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201301150318_0001_r_000000_0  was -1
2013-01-15 08:10:56,621 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2013-01-15 08:10:56,679 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201301150318_0001_r_-56086075 exited with exit code 0. Number of tasks
it ran: 1
2013-01-15 08:11:07,483 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.87.145.204:50060,
dest: 10.123.73.45:42479, bytes: 6394091, op: MAPRED_SHUFFLE, cliID:
attempt_201301150318_0001_m_000000_0, duration: 111147314
2013-01-15 08:11:07,831 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.87.145.204:50060,
dest: 10.123.73.45:42479, bytes: 6163890, op: MAPRED_SHUFFLE, cliID:
attempt_201301150318_0001_m_000001_0, duration: 126898055


2013/1/15 Charlie A. <hadoop90@163.com>

> Hi, yaotian
> I think you should check logs on the very tasktracker, it'll tell you why.
> And here's some tips on deploying a MR job.
> http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/
>
> Charlie
>
>
>
> At 2013-01-15 16:34:27,yaotian <yaotian@gmail.com> wrote:
>
> I set mapred.reduce.tasks from -1 to "AutoReduce"
> And the hadoop created 450 tasks for Map. But 1 task for Reduce. It seems
> that this reduce only run on 1 slave (I have two slaves).
>
> But when it was running on 66%, the error report again "Task
> attempt_201301150318_0001_r_000000_0 failed to report status for 601
> seconds. Killing!"
>
>
>
> 2013/1/14 yaotian <yaotian@gmail.com>
>
>> How to judge which counter would work?
>>
>>
>> 2013/1/11 <bejoy.hadoop@gmail.com>
>>
>> **
>>> Hi
>>>
>>> To add on to Harsh's comments.
>>>
>>> You need not have to change the task time out.
>>>
>>> In your map/reduce code, you can increment a counter or report status
>>> intermediate on intervals so that there is communication from the task and
>>> hence won't have a task time out.
>>>
>>> Every map and reduce task run on its own jvm limited by a jvm size. If
>>> you try to holds too much data in memory then it can go beyond the jvm size
>>> and cause OOM errors.
>>>
>>> Regards
>>> Bejoy KS
>>>
>>> Sent from remote device, Please excuse typos
>>> ------------------------------
>>> *From: * yaotian <yaotian@gmail.com>
>>> *Date: *Fri, 11 Jan 2013 14:35:07 +0800
>>> *To: *<user@hadoop.apache.org>
>>> *ReplyTo: * user@hadoop.apache.org
>>> *Subject: *Re: I am running MapReduce on a 30G data on 1master/2 slave,
>>> but failed.
>>>
>>> See inline.
>>>
>>>
>>> 2013/1/11 Harsh J <harsh@cloudera.com>
>>>
>>>> If the per-record processing time is very high, you will need to
>>>> periodically report a status. Without a status change report from the task
>>>> to the tracker, it will be killed away as a dead task after a default
>>>> timeout of 10 minutes (600s).
>>>>
>>> =====================> Do you mean to increase the report time: "*
>>> mapred.task.timeout"*?
>>>
>>>
>>>> Also, beware of holding too much memory in a reduce JVM - you're still
>>>> limited there. Best to let the framework do the sort or secondary sort.
>>>>
>>> =======================>  You mean use the default value ? This is my
>>> value.
>>> *mapred.job.reduce.memory.mb*-1
>>>
>>>>
>>>>
>>>> On Fri, Jan 11, 2013 at 10:58 AM, yaotian <yaotian@gmail.com> wrote:
>>>>
>>>>> Yes, you are right. The data is GPS trace related to corresponding
>>>>> uid. The reduce is doing this: Sort user to get this kind of result:
uid,
>>>>> gps1, gps2, gps3........
>>>>> Yes, the gps data is big because this is 30G data.
>>>>>
>>>>> How to solve this?
>>>>>
>>>>>
>>>>>
>>>>> 2013/1/11 Mahesh Balija <balijamahesh.mca@gmail.com>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>           2 reducers are successfully completed and 1498 have been
>>>>>> killed. I assume that you have the data issues. (Either the data
is huge or
>>>>>> some issues with the data you are trying to process)
>>>>>>           One possibility could be you have many values associated
to
>>>>>> a single key, which can cause these kind of issues based on the operation
>>>>>> you do in your reducer.
>>>>>>           Can you put some logs in your reducer and try to trace
out
>>>>>> what is happening.
>>>>>>
>>>>>> Best,
>>>>>> Mahesh Balija,
>>>>>> Calsoft Labs.
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 11, 2013 at 8:53 AM, yaotian <yaotian@gmail.com>
wrote:
>>>>>>
>>>>>>> I have 1 hadoop master which name node locates and 2 slave which
>>>>>>> datanode locate.
>>>>>>>
>>>>>>> If i choose a small data like 200M, it can be done.
>>>>>>>
>>>>>>> But if i run 30G data, Map is done. But the reduce report error.
Any
>>>>>>> sugggestion?
>>>>>>>
>>>>>>>
>>>>>>> This is the information.
>>>>>>>
>>>>>>> *Black-listed TaskTrackers:* 1<http://23.20.27.135:9003/jobblacklistedtrackers.jsp?jobid=job_201301090834_0041>
>>>>>>> ------------------------------
>>>>>>> Kind % CompleteNum Tasks PendingRunningComplete KilledFailed/Killed
>>>>>>> Task Attempts<http://23.20.27.135:9003/jobfailures.jsp?jobid=job_201301090834_0041>
>>>>>>> map<http://23.20.27.135:9003/jobtasks.jsp?jobid=job_201301090834_0041&type=map&pagenum=1>
>>>>>>> 100.00%4500 0450<http://23.20.27.135:9003/jobtasks.jsp?jobid=job_201301090834_0041&type=map&pagenum=1&state=completed>
>>>>>>> 00 / 1<http://23.20.27.135:9003/jobfailures.jsp?jobid=job_201301090834_0041&kind=map&cause=killed>
>>>>>>> reduce<http://23.20.27.135:9003/jobtasks.jsp?jobid=job_201301090834_0041&type=reduce&pagenum=1>
>>>>>>> 100.00%1500 0 02<http://23.20.27.135:9003/jobtasks.jsp?jobid=job_201301090834_0041&type=reduce&pagenum=1&state=completed>
>>>>>>> 1498<http://23.20.27.135:9003/jobtasks.jsp?jobid=job_201301090834_0041&type=reduce&pagenum=1&state=killed>
>>>>>>> 12<http://23.20.27.135:9003/jobfailures.jsp?jobid=job_201301090834_0041&kind=reduce&cause=failed>
>>>>>>>  / 3<http://23.20.27.135:9003/jobfailures.jsp?jobid=job_201301090834_0041&kind=reduce&cause=killed>
>>>>>>>
>>>>>>>
>>>>>>> TaskCompleteStatusStart TimeFinish TimeErrorsCounters
>>>>>>> task_201301090834_0041_r_000001<http://23.20.27.135:9003/taskdetails.jsp?tipid=task_201301090834_0041_r_000001>
>>>>>>> 0.00%
>>>>>>> 10-Jan-2013 04:18:54
>>>>>>> 10-Jan-2013 06:46:38 (2hrs, 27mins, 44sec)
>>>>>>>
>>>>>>> Task attempt_201301090834_0041_r_000001_0 failed to report status
for 600 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000001_1 failed to report status
for 602 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000001_2 failed to report status
for 602 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000001_3 failed to report status
for 602 seconds. Killing!
>>>>>>>
>>>>>>>
>>>>>>> 0<http://23.20.27.135:9003/taskstats.jsp?tipid=task_201301090834_0041_r_000001>
>>>>>>> task_201301090834_0041_r_000002<http://23.20.27.135:9003/taskdetails.jsp?tipid=task_201301090834_0041_r_000002>
>>>>>>> 0.00%
>>>>>>> 10-Jan-2013 04:18:54
>>>>>>> 10-Jan-2013 06:46:38 (2hrs, 27mins, 43sec)
>>>>>>>
>>>>>>> Task attempt_201301090834_0041_r_000002_0 failed to report status
for 601 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000002_1 failed to report status
for 600 seconds. Killing!
>>>>>>>
>>>>>>>
>>>>>>> 0<http://23.20.27.135:9003/taskstats.jsp?tipid=task_201301090834_0041_r_000002>
>>>>>>> task_201301090834_0041_r_000003<http://23.20.27.135:9003/taskdetails.jsp?tipid=task_201301090834_0041_r_000003>
>>>>>>> 0.00%
>>>>>>> 10-Jan-2013 04:18:57
>>>>>>> 10-Jan-2013 06:46:38 (2hrs, 27mins, 41sec)
>>>>>>>
>>>>>>> Task attempt_201301090834_0041_r_000003_0 failed to report status
for 602 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000003_1 failed to report status
for 602 seconds. Killing!
>>>>>>> Task attempt_201301090834_0041_r_000003_2 failed to report status
for 602 seconds. Killing!
>>>>>>>
>>>>>>>
>>>>>>> 0<http://23.20.27.135:9003/taskstats.jsp?tipid=task_201301090834_0041_r_000003>
>>>>>>> task_201301090834_0041_r_000005<http://23.20.27.135:9003/taskdetails.jsp?tipid=task_201301090834_0041_r_000005>
>>>>>>> 0.00%
>>>>>>> 10-Jan-2013 06:11:07
>>>>>>> 10-Jan-2013 06:46:38 (35mins, 31sec)
>>>>>>>
>>>>>>> Task attempt_201301090834_0041_r_000005_0 failed to report status
for 600 seconds. Killing!
>>>>>>>
>>>>>>>
>>>>>>> 0<http://23.20.27.135:9003/taskstats.jsp?tipid=task_201301090834_0041_r_000005>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>
>
>

Mime
View raw message