hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Basu,Indrashish" <indrash...@ufl.edu>
Subject Re: Error while running Hadoop Source Code
Date Thu, 07 Nov 2013 16:28:57 GMT
  

Hi Vinod, 

Please find the below details requested: 

A) ALL OF
THE TASKTRACKER LOG : 

Same what I have provided in the earlier email. 


B) THE TASK-LOGS : 

 SYSLOG : 

2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer =
262144/327680
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
 at
java.io.DataInputStream.readByte(DataInputStream.java:267)
 at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
 at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
 at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

2013-11-06
16:37:23,157 INFO org.apache.hadoop.mapred.pipes.Application: Aborting
because of java.net.SocketException: Broken pipe
 at
java.net.SocketOutputStream.socketWrite0(Native Method)
 at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
 at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
 at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
 at
java.io.DataOutputStream.write(DataOutputStream.java:107)
 at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)

at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)

at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
 at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
 at
org.apache.hadoop.mapred.Child.main(Child.java:170)

2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
java.io.IOException: pipe child exception
 at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)

at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
 at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
 at
org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by:
java.net.SocketException: Broken pipe
 at
java.net.SocketOutputStream.socketWrite0(Native Method)
 at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
 at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
 at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
 at
java.io.DataOutputStream.write(DataOutputStream.java:107)
 at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)

at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)

at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)

... 3 more

 STDERR : 

 Command:0received before authentication.
Exiting.. 

 STDOUT :  

 cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo

cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log

s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
 deviceID: 

C)
SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING : 


attempt_201311061630_0001_m_000000_0 


attempt_201311061630_0001_m_000001_0  


attempt_201311061630_0001_m_000002_0  


attempt_201311061630_0001_m_000000_1  


attempt_201311061630_0001_m_000001_1  


attempt_201311061630_0001_m_000003_0 


attempt_201311061630_0001_m_000000_2  


attempt_201311061630_0001_m_000001_2  


attempt_201311061630_0001_m_000000_3  


attempt_201311061630_0001_m_000001_3 

Kindly let me know in case of any
more details required. 

Thanks again for your time and patience
regarding this. 

Regards, 

Indrashish. 

On Wed, 6 Nov 2013 16:58:53
-0800, Vinod Kumar Vavilapalli wrote: 

> Don't see anything in the logs
that you pasted. 
> Can you paste the following in say pastebin? 
> -
All of the TaskTracker log 
> - The task-logs. These are syslog, stderr,
stdout files for a specific TaskAttempt. 
> - And specific TaskAttempt's
TaskAttemtpID that is failing. 
> 
> Thanks, 
> +Vinod 
> 
> On Nov 6,
2013, at 2:50 PM, Basu,Indrashish wrote: 
> 
>> Can anyone please assist
regarding this ? 
>> 
>> Thanks in advance 
>> 
>> Regards, 
>> 
>>
Indra 
>> 
>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:

>> 
>>> Hi Vinod, 
>>> 
>>> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output. 
>>> 
>>>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588 
>>>

>>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_1100862588 spawned. 
>>> 
>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766 
>>>

>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned. 
>>> 
>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060 
>>> 
>>> 636_0001_m_000001_3 
>>> 
>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106 
>>>

>>> 0636_0001_m_000000_3 
>>> 
>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false 
>>> 
>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false 
>>> 
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0% 
>>> 
>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0% 
>>> 
>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0

>>> 
>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0

>>> 
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false 
>>> 
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false 
>>> 
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files. 
>>> 
>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3 
>>> 
>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files. 
>>> 
>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4 
>>> 
>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false 
>>> 
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false 
>>> 
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false

>>> 
>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false 
>>> 
>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's 
>>> 
>>> state:UNASSIGNED 
>>> 
>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU 
>>> 
>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013 
>>> 
>>>
11060636_0001_m_000002_0 
>>> 
>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3 
>>> 
>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3 
>>> 
>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908 
>>> 
>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned. 
>>> 
>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606 
>>>

>>> 36_0001_m_000002_0 
>>> 
>>> Regards, 
>>> 
>>> Indrashish 
>>>

>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:

>>> 
>>>> It seems like your pipes mapper is exiting before consuming
all the input. Did you check the task-logs on the web UI? 
>>>> 
>>>>
Thanks, 
>>>> +Vinod 
>>>> 
>>>> On Nov 5, 2013, at 7:25 AM,
Basu,Indrashish wrote: 
>>>> 
>>>>> Hi,
>>>>> 
>>>>> Can anyone kindly
assist on this ?
>>>>> 
>>>>> Regards,
>>>>> Indrashish
>>>>> 
>>>>> On
Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>>> 
>>>>>> Hi
All,
>>>>> 
>>>>>> Any update on the below post ?
>>>>> solution as "
The *nopipe* example needs more documentation. It is run with the
InputFormat from
>>>>> x; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">src/test/org/apache/*hadoop*/mapred/*p
>>>>> 
>>>>>>
: 5px; width: 100%;">*WordCountInputFormat*.java
>>>>> very specific
input split probably be recoded *to* use libhdfs *too*, but t
>>>>>
blockquote> if you have nfs or something working across the cluster.
"
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"
>>>>> 
>>>>>> ding-left: 5px;
border-left: #1010ff 2px solid; margin-l
>>>>> th: 100%;">could
elaborate a bit about the same as what needs to be done exactly.
>>>>>

>>>>>>> To mention, I am trying to run a sample KMeans algorithm on a
GPU
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">usin
>>>>> 
>>>>>> eft: 5px;
width: 100%;">Thanks in advance.
>>>>> g-left: 5px; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;">Regards, ndrashish.
>>>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;">On Thu, 31 Oct
2013 20:00:10 -0400, Basu,Indrashish wrote: : 100%;">
>>>>> id;
margin-left: 5px; width: 100%;">Hi, I am trying to run a sample Hadoop
GPU source
>>>>> algorithm) M processor and getting the below error. Can
anyone please
>>>>> ckquote> 2px solid; marg
>>>>> width: 100%;">rmr:
cannot remove output: No such file or directory. uote sty
>>>>> eft:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">13/10/31 13:43:12 WARN mapred.
>>>>> 
>>>>>> px; border-
>>>>>
2px solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to eft: 5px; width: 100%;">

>>>>>> 
>>>>>>> process : 1
>>>>> 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:13 INFO
mapred.JobClient: 
>>>>> 
>>>>>> er-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:14
>>>>> JobClient: map
0% reduce 0% attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>> olid; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
idth: 100%;"> 
>>>>>> 
>>>>>>> at
>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apach
>>>>> 
>>>>>>
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">
>>>>> 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) ;at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> d;
margin-left: 5px; width: 100%;"> at org.apac
>>>>> 
>>>>>> 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> Caused
by
>>>>> cketException: Broken pipe at
java.net.SocketOutputStream.socketWrite(Soc
>>>>> 
>>>>>> #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at java.net.Socke
>>>>>
.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
100%;"> at java.io.DataOutputStream.write(DataO
>>>>> ava:107) 
>>>>>
">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)

>>>>> 
>>>>>> at
>>>>> padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)

>>>>> 
>>>>>>> 
>>>>>> 
>>>>>>> at
>>>>> olid; margin-left: 5px;
width: 100%;">
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> ... 3 more
>>>>> ote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans
>>>>>
te> < /dev/null 1>>
>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>> px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
Regards, 
>>>>> -- 
>>>>> Indrashish Basu
>>>>> Graduate Student
>>>>>
Department o
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>>

>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>>

>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>>

>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>>

>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>>> 
>>>>>> 
>>>> 
>>>>
CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>> 
>>> -- 
>>> 
>>> Indrashish Basu

>>> Graduate Student 
>>> Department of Electrical and Computer
Engineering 
>>> University of Florida
>> 
>> -- 
>> 
>> Indrashish Basu

>> Graduate Student 
>> Department of Electrical and Computer
Engineering 
>> University of Florida
> 
> CONFIDENTIALITY NOTICE
>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.

-- 

Indrashish Basu 
Graduate Student 
Department of Electrical
and Computer Engineering 
University of Florida

  
Mime
View raw message