Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of karim.awara@kaust.edu.sa
 designates 74.125.82.44 as permitted sender)
MIME-Version: 1.0
From: Karim Awara <karim.awara@kaust.edu.sa>
Date: Thu, 17 Apr 2014 16:23:08 +0300
Message-ID: 
 <CAHqL09qimydJRcz49z0JWgOUAt3mth_wJsFSmBXPJPqHreXS5w@mail.gmail.com>
Subject: Problem with jobtracker hadoop 1.2
To: user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c37b6254daa704f73cf0e0

--001a11c37b6254daa704f73cf0e0
Content-Type: text/plain; charset=US-ASCII

Hi,

Whenever I start the hadoop on 24 machines, the following exception happens
on the jobtracker log file on the namenode:   I would appreciate any help.
Thank you.


2014-04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting
safe mode to false. Requested by : karim
2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2014-04-17 16:16:31,429 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2014-04-17 16:16:31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating
DONE folder at file:/home/karim/systems/hadoop-1.1.2/logs/history/done
2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History
server being initialized in embedded mode
2014-04-17 16:16:31,447 INFO org.apache.hadoop.mapred.JobHistoryServer:
Started job history server at: localhost:50030
2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job
History Server web address: localhost:50030
2014-04-17 16:16:31,449 INFO
org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
inactive
2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    at org.apache.hadoop.ipc.Client.call(Client.java:1107)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at $Proxy7.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
    at $Proxy7.addBlock(Unknown Source)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not
get block locations. Source file
"/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info" - Aborting...
2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing
to file
hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info failed!
2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker:
FileSystem is not ready yet!
2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapred.JobTracker: Failed to
initialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-karim/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

--
Best Regards,
Karim Ahmed Awara

-- 

------------------------------
This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.

--001a11c37b6254daa704f73cf0e0
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi, <br><br></div>Whenever I start the hadoop on 24 m=
achines, the following exception happens on the jobtracker log file on the =
namenode: =A0 I would appreciate any help. Thank you. <br><br><br><br>2014-=
04-17 16:16:31,391 INFO org.apache.hadoop.mapred.JobTracker: Setting safe m=
ode to false. Requested by : karim<br>

2014-04-17 16:16:31,400 INFO org.apache.hadoop.util.NativeCodeLoader: Loade=
d the native-hadoop library<br>2014-04-17 16:16:31,429 INFO org.apache.hado=
op.mapred.JobTracker: Cleaning up the system directory<br>2014-04-17 16:16:=
31,443 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE folder at fi=
le:/home/karim/systems/hadoop-1.1.2/logs/history/done<br>

2014-04-17 16:16:31,445 INFO org.apache.hadoop.mapred.JobTracker: History s=
erver being initialized in embedded mode<br>2014-04-17 16:16:31,447 INFO or=
g.apache.hadoop.mapred.JobHistoryServer: Started job history server at: loc=
alhost:50030<br>

2014-04-17 16:16:31,448 INFO org.apache.hadoop.mapred.JobTracker: Job Histo=
ry Server web address: localhost:50030<br>2014-04-17 16:16:31,449 INFO org.=
apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inacti=
ve<br>

<span style=3D"color:rgb(255,0,0)">2014-04-17 16:16:31,474 WARN org.apache.=
hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.Remote=
Exception: java.io.IOException: File /home/karim/data/hadoop-1.1.2_data/had=
oop_tmp-karim/mapred/system/<a href=3D"http://jobtracker.info">jobtracker.i=
nfo</a> could only be replicated to 0 nodes, instead of 1<br>

=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditio=
nalBlock(FSNamesystem.java:1639)<br>=A0=A0=A0 at org.apache.hadoop.hdfs.ser=
ver.namenode.NameNode.addBlock(NameNode.java:736)<br>=A0=A0=A0 at sun.refle=
ct.NativeMethodAccessorImpl.invoke0(Native Method)<br>

=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess=
orImpl.java:39)<br>=A0=A0=A0 at </span>sun.reflect.DelegatingMethodAccessor=
Impl.invoke(DelegatingMethodAccessorImpl.java:25)<br>=A0=A0=A0 at java.lang=
.reflect.Method.invoke(Method.java:597)<br>

=A0=A0=A0 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)<br>=A0=A0=
=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)<br>=A0=
=A0=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)<br>=
=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)<br>

=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)<br>=A0=A0=
=A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform=
ation.java:1149)<br>=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(S=
erver.java:1387)<br>

<br>=A0=A0=A0 at org.apache.hadoop.ipc.Client.call(Client.java:1107)<br>=A0=
=A0=A0 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)<br>=A0=A0=
=A0 at $Proxy7.addBlock(Unknown Source)<br>=A0=A0=A0 at sun.reflect.NativeM=
ethodAccessorImpl.invoke0(Native Method)<br>

=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess=
orImpl.java:39)<br>=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.in=
voke(DelegatingMethodAccessorImpl.java:25)<br>=A0=A0=A0 at java.lang.reflec=
t.Method.invoke(Method.java:597)<br>

=A0=A0=A0 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod=
(RetryInvocationHandler.java:85)<br>=A0=A0=A0 at org.apache.hadoop.io.retry=
.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)<br>=A0=A0=A0=
 at $Proxy7.addBlock(Unknown Source)<br>

=A0=A0=A0 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowi=
ngBlock(DFSClient.java:3686)<br>=A0=A0=A0 at org.apache.hadoop.hdfs.DFSClie=
nt$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)<br>=A0=A0=A0 =
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.j=
ava:2749)<br>

=A0=A0=A0 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.=
run(DFSClient.java:2989)<br><br>2014-04-17 16:16:31,474 WARN org.apache.had=
oop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes =3D=
=3D null<br>

2014-04-17 16:16:31,474 WARN org.apache.hadoop.hdfs.DFSClient: Could not ge=
t block locations. Source file &quot;/home/karim/data/hadoop-1.1.2_data/had=
oop_tmp-karim/mapred/system/<a href=3D"http://jobtracker.info">jobtracker.i=
nfo</a>&quot; - Aborting...<br>

2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: Writing t=
o file hdfs://cloud1:9100/home/karim/data/hadoop-1.1.2_data/hadoop_tmp-kari=
m/mapred/system/<a href=3D"http://jobtracker.info">jobtracker.info</a> fail=
ed!<br>

2014-04-17 16:16:31,474 WARN org.apache.hadoop.mapred.JobTracker: FileSyste=
m is not ready yet!<br>2014-04-17 16:16:31,477 WARN org.apache.hadoop.mapre=
d.JobTracker: Failed to initialize recovery manager. <br>org.apache.hadoop.=
ipc.RemoteException: java.io.IOException: File /home/karim/data/hadoop-1.1.=
2_data/hadoop_tmp-karim/mapred/system/<a href=3D"http://jobtracker.info">jo=
btracker.info</a> could only be replicated to 0 nodes, instead of 1<br>

=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditio=
nalBlock(FSNamesystem.java:1639)<br>=A0=A0=A0 at org.apache.hadoop.hdfs.ser=
ver.namenode.NameNode.addBlock(NameNode.java:736)<br>=A0=A0=A0 at sun.refle=
ct.NativeMethodAccessorImpl.invoke0(Native Method)<br>

=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess=
orImpl.java:39)<br>=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.in=
voke(DelegatingMethodAccessorImpl.java:25)<br>=A0=A0=A0 at java.lang.reflec=
t.Method.invoke(Method.java:597)<br>

=A0=A0=A0 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)<br>=A0=A0=
=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)<br>=A0=
=A0=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)<br>=
=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)<br>

=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)<br>=A0=A0=
=A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform=
ation.java:1149)<br>=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(S=
erver.java:1387)<br>

=A0<br clear=3D"all"><div><div><div><div dir=3D"ltr">--<br>Best Regards,<br=
>Karim Ahmed Awara</div></div>
</div></div></div>

<br>
<div><hr></div><font face=3D"Arial" size=3D"1">This message and its content=
s, including attachments are intended solely for the original recipient. If=
 you are not the intended recipient or have received this message in error,=
 please notify me immediately and delete this message from your computer sy=
stem. Any unauthorized use or distribution is prohibited. Please consider t=
he environment before printing this email.</font>
--001a11c37b6254daa704f73cf0e0--