Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F3E3FDF4C for ; Fri, 20 Jul 2012 14:16:14 +0000 (UTC) Received: (qmail 12542 invoked by uid 500); 20 Jul 2012 14:16:10 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 12391 invoked by uid 500); 20 Jul 2012 14:16:09 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 12363 invoked by uid 99); 20 Jul 2012 14:16:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jul 2012 14:16:08 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [141.51.167.101] (HELO gundel.cs.uni-kassel.de) (141.51.167.101) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jul 2012 14:16:01 +0000 Received: from localhost (localhost [127.0.0.1]) by gundel.cs.uni-kassel.de (Postfix) with ESMTP id 8E1804EA52D for ; Fri, 20 Jul 2012 16:15:38 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at gundel.cs.uni-kassel.de Received: from gundel.cs.uni-kassel.de ([127.0.0.1]) by localhost (gundel.cs.uni-kassel.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cnqHFNDEZoFH for ; Fri, 20 Jul 2012 16:15:23 +0200 (CEST) Received: from [141.51.123.117] (fudd.cs.uni-kassel.de [141.51.123.117]) by gundel.cs.uni-kassel.de (Postfix) with ESMTPSA id DE7714EA546 for ; Fri, 20 Jul 2012 16:15:23 +0200 (CEST) Message-ID: <50096809.9060001@cs.uni-kassel.de> Date: Fri, 20 Jul 2012 16:15:37 +0200 From: =?ISO-8859-1?Q?Bj=F6rn-Elmar_Macek?= User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: hdfs-user@hadoop.apache.org Subject: Re: Namenode and Jobtracker dont start References: <5006C838.3070902@cs.uni-kassel.de> In-Reply-To: Content-Type: multipart/alternative; boundary="------------020007060208030508090502" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------020007060208030508090502 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Hi Srinivas, thanks for your reply! I have been following your link and idea and been playing around alot, but still got problems with the connection (though they are different now): \_______ JAVA VERSION_________ "which java" tells me it is 1.6.0_01. If i got it right version 1.7 got problems with ssh. \_______MY TESTS_____________ According to your suggestion to look for processes running on that port i changed ports alot: When i was posting the first post of this thread. i was using ports 999 for namenode and 1000 for jobtracker. Since due to some reasons commands like "lsof -i" etc dont give me any output when usedin the cluster enviroment. So i started looking for ports that are in general unused by programs. When i changed the ports to 9004 and 9005 i got different errors which look very much like the ones you posted in the beginning of this year in the lucene section ( http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html ). It seems as if a DataNode can not communicate with the NameNode. The logs look like the following: \_______TEST RESULTS__________ ########## A DataNode ############# 2012-07-20 14:47:59,536 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = its-cs102.its.uni-kassel.de/141.51.205.12 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.2 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 ************************************************************/ 2012-07-20 14:47:59,824 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-07-20 14:47:59,841 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-07-20 14:47:59,843 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-07-20 14:47:59,844 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-07-20 14:47:59,969 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-07-20 14:48:26,792 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: its-cs100/141.51.205.10:9005. Already tried 0 time(s). 2012-07-20 14:48:26,889 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean 2012-07-20 14:48:26,934 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010 2012-07-20 14:48:26,949 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2012-07-20 14:48:27,014 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2012-07-20 14:48:27,147 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2012-07-20 14:48:27,160 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false 2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2012-07-20 14:48:27,160 INFO org.mortbay.log: jetty-6.1.26 2012-07-20 14:48:27,805 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 2012-07-20 14:48:27,811 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2012-07-20 14:48:27,813 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source DataNode registered. 2012-07-20 14:48:28,484 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort50020 registered. 2012-07-20 14:48:28,484 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort50020 registered. 2012-07-20 14:48:28,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(its-cs102.its.uni-kassel.de:50010, storageID=, infoPort=50075, ipcPort=50020) 2012-07-20 14:48:28,489 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2012-07-20 14:48:38,706 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391) at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) at org.apache.hadoop.ipc.Client.call(Client.java:1066) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy5.register(Unknown Source) at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:673) at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1480) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1540) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682) 2012-07-20 14:48:38,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at its-cs102.its.uni-kassel.de/141.51.205.12 ************************************************************/ ####### NameNode ########################## CAUTION: Please recognize, that the file mentioned in the first error log message (/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info) does not exist on the NameNode, when i checked for it. The only path that has a simiar name is: /home/work/bmacek/hadoop/hdfs/slave/tmp (containing no further subfolders or files) 2012-07-20 14:47:58,033 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = its-cs100.its.uni-kassel.de/141.51.205.10 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.2 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 ************************************************************/ 2012-07-20 14:47:58,985 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-07-20 14:47:59,037 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2012-07-20 14:47:59,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-07-20 14:47:59,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started 2012-07-20 14:47:59,622 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2012-07-20 14:47:59,685 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2012-07-20 14:47:59,703 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source NameNode registered. 2012-07-20 14:47:59,896 INFO org.apache.hadoop.hdfs.util.GSet: VM type = 64-bit 2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB 2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: capacity = 2^21 = 2097152 entries 2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152 2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=bmacek 2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2012-07-20 14:48:00,083 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100 2012-07-20 14:48:00,084 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 2012-07-20 14:48:01,573 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean 2012-07-20 14:48:01,643 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times 2012-07-20 14:48:01,686 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1 2012-07-20 14:48:01,712 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0 2012-07-20 14:48:01,713 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 112 loaded in 0 seconds. 2012-07-20 14:48:01,796 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Invalid opcode, reached end of edit log Number of transactions found 53 2012-07-20 14:48:01,796 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /home/work/bmacek/hadoop/master/current/edits of size 1049092 edits # 53 loaded in 0 seconds. 2012-07-20 14:48:01,797 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 861 saved in 0 seconds. 2012-07-20 14:48:02,003 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 861 saved in 0 seconds. 2012-07-20 14:48:02,159 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 1 entries 11 lookups 2012-07-20 14:48:02,159 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 2135 msecs 2012-07-20 14:48:02,203 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode termination scan for invalid, over- and under-replicated blocks completed in 44 msec 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 2 secs. 2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes 2012-07-20 14:48:02,205 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2012-07-20 14:48:02,265 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list 2012-07-20 14:48:02,275 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec 2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles 2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec 2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles 2012-07-20 14:48:02,281 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source FSNamesystemMetrics registered. 2012-07-20 14:48:02,336 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9005 registered. 2012-07-20 14:48:02,337 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9005 registered. 2012-07-20 14:48:02,341 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2012-07-20 14:48:02,356 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: its-cs100.its.uni-kassel.de/141.51.205.10:9005 2012-07-20 14:48:02,878 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2012-07-20 14:48:03,312 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2012-07-20 14:48:03,426 INFO org.apache.hadoop.http.HttpServer: dfs.webhdfs.enabled = false 2012-07-20 14:48:03,465 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070 2012-07-20 14:48:03,511 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070 2012-07-20 14:48:03,511 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070 2012-07-20 14:48:03,511 INFO org.mortbay.log: jetty-6.1.26 2012-07-20 14:48:06,528 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50070 2012-07-20 14:48:06,528 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070 2012-07-20 14:48:06,561 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-07-20 14:48:06,593 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9005: starting 2012-07-20 14:48:06,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005: starting 2012-07-20 14:48:06,685 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9005: starting 2012-07-20 14:48:06,731 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005: starting 2012-07-20 14:48:06,759 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9005: starting 2012-07-20 14:48:06,791 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9005: starting 2012-07-20 14:48:06,849 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005: starting 2012-07-20 14:48:06,874 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9005: starting 2012-07-20 14:48:06,898 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005: starting 2012-07-20 14:48:06,921 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9005: starting 2012-07-20 14:48:06,974 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005: starting 2012-07-20 14:48:27,222 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:48:27,224 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56513: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:48:38,701 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010 2012-07-20 14:48:38,701 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call register(DatanodeRegistration(its-cs102.its.uni-kassel.de:50010, storageID=DS-1791721778-141.51.205.12-50010-1342788518692, infoPort=50075, ipcPort=50020)) from 141.51.205.12:33789: error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010 org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391) at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:48:54,331 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:48:54,331 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56514: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:49:18,079 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 13 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 9 SyncTimes(ms): 111 2012-07-20 14:49:18,151 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:49:18,151 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56515: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:49:41,419 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:49:41,419 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56516: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:50:04,474 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:50:04,474 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56517: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:50:26,299 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 25 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 170 2012-07-20 14:50:26,359 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:50:26,359 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56518: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:50:47,243 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:50:47,243 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56519: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:51:06,865 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010 2012-07-20 14:51:06,865 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9005, call register(DatanodeRegistration(its-cs103.its.uni-kassel.de:50010, storageID=DS-1725464844-141.51.205.13-50010-1342788666863, infoPort=50075, ipcPort=50020)) from 141.51.205.13:48227: error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010 org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391) at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:51:08,305 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:51:08,305 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56520: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:51:34,855 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 37 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 27 SyncTimes(ms): 256 2012-07-20 14:51:34,932 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:51:34,932 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56521: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:51:57,128 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:51:57,128 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56522: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:52:21,974 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:52:21,976 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56523: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) 2012-07-20 14:52:43,473 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 49 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 36 SyncTimes(ms): 341 2012-07-20 14:52:43,570 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 2012-07-20 14:52:43,570 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56524: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) I am not Am 18.07.2012 19:47, schrieb Suresh Srinivas: > Can you share information on the java version that you are using. > - Is it as obvious as some previous processes still running and new > processes cannot bind to the port? > - Another pointer - > http://stackoverflow.com/questions/8360913/weird-java-net-socketexception-permission-denied-connect-error-when-running-groo > > On Wed, Jul 18, 2012 at 7:29 AM, Bj�rn-Elmar Macek > > wrote: > > Hi, > > i have lately been running into problems since i started running > hadoop on a cluster: > > The setup is the following: > 1 Computer is NameNode and Jobtracker > 1 Computer is SecondaryNameNode > 2 Computers are TaskTracker and DataNode > > I ran into problems with running the wordcount example: NameNode > and Jobtracker do not start properly both having connection > problems of some kind. > And this is although ssh is configured that way, that no prompt > happens when i connect from any node in the cluster to any other. > > Is there any reason why this happens? > > The logs look like the following: > \________ JOBTRACKER__________________________________________________ > 2012-07-18 16:08:05,808 INFO org.apache.hadoop.mapred.JobTracker: > STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting JobTracker > STARTUP_MSG: host = its-cs100.its.uni-kassel.de/141.51.205.10 > > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.0.2 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 > -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 > ************************************************************/ > 2012-07-18 16:08:06,479 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties > from hadoop-metrics2.properties > 2012-07-18 16:08:06,534 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source MetricsSystem,sub=Stats registered. > 2012-07-18 16:08:06,554 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled > snapshot period at 10 second(s). > 2012-07-18 16:08:06,554 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker > metrics system started > 2012-07-18 16:08:07,157 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source QueueMetrics,q=default registered. > 2012-07-18 16:08:10,395 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source ugi registered. > 2012-07-18 16:08:10,417 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation tokens > 2012-07-18 16:08:10,436 INFO org.apache.hadoop.mapred.JobTracker: > Scheduler configured with (memSizeForMapSlotOnJT, > memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, > limitMaxMemForReduceTasks) (-1, -1, -1, -1) > 2012-07-18 16:08:10,438 INFO > org.apache.hadoop.util.HostsFileReader: Refreshing hosts > (include/exclude) list > 2012-07-18 16:08:10,440 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Starting expired delegation token remover thread, > tokenRemoverScanInterval=60 min(s) > 2012-07-18 16:08:10,465 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation tokens > 2012-07-18 16:08:10,510 INFO org.apache.hadoop.mapred.JobTracker: > Starting jobtracker with owner as bmacek > 2012-07-18 16:08:10,620 WARN org.apache.hadoop.mapred.JobTracker: > Error starting tracker: java.net.SocketException: Permission denied > at sun.nio.ch.Net.bind(Native Method) > at sun.nio.ch > .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119) > at sun.nio.ch > .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at org.apache.hadoop.ipc.Server.bind(Server.java:225) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:301) > at org.apache.hadoop.ipc.Server.(Server.java:1483) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:545) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2306) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2192) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2186) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978) > > 2012-07-18 16:08:13,861 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name > QueueMetrics,q=default already exists! > 2012-07-18 16:08:13,885 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi > already exists! > 2012-07-18 16:08:13,885 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation tokens > 2012-07-18 16:08:13,910 INFO org.apache.hadoop.mapred.JobTracker: > Scheduler configured with (memSizeForMapSlotOnJT, > memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, > limitMaxMemForReduceTasks) (-1, -1, -1, -1) > 2012-07-18 16:08:13,911 INFO > org.apache.hadoop.util.HostsFileReader: Refreshing hosts > (include/exclude) list > 2012-07-18 16:08:13,911 INFO org.apache.hadoop.mapred.JobTracker: > Starting jobtracker with owner as bmacek > 2012-07-18 16:08:13,912 WARN org.apache.hadoop.mapred.JobTracker: > Error starting tracker: java.net.SocketException: Permission denied > at sun.nio.ch.Net.bind(Native Method) > at sun.nio.ch > .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119) > at sun.nio.ch > .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at org.apache.hadoop.ipc.Server.bind(Server.java:225) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:301) > at org.apache.hadoop.ipc.Server.(Server.java:1483) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:545) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2306) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2192) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2186) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978) > > 2012-07-18 16:08:13,912 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Starting expired delegation token remover thread, > tokenRemoverScanInterval=60 min(s) > 2012-07-18 16:08:13,913 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation tokens > 2012-07-18 16:08:21,348 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name > QueueMetrics,q=default already exists! > 2012-07-18 16:08:21,390 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi > already exists! > 2012-07-18 16:08:21,390 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation tokens > 2012-07-18 16:08:21,426 INFO org.apache.hadoop.mapred.JobTracker: > Scheduler configured with (memSizeForMapSlotOnJT, > memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, > limitMaxMemForReduceTasks) (-1, -1, -1, -1) > 2012-07-18 16:08:21,427 INFO > org.apache.hadoop.util.HostsFileReader: Refreshing hosts > (include/exclude) list > 2012-07-18 16:08:21,427 INFO org.apache.hadoop.mapred.JobTracker: > Starting jobtracker with owner as bmacek > 2012-07-18 16:08:21,428 WARN org.apache.hadoop.mapred.JobTracker: > Error starting tracker: java.net.SocketException: Permission denied > at sun.nio.ch.Net.bind(Native Method) > at sun.nio.ch > .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119) > at sun.nio.ch > .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at org.apache.hadoop.ipc.Server.bind(Server.java:225) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:301) > at org.apache.hadoop.ipc.Server.(Server.java:1483) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:545) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2306) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2192) > at > org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2186) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978) > > > \________ DATANODE__________________________________________________ > 2012-07-18 16:07:58,759 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting NameNode > STARTUP_MSG: host = its-cs100.its.uni-kassel.de/141.51.205.10 > > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.0.2 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 > -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 > ************************************************************/ > 2012-07-18 16:07:59,738 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties > from hadoop-metrics2.properties > 2012-07-18 16:07:59,790 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source MetricsSystem,sub=Stats registered. > 2012-07-18 16:07:59,807 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled > snapshot period at 10 second(s). > 2012-07-18 16:07:59,807 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode > metrics system started > 2012-07-18 16:08:00,382 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source ugi registered. > 2012-07-18 16:08:00,454 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source jvm registered. > 2012-07-18 16:08:00,456 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source NameNode registered. > 2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: VM > type = 64-bit > 2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: 2% > max memory = 17.77875 MB > 2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: > capacity = 2^21 = 2097152 entries > 2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: > recommended=2097152, actual=2097152 > 2012-07-18 16:08:00,812 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=bmacek > 2012-07-18 16:08:00,812 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > supergroup=supergroup > 2012-07-18 16:08:00,824 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isPermissionEnabled=true > 2012-07-18 16:08:00,846 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.block.invalidate.limit=100 > 2012-07-18 16:08:00,846 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), > accessTokenLifetime=0 min(s) > 2012-07-18 16:08:02,746 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered > FSNamesystemStateMBean and NameNodeMXBean > 2012-07-18 16:08:02,868 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file > names occuring more than 10 times > 2012-07-18 16:08:02,932 INFO > org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1 > 2012-07-18 16:08:02,963 INFO > org.apache.hadoop.hdfs.server.common.Storage: Number of files > under construction = 0 > 2012-07-18 16:08:02,966 INFO > org.apache.hadoop.hdfs.server.common.Storage: Image file of size > 112 loaded in 0 seconds. > 2012-07-18 16:08:02,975 INFO > org.apache.hadoop.hdfs.server.common.Storage: Edits file > /home/work/bmacek/hadoop/master/current/edits of size 4 edits # 0 > loaded in 0 seconds. > 2012-07-18 16:08:02,977 INFO > org.apache.hadoop.hdfs.server.common.Storage: Image file of size > 112 saved in 0 seconds. > 2012-07-18 16:08:03,191 INFO > org.apache.hadoop.hdfs.server.common.Storage: Image file of size > 112 saved in 0 seconds. > 2012-07-18 16:08:03,334 INFO > org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with > 0 entries 0 lookups > 2012-07-18 16:08:03,334 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished > loading FSImage in 2567 msecs > 2012-07-18 16:08:03,401 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number > of blocks = 0 > 2012-07-18 16:08:03,401 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > invalid blocks = 0 > 2012-07-18 16:08:03,401 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > under-replicated blocks = 0 > 2012-07-18 16:08:03,401 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > over-replicated blocks = 0 > 2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.StateChange: > STATE* Safe mode termination scan for invalid, over- and > under-replicated blocks completed in 61 msec > 2012-07-18 16:08:03,402 INFO org.apache.hadoop.hdfs.StateChange: > STATE* Leaving safe mode after 2 secs. > 2012-07-18 16:08:03,412 INFO org.apache.hadoop.hdfs.StateChange: > STATE* Network topology has 0 racks and 0 datanodes > 2012-07-18 16:08:03,412 INFO org.apache.hadoop.hdfs.StateChange: > STATE* UnderReplicatedBlocks has 0 blocks > 2012-07-18 16:08:03,472 INFO > org.apache.hadoop.util.HostsFileReader: Refreshing hosts > (include/exclude) list > 2012-07-18 16:08:03,488 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > ReplicateQueue QueueProcessingStatistics: First cycle completed 0 > blocks in 1 msec > 2012-07-18 16:08:03,490 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 > blocks in 1 msec processing time, 1 msec clock time, 1 cycles > 2012-07-18 16:08:03,490 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > InvalidateQueue QueueProcessingStatistics: First cycle completed 0 > blocks in 0 msec > 2012-07-18 16:08:03,490 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 > blocks in 0 msec processing time, 0 msec clock time, 1 cycles > 2012-07-18 16:08:03,495 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > source FSNamesystemMetrics registered. > 2012-07-18 16:08:03,553 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > ReplicationMonitor thread received > InterruptedException.java.lang.InterruptedException: sleep interrupted > 2012-07-18 16:08:03,555 INFO > org.apache.hadoop.hdfs.server.namenode.DecommissionManager: > Interrupted Monitor > java.lang.InterruptedException: sleep interrupted > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65) > at java.lang.Thread.run(Thread.java:619) > 2012-07-18 16:08:03,556 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of > transactions: 0 Total time for transactions(ms): 0Number of > transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 > 2012-07-18 16:08:03,594 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: > java.net.SocketException: Permission denied > at sun.nio.ch.Net.bind(Native Method) > at sun.nio.ch > .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119) > at sun.nio.ch > .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at org.apache.hadoop.ipc.Server.bind(Server.java:225) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:301) > at org.apache.hadoop.ipc.Server.(Server.java:1483) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:545) > at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:294) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:496) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) > > 2012-07-18 16:08:03,627 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at > its-cs100.its.uni-kassel.de/141.51.205.10 > > ************************************************************/ > > > > > -- > http://hortonworks.com/download/ > --------------020007060208030508090502 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Hi Srinivas,

thanks for your reply! I have been following your link and idea and been playing around alot, but still got problems with the connection (though they are different now):

\_______ JAVA VERSION_________
"which java" tells me it is 1.6.0_01. If i got it right version 1.7 got problems with ssh.

\_______MY TESTS_____________
According to your suggestion to look for processes running on that port i changed ports alot:
When i was posting the first post of this thread. i was using ports 999 for namenode and 1000 for jobtracker.
Since due to some reasons commands like "lsof -i" etc dont give me any output when usedin the cluster enviroment. So i started looking for ports that are in general unused by programs.
When i changed the ports to 9004 and 9005 i got different errors which look very much like the ones you posted in the beginning of this year in the lucene section ( http://lucene.472066.n3.nabble.com/Unable-to-start-hadoop-0-20-2-but-able-to-start-hadoop-0-20-203-cluster-td2991350.html ).

It seems as if a DataNode can not communicate with the NameNode.

The logs look like the following:

\_______TEST RESULTS__________
########## A DataNode #############
2012-07-20 14:47:59,536 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = its-cs102.its.uni-kassel.de/141.51.205.12
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.0.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-20 14:47:59,824 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-07-20 14:47:59,841 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-07-20 14:47:59,843 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-07-20 14:47:59,844 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2012-07-20 14:47:59,969 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-07-20 14:48:26,792 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: its-cs100/141.51.205.10:9005. Already tried 0 time(s).
2012-07-20 14:48:26,889 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
2012-07-20 14:48:26,934 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
2012-07-20 14:48:26,949 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2012-07-20 14:48:27,014 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-07-20 14:48:27,147 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-07-20 14:48:27,160 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
2012-07-20 14:48:27,160 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
2012-07-20 14:48:27,160 INFO org.mortbay.log: jetty-6.1.26
2012-07-20 14:48:27,805 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
2012-07-20 14:48:27,811 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2012-07-20 14:48:27,813 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source DataNode registered.
2012-07-20 14:48:28,484 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort50020 registered.
2012-07-20 14:48:28,484 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort50020 registered.
2012-07-20 14:48:28,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(its-cs102.its.uni-kassel.de:50010, storageID=, infoPort=50075, ipcPort=50020)
2012-07-20 14:48:28,489 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2012-07-20 14:48:38,706 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

    at org.apache.hadoop.ipc.Client.call(Client.java:1066)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy5.register(Unknown Source)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:673)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1480)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1540)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

2012-07-20 14:48:38,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at its-cs102.its.uni-kassel.de/141.51.205.12
************************************************************/


####### NameNode ##########################
CAUTION: Please recognize, that the file mentioned in the first error log message (/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info) does not exist on the NameNode, when i checked for it.
The only path that has a simiar name is: /home/work/bmacek/hadoop/hdfs/slave/tmp (containing no further subfolders or files)



2012-07-20 14:47:58,033 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = its-cs100.its.uni-kassel.de/141.51.205.10
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.0.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-20 14:47:58,985 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-07-20 14:47:59,037 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-07-20 14:47:59,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-07-20 14:47:59,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2012-07-20 14:47:59,622 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-07-20 14:47:59,685 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2012-07-20 14:47:59,703 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source NameNode registered.
2012-07-20 14:47:59,896 INFO org.apache.hadoop.hdfs.util.GSet: VM type       = 64-bit
2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: capacity      = 2^21 = 2097152 entries
2012-07-20 14:47:59,897 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=bmacek
2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2012-07-20 14:48:00,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2012-07-20 14:48:00,083 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
2012-07-20 14:48:00,084 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2012-07-20 14:48:01,573 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean
2012-07-20 14:48:01,643 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2012-07-20 14:48:01,686 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1
2012-07-20 14:48:01,712 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
2012-07-20 14:48:01,713 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 112 loaded in 0 seconds.
2012-07-20 14:48:01,796 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Invalid opcode, reached end of edit log Number of transactions found 53
2012-07-20 14:48:01,796 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /home/work/bmacek/hadoop/master/current/edits of size 1049092 edits # 53 loaded in 0 seconds.
2012-07-20 14:48:01,797 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 861 saved in 0 seconds.
2012-07-20 14:48:02,003 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 861 saved in 0 seconds.
2012-07-20 14:48:02,159 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 1 entries 11 lookups
2012-07-20 14:48:02,159 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 2135 msecs
2012-07-20 14:48:02,203 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of  over-replicated blocks = 0
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode termination scan for invalid, over- and under-replicated blocks completed in 44 msec
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 2 secs.
2012-07-20 14:48:02,204 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
2012-07-20 14:48:02,205 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2012-07-20 14:48:02,265 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-07-20 14:48:02,275 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec
2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles
2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec
2012-07-20 14:48:02,277 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles
2012-07-20 14:48:02,281 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source FSNamesystemMetrics registered.
2012-07-20 14:48:02,336 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9005 registered.
2012-07-20 14:48:02,337 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9005 registered.
2012-07-20 14:48:02,341 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2012-07-20 14:48:02,356 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: its-cs100.its.uni-kassel.de/141.51.205.10:9005
2012-07-20 14:48:02,878 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-07-20 14:48:03,312 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-07-20 14:48:03,426 INFO org.apache.hadoop.http.HttpServer: dfs.webhdfs.enabled = false
2012-07-20 14:48:03,465 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2012-07-20 14:48:03,511 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2012-07-20 14:48:03,511 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2012-07-20 14:48:03,511 INFO org.mortbay.log: jetty-6.1.26
2012-07-20 14:48:06,528 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50070
2012-07-20 14:48:06,528 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070
2012-07-20 14:48:06,561 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2012-07-20 14:48:06,593 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9005: starting
2012-07-20 14:48:06,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005: starting
2012-07-20 14:48:06,685 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9005: starting
2012-07-20 14:48:06,731 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005: starting
2012-07-20 14:48:06,759 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9005: starting
2012-07-20 14:48:06,791 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9005: starting
2012-07-20 14:48:06,849 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005: starting
2012-07-20 14:48:06,874 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9005: starting
2012-07-20 14:48:06,898 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005: starting
2012-07-20 14:48:06,921 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9005: starting
2012-07-20 14:48:06,974 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005: starting
2012-07-20 14:48:27,222 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:48:27,224 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56513: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:48:38,701 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010
2012-07-20 14:48:38,701 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call register(DatanodeRegistration(its-cs102.its.uni-kassel.de:50010, storageID=DS-1791721778-141.51.205.12-50010-1342788518692, infoPort=50075, ipcPort=50020)) from 141.51.205.12:33789: error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs102.its.uni-kassel.de:50010
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:48:54,331 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:48:54,331 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56514: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:49:18,079 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 13 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 9 SyncTimes(ms): 111
2012-07-20 14:49:18,151 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:49:18,151 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56515: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:49:41,419 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:49:41,419 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56516: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:50:04,474 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:50:04,474 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56517: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:50:26,299 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 25 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 170
2012-07-20 14:50:26,359 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:50:26,359 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56518: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:50:47,243 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:50:47,243 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56519: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:51:06,865 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010
2012-07-20 14:51:06,865 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9005, call register(DatanodeRegistration(its-cs103.its.uni-kassel.de:50010, storageID=DS-1725464844-141.51.205.13-50010-1342788666863, infoPort=50075, ipcPort=50020)) from 141.51.205.13:48227: error: org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: its-cs103.its.uni-kassel.de:50010
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2391)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.register(NameNode.java:973)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:51:08,305 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:51:08,305 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56520: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:51:34,855 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 37 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 27 SyncTimes(ms): 256
2012-07-20 14:51:34,932 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:51:34,932 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56521: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:51:57,128 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:51:57,128 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56522: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:52:21,974 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:52:21,976 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56523: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
2012-07-20 14:52:43,473 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 49 Total time for transactions(ms): 22Number of transactions batched in Syncs: 0 Number of syncs: 36 SyncTimes(ms): 341
2012-07-20 14:52:43,570 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bmacek cause:java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
2012-07-20 14:52:43,570 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9005, call addBlock(/home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info, DFSClient_-1997886712, null) from 141.51.205.10:56524: error: java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/work/bmacek/hadoop/hdfs/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)





I am not

Am 18.07.2012 19:47, schrieb Suresh Srinivas:
Can you share information on the java version that you are using.
- Is it as obvious as some previous processes still running and new processes cannot bind to the port?
- Another pointer - http://stackoverflow.com/questions/8360913/weird-java-net-socketexception-permission-denied-connect-error-when-running-groo

On Wed, Jul 18, 2012 at 7:29 AM, Björn-Elmar Macek <macek@cs.uni-kassel.de> wrote:
Hi,

i have lately been running into problems since i started running hadoop on a cluster:

The setup is the following:
1 Computer is NameNode and Jobtracker
1 Computer is SecondaryNameNode
2 Computers are TaskTracker and DataNode

I ran into problems with running the wordcount example: NameNode and Jobtracker do not start properly both having connection problems of some kind.
And this is although ssh is configured that way, that no prompt happens when i connect from any node in the cluster to any other.

Is there any reason why this happens?

The logs look like the following:
\________ JOBTRACKER__________________________________________________
2012-07-18 16:08:05,808 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host = its-cs100.its.uni-kassel.de/141.51.205.10
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.0.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-18 16:08:06,479 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-07-18 16:08:06,534 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-07-18 16:08:06,554 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-07-18 16:08:06,554 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2012-07-18 16:08:07,157 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
2012-07-18 16:08:10,395 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-07-18 16:08:10,417 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-07-18 16:08:10,436 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-07-18 16:08:10,438 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-07-18 16:08:10,440 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2012-07-18 16:08:10,465 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-07-18 16:08:10,510 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as bmacek
2012-07-18 16:08:10,620 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: java.net.SocketException: Permission denied
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:225)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2306)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2012-07-18 16:08:13,861 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name QueueMetrics,q=default already exists!
2012-07-18 16:08:13,885 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2012-07-18 16:08:13,885 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-07-18 16:08:13,910 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-07-18 16:08:13,911 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-07-18 16:08:13,911 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as bmacek
2012-07-18 16:08:13,912 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: java.net.SocketException: Permission denied
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:225)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2306)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2012-07-18 16:08:13,912 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2012-07-18 16:08:13,913 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-07-18 16:08:21,348 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name QueueMetrics,q=default already exists!
2012-07-18 16:08:21,390 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2012-07-18 16:08:21,390 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-07-18 16:08:21,426 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-07-18 16:08:21,427 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-07-18 16:08:21,427 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as bmacek
2012-07-18 16:08:21,428 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: java.net.SocketException: Permission denied
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:225)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2306)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)


\________ DATANODE__________________________________________________
2012-07-18 16:07:58,759 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = its-cs100.its.uni-kassel.de/141.51.205.10
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.0.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012
************************************************************/
2012-07-18 16:07:59,738 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-07-18 16:07:59,790 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-07-18 16:07:59,807 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-07-18 16:07:59,807 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2012-07-18 16:08:00,382 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-07-18 16:08:00,454 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2012-07-18 16:08:00,456 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source NameNode registered.
2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: VM type       = 64-bit
2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: capacity      = 2^21 = 2097152 entries
2012-07-18 16:08:00,645 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
2012-07-18 16:08:00,812 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=bmacek
2012-07-18 16:08:00,812 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2012-07-18 16:08:00,824 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2012-07-18 16:08:00,846 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
2012-07-18 16:08:00,846 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2012-07-18 16:08:02,746 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean
2012-07-18 16:08:02,868 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2012-07-18 16:08:02,932 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1
2012-07-18 16:08:02,963 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
2012-07-18 16:08:02,966 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 112 loaded in 0 seconds.
2012-07-18 16:08:02,975 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /home/work/bmacek/hadoop/master/current/edits of size 4 edits # 0 loaded in 0 seconds.
2012-07-18 16:08:02,977 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 112 saved in 0 seconds.
2012-07-18 16:08:03,191 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 112 saved in 0 seconds.
2012-07-18 16:08:03,334 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2012-07-18 16:08:03,334 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 2567 msecs
2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0
2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0
2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0
2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0
2012-07-18 16:08:03,401 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode termination scan for invalid, over- and under-replicated blocks completed in 61 msec
2012-07-18 16:08:03,402 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 2 secs.
2012-07-18 16:08:03,412 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
2012-07-18 16:08:03,412 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2012-07-18 16:08:03,472 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-07-18 16:08:03,488 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec
2012-07-18 16:08:03,490 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles
2012-07-18 16:08:03,490 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec
2012-07-18 16:08:03,490 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles
2012-07-18 16:08:03,495 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source FSNamesystemMetrics registered.
2012-07-18 16:08:03,553 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted
2012-07-18 16:08:03,555 INFO org.apache.hadoop.hdfs.server.namenode.DecommissionManager: Interrupted Monitor
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
    at java.lang.Thread.run(Thread.java:619)
2012-07-18 16:08:03,556 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
2012-07-18 16:08:03,594 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.SocketException: Permission denied
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:225)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:294)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)

2012-07-18 16:08:03,627 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at its-cs100.its.uni-kassel.de/141.51.205.10
************************************************************/



--
http://hortonworks.com/download/



--------------020007060208030508090502--