hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rkevinbur...@charter.net
Subject Re: Can't initialize cluster
Date Wed, 01 May 2013 13:42:27 GMT

  I made these changes. The log files in /var/log/hadoop-0.20-mapreduce 
do not reflect todays date so they are not updated. The datanode log in 
/var/log/hadoop-hdfs only shows this warning:

2013-05-01 08:16:12,678 WARN 
org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to 
server: devUbuntu05/172.16.26.68:9000

The entries in the log after that (and there are many) all are INFO 
messages. Not more warnings or errors.

The namenode log is just INFO messages.

The secondarynamenode log (on the node that is acting as the secondary 
namenode) contains the following error exception:

2013-05-01 08:34:16,431 ERROR 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in 
doCheckpoint
java.io.IOException: Inconsistent checkpoint fields.
LV = -40 namespaceID = 1097063765 cTime = 0 ; clusterId = 
CID-23b9f9c7-2c25-411f-8bd2-4d5c9d7c25a1 ; blockpoolId = 
BP-1306349046-172.16.26.68-1367256199559.
Expecting respectively: -40; 1666002936; 0; 
CID-e3f6b811-c1b4-4778-a31e-14dea8b2cca8; 
BP-1964515668-172.16.26.68-1366403021016.
         at 
org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:133)
         at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:475)

The logs for the any of the nodes serving as the task tracker are out of 
date.

When I try to attach to the master with a browser on port 50070 to look 
at the HDFS the page responds but when I try to connect to the master at 
port 50030 nothing is listening so it seems the JobTracker is down.

So it seems from what I can gather the reason that the JobTracker 
doesn't come up is because of the error on the secondary name node. Do 
you understand how I can get rid of this error?

Kevin

On Tue, Apr 30, 2013 at 8:55 PM, Mohammad Tariq wrote:

  Are you sure about the case in  devuBuntu06
devuBuntu07
devuBuntu08

Is 'B' actually in upper case?

Also,  mapred.local.dir  belongs to mapred-site.xml and  hadoop.tmp.dir 
belongs to core-site.xml file.

Warm Regards, Tariq
https://mtariq.jux.com/ <https://mtariq.jux.com/>
  <https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com>
  <http://cloudfront.blogspot.com>

On Wed, May 1, 2013 at 7:09 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote:
Here are what I consider the most applicable conf files. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Again thank you for your help. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Kevin 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 6:43 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Sorry for the late response Kevin. I was away for a while. I saw the 
logs. There seems to be some problem with your configuration or name 
resolution. Could you please show me your configuration files? 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 4:27 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
The JobTracker logs don’t seem to be updated: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:/etc/hadoop/conf$ cd /var/log/hadoop-0.20-mapreduce/ 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')> 
ls -l 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
total 3028 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
-rw-rw-r-- 1 mapred mapred  383838 Apr 30 15:11 
hadoop-hadoop-jobtracker-devUbuntu05.log 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
  
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
But I have attached the last lines of the JT log along with the TT logs. 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
  
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
Kevin 
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
  
<javascript:parent.wgMail.openComposeWindow('kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 5:38 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
sorry for the typo in the last mail. it was 50070 and not 
9000..anyways.. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
it's correct..it shows only the slaves..can you show me the JT and TT 
logs? 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 4:00 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
There is something running. Here is the master: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~$ ps aux | grep hadoop 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
hdfs      2004  0.3  4.4 1977848 178396 ?      Sl   16:59   0:07 
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Dproc_namenode -Xmx1000m 
-Dhadoop.log.dir=/var/log/hadoop-hdfs 
-Dhadoop.log.file=hadoop-hdfs-namenode-devUbuntu05.log 
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs 
-Dhadoop.root.logger=INFO,RFA 
-Djava.library.path=/usr/lib/hadoop/lib/native 
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote -Dhadoop.security.logger=INFO,RFAS 
org.apache.hadoop.hdfs.server.namenode.NameNode 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin     2743  0.0  0.0   9384   924 pts/0    S+   17:28   0:00 grep 
--color=auto hadoop 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 5:21 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
if jps is showing just "jps" as output that means nothing is running. 
you have one more way to cross check. point your web browser to  
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
"namenode_machine:9000" and see if the cluster is behaving properly. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
if you don't mind, can we switch to chat?? 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 3:48 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
There must be some option to run jps because running jps on all machines 
give something like: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~/WordCount$ jps 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
2398 Jps 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 5:11 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Do jps and see which process is not running. Show me the corresponding 
log.  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 3:36 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
The task tracker and job tracker did give any indication of success or 
failure when I started them. The namenode, secondarynamenode, datanode 
daemons all gave back a message the they were started. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 4:45 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Ok..things are getting complex..we'll make sure that you cluster is 
running fine first. stop everything and restart all 5 daemons and let me 
know if you are able to do it properly. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 3:10 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
This didn’t seem to help. I get about 10 retries then (running the job): 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~/WordCount$ hadoop jar WordCount.jar 
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output
. . . . .
Exception in thread "main" java.net.ConnectException: Call From 
devUbuntu05/ 172.16.26.68 <http://172.16.26.68> to devUbuntu05:9001 
failed on connection exception: java.net.ConnectException: Connection 
refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused 
<http://wiki.apache.org/hadoop/ConnectionRefused> 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)  <http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
java.lang.reflect.Constructor.newInstance(Constructor.java:525) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:779) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:726) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at org.apache.hadoop.ipc.Client.call(Client.java:1229) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEng 
<http://wiki.apache.org/hadoop/ConnectionRefused>
   <http://wiki.apache.org/hadoop/ConnectionRefused>
But still no indication why the JobTracker will not start. 
<http://wiki.apache.org/hadoop/ConnectionRefused>
   <http://wiki.apache.org/hadoop/ConnectionRefused>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 4:28 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Do this : 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
hadoop dfsadmin -safemode leave 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Then restart the daemons and re-run the job. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 2:50 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Now it looks like the JobTracker isn’t start at all and there doesn’t 
seem to be a log entry in /var/log/hadoop-0.20-mapreduce to indicate the 
reason. I looked in /var/log/hadoop-hdfs and for the namenode I found 
what looked like an exception : 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
The reported blocks 0 needs additional 4 blocks to reach the threshold 
0.9990 of total blocks 4. Safe mode will be turned off automatically. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4568) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:742) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:129) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8762) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1695) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1691) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Then I looked in the datanode log and I didn’t see what looked like an 
exception or error. The secondary name node log looked like: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
2013-04-29 14:11:25,465 ERROR 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in 
doCheckpoint 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.net.ConnectException: Call From devUbuntu05/ 172.16.26.68 
<http://172.16.26.68> to devUbuntu05:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused 
<http://wiki.apache.org/hadoop/ConnectionRefused> 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown 
Source)  <http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
java.lang.reflect.Constructor.newInstance(Constructor.java:525) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:779) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
        at 
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:726) 
<http://wiki.apache.org/hadoop/ConnectionRefused>
   <http://wiki.apache.org/hadoop/ConnectionRefused>
   <http://wiki.apache.org/hadoop/ConnectionRefused>
   <http://wiki.apache.org/hadoop/ConnectionRefused>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 3:41 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
add the libraries to your class path and re-run the job. if it still 
doesn't work, add all the libraries individually on the command line 
while running the job. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 2:01 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Eliminating it then takes the first file in $HADOOP_HOME/*.jar as the 
jar containing ‘main’ and I get another error: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~/WordCount$ hadoop jar 
$HADOOP_HOME/*.jar:$HADOOP_HOME/lib/* WordCount.jar 
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output
Exception in thread "main" java.io.IOException: Error opening job jar: 
/usr/lib/hadoop/*.jar:/usr/lib/hadoop/lib/*
        at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.io.FileNotFoundException: 
/usr/lib/hadoop/*.jar:/usr/lib/hadoop/lib/* (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:214)
        at java.util.zip.ZipFile.<init>(ZipFile.java:144)
        at java.util.jar.JarFile.<init>(JarFile.java:153)
        at java.util.jar.JarFile.<init>(JarFile.java:90)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
 
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 3:27 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
it was a typo..eliminate it 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 1:55 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Stilll the ‘-‘ is not accepted on my end. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~/WordCount$ hadoop jar - 
$HADOOP_HOME/*.jar:$HADOOP_HOME/lib/* WordCount.jar 
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output
Exception in thread "main" java.io.IOException : Error opening job jar: 
-
        at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.io.FileNotFoundException: - (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:214)
        at java.util.zip.ZipFile.<init>(ZipFile.java:144)
        at java.util.jar.JarFile.<init>(JarFile.java:153)
        at java.util.jar.JarFile.<init>(JarFile.java:90)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
 
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 3:23 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Sorry. My bad. I forgot that you are using CDH. use this : 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
hadoop jar - $HADOOP_HOME/*.jar:$HADOOP_HOME/lib/* WordCount.jar  
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output

Warm Regards,
Tariq
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 1:47 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
I look at what could be HADOOP_HOME. It looks like: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
kevin@devUbuntu05:~$ ls /usr/lib/hadoop/ 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
bin                                    hadoop-auth.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
client                                 hadoop-common-2.0.0-cdh4.2.1.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
client-0.20                            
hadoop-common-2.0.0-cdh4.2.1-tests.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
cloudera                               hadoop-common.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
etc                                    lib 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
hadoop-annotations-2.0.0-cdh4.2.1.jar  libexec 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
hadoop-annotations.jar                 sbin 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
hadoop-auth-2.0.0-cdh4.2.1.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
But notice there isn’t a hadoop-core-*.jar. Have I got the right folder? 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Also when I run the command as you suggested I immediately get an error 
that ‘-‘ doesn’t exist. If I eliminate the ‘-‘ then there has to be 
someway to pass in the what looks like you are forming a CLASSPATH. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 2:51 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
"HADOOP_HOME is depricated" is not a problem. Just a warning. Add this 
line "export HADOOP_HOME_WARN_SUPPRESS=true" into your hadoop-env.sh 
file if you want to get rid of it. It's the path where your hadoop 
folder, which contain all the binaries, is present. 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Use this command and see if it helps : 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
hadoop jar - 
$HADOOP_HOME/hadoop-core-*.jar:$HADOOP_HOME/lib/* WordCount.jar  
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output
 
Assuming you are already inside  /home/kevin/WordCount/

Warm Regards,
Tariq
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 1:01 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
The path to the jar is /home/kevin/WordCount/WorCount.jar 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
I have not set HADOOP_HOME because on an earlier installation (the 
Apache release not the CDH distro) I kept getting a message that 
HADOOP_HOME was deprecated.  Should it be set? If so should its path be 
the same as HADOOP_MAPRED_HOME? 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
The command that throws the exception is hadoop jar WordCount.jar 
file:///home/kevin/WordCount/inputfile:///home/kevin/WordCount/output
 
Thanks again.
 
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 2:19 PM 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

To: Kevin Burton
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Could you please answer these 3 questions : 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
1-Where is your jar located?(I mean the directory path) 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
2-Have you set  HADOOP_HOME? If yes then what is the path? 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
3-What is the command which you are using? 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Wed, May 1, 2013 at 12:42 AM, Kevin Burton < rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Tariq, 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Thank you. I tried this and the summary of the map reduce job looks 
like: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient: Job complete: 
job_201304301251_0004 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient: Counters: 7 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:   Job Counters 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Failed map tasks=1 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Launched map tasks=27 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Rack-local map tasks=27 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Total time spent by all 
maps in occupied slots (ms)=151904 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Total time spent by all 
reduces in occupied slots (ms)=0 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:35 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
But there were a number of exceptions thrown and it seemed to take 
longer than just running it standalone (I should have at least 4 
machines working on this). The exceptions are my main concern now: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
(there were quite a few) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:27 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000005_1, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/hadoop-core-2.0.0-mr1-cdh4.2.1.jar does 
not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:28 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000006_1, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/guava-11.0.2.jar does not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:28 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000008_0, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/zookeeper-3.4.5-cdh4.2.1.jar does not 
exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:28 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000001_2, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/tools.jar does not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:28 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000000_2, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/Websters.txt does not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:33 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000002_2, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/hadoop-hdfs-2.0.0-cdh4.2.1.jar does not 
exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
. . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:33 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000004_2, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/hadoop-common-2.0.0-cdh4.2.1.jar does 
not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
.  . . . 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 14:02:33 INFO mapred.JobClient: Task Id : 
attempt_201304301251_0004_m_000003_2, Status : FAILED 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java.io.FileNotFoundException: File 
file:/home/kevin/WordCount/input/core-3.1.1.jar does not exist 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
No output folder was created (probably because of the numerous errors). 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Kevin 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 1:32 PM
To: Kevin Burton 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>

Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
Hello again Kevin, 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
     Good that you are making progress. This is happening because when 
you are running it as a hadoop job, it looks for the the files in HDFS 
and when you run it as a job program it looks into the local FS. Use 
this as your input in your code and see if it helps :  
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
   <javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')>
file:///home/kevin/input
 

Warm Regards,
Tariq
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Tue, Apr 30, 2013 at 11:36 PM, Kevin Burton < 
rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
We/I are/am making progress. Now I get the error: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 12:59:40 WARN mapred.JobClient: Use GenericOptionsParser for 
parsing the arguments. Applications should implement Tool for the same. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 12:59:40 INFO mapred.JobClient: Cleaning up the staging area 
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/staging/kevin/.staging/job_201304301251_0003 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
13/04/30 12:59:40 ERROR security.UserGroupInformation: 
PriviledgedActionException as:kevin (auth:SIMPLE) 
cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input 
path does not exist: hdfs://devubuntu05:9000/user/kevin/input 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Exception in thread "main" 
org.apache.hadoop.mapreduce.lib.input.InvalidInputException : Input path 
does not exist: hdfs://devubuntu05:9000/user/kevin/input 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
When I run it with java –jar the input and output is the local folder. 
When running it with hadoop jar it seems to be expecting the folders 
(input and output) to be on the HDFS file system. I am not sure why 
these two methods of invocation don’t make the same file system 
assumptions. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
It is 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
hadoop jar WordCount.jar input output (which gives the above exception) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
versus 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
java –jar WordCount.jar input output (which outputs the wordcount 
statistics to the output folder) 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
This is run in the local /home/kevin/WordCount folder. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
Kevin 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Mohammad Tariq [mailto: dontariq@gmail.com 
<javascript:parent.wgMail.openComposeWindow('dontariq@gmail.com')> ]
Sent: Tuesday, April 30, 2013 12:33 PM
To: user@hadoop.apache.org 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
Subject: Re: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
  
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
Set " HADOOP_MAPRED_HOME " in your hadoop-env.sh file and re-run the 
job. See if it helps. 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>

  <javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
Warm Regards, 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
Tariq 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
https://mtariq.jux.com/ <https://mtariq.jux.com/> 
<https://mtariq.jux.com/>
cloudfront.blogspot.com <http://cloudfront.blogspot.com> 
<http://cloudfront.blogspot.com>
   <http://cloudfront.blogspot.com>
On Tue, Apr 30, 2013 at 10:10 PM, Kevin Burton < 
rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
To be clear when this code is run with ‘java –jar’ it runs without 
exception. The exception occurs when I run with ‘hadoop jar’. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')>
From: Kevin Burton [mailto: rkevinburton@charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton@charter.net')> 
]
Sent: Tuesday, April 30, 2013 11:36 AM
To: user@hadoop.apache.org 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
Subject: Can't initialize cluster 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
  
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
I have a simple MapReduce job that I am trying to get to run on my 
cluster. When I run it I get: 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
  
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
13/04/30 11:27:45 INFO mapreduce.Cluster: Failed to use 
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: 
Invalid "mapreduce.jobtracker.address" configuration value for 
LocalJobRunner : "devubuntu05:9001" 
<javascript:parent.wgMail.openComposeWindow('user@hadoop.apache.org')>
13/04/30 11:27:45 ERROR security.UserGroupInformation: 
PriviledgedActionException as:kevin (auth:SIMPLE) 
cause:java.io.IOException: Cannot initialize Cluster . Please check your 
configuration for mapreduce.framework.name 
<http://mapreduce.framework.name> and the correspond server addresses. 
<http://mapreduce.framework.name>
Exception in thread "main" java.io.IOException: Cannot initialize 
Cluster. Please check your configuration for mapreduce.framework.name 
<http://mapreduce.framework.name> and the correspond server addresses. 
<http://mapreduce.framework.name>
   <http://mapreduce.framework.name>
My core-site.xml looks like:  <http://mapreduce.framework.name>
   <http://mapreduce.framework.name>
<property>  <http://mapreduce.framework.name>
  <name> fs.default.name <http://fs.default.name> </name> 
<http://fs.default.name>
  <value>hdfs://devubuntu05:9000</value>  <http://fs.default.name>
  <description>The name of the default file system. A URI whose scheme 
and authority determine the FileSystem implementation. </description> 
<http://fs.default.name>
</property>  <http://fs.default.name>
   <http://fs.default.name>
So I am unclear as to why it is looking at devubuntu05:9001? 
<http://fs.default.name>
   <http://fs.default.name>
Here is the code:  <http://fs.default.name>
   <http://fs.default.name>
    public static void WordCount( String[] args )  throws Exception { 
<http://fs.default.name>
        Configuration conf = new Configuration(); 
<http://fs.default.name>
        String[] otherArgs = new GenericOptionsParser(conf, 
args).getRemainingArgs();  <http://fs.default.name>
        if (otherArgs.length != 2) {  <http://fs.default.name>
            System.err.println("Usage: wordcount <in> <out>"); 
<http://fs.default.name>
            System.exit(2);  <http://fs.default.name>
        }  <http://fs.default.name>
        Job job = new Job(conf, "word count");  <http://fs.default.name>
        job.setJarByClass(WordCount.class);  <http://fs.default.name>
        job.setMapperClass(WordCount.TokenizerMapper.class); 
<http://fs.default.name>
        job.setCombinerClass(WordCount.IntSumReducer.class); 
<http://fs.default.name>
        job.setReducerClass(WordCount.IntSumReducer.class); 
<http://fs.default.name>
        job.setOutputKeyClass(Text.class);  <http://fs.default.name>
        job.setOutputValueClass(IntWritable.class); 
<http://fs.default.name>
        
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(job, 
new Path(otherArgs[0]));  <http://fs.default.name>
        
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, 
new Path(otherArgs[1]));  <http://fs.default.name>
        System.exit(job.waitForCompletion(true) ? 0 : 1); 
<http://fs.default.name>
   <http://fs.default.name>
Ideas?  <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>
   <http://fs.default.name>

Mime
View raw message