hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From A Df <abbey_dragonfor...@yahoo.com>
Subject Re: cygwin not connecting to Hadoop server
Date Wed, 27 Jul 2011 17:00:50 GMT
See inline at **. More questions and many Thanks :D




>________________________________
>From: Uma Maheswara Rao G 72686 <maheswara@huawei.com>
>To: common-user@hadoop.apache.org; A Df <abbey_dragonforest@yahoo.com>
>Cc: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
>Sent: Wednesday, 27 July 2011, 17:31
>Subject: Re: cygwin not connecting to Hadoop server
>
>
>Hi A Df,
>
>Did you format the NameNode first?
>
>** I had formatted it already but then I had reinstalled Java and upgraded the plugins
in cygwin so I reformatted it again. :D yes it worked!! I am not sure all the steps that got
it to finally work but I will have to document it to prevent this headache in the future.
Although I typed ssh localhost too , so question is, do I need to type ssh localhost each
time I need to run hadoop?? Also, since I need to work with Eclipse maybe you can have a look
at my post about the plugin cause I can get the patch to work. The subject is "Re: Cygwin
not working with Hadoop and Eclipse Plugin". I plan to read up on how to write programs for
Hadoop. I am using the tutorial at Yahoo but if you know of any really good about coding with
Hadoop or just about understanding Hadoop then please let me know.
>
>Can you check the NN logs whether NN is started or not?
>** I checked and the previous runs had some logs missing but now the last one have all
5 logs and I got two conf files in xml. I also copied out the other output files which I plan
to examine. Where do I specify the output extension format that I want for my output file?
I was hoping for an txt file it shows the output in a file with no extension even though I
can read it in Notepad++. I also got to view the web interface at:
>    NameNode - http://localhost:50070/
>    JobTracker - http://localhost:50030/
>
>** See below for the working version, finally!! Thanks
><CMD>
>Williams@TWilliams-LTPC ~/hadoop-0.20.2
>$ bin/hadoop jar hadoop-0.20.2-examples.jar grep input
>11/07/27 17:42:20 INFO mapred.FileInputFormat: Total in
>
>11/07/27 17:42:20 INFO mapred.JobClient: Running job: j
>11/07/27 17:42:21 INFO mapred.JobClient:  map 0% reduce
>11/07/27 17:42:33 INFO mapred.JobClient:  map 15% reduc
>11/07/27 17:42:36 INFO mapred.JobClient:  map 23% reduc
>11/07/27 17:42:39 INFO mapred.JobClient:  map 38% reduc
>11/07/27 17:42:42 INFO mapred.JobClient:  map 38% reduc
>11/07/27 17:42:45 INFO mapred.JobClient:  map 53% reduc
>11/07/27 17:42:48 INFO mapred.JobClient:  map 69% reduc
>11/07/27 17:42:51 INFO mapred.JobClient:  map 76% reduc
>11/07/27 17:42:54 INFO mapred.JobClient:  map 92% reduc
>11/07/27 17:42:57 INFO mapred.JobClient:  map 100% redu
>11/07/27 17:43:06 INFO mapred.JobClient:  map 100% redu
>11/07/27 17:43:09 INFO mapred.JobClient: Job complete:
>11/07/27 17:43:09 INFO mapred.JobClient: Counters: 18
>11/07/27 17:43:09 INFO mapred.JobClient:   Job Counters
>11/07/27 17:43:09 INFO mapred.JobClient:     Launched r
>11/07/27 17:43:09 INFO mapred.JobClient:     Launched m
>11/07/27 17:43:09 INFO mapred.JobClient:     Data-local
>11/07/27 17:43:09 INFO mapred.JobClient:   FileSystemCo
>11/07/27 17:43:09 INFO mapred.JobClient:     FILE_BYTES
>11/07/27 17:43:09 INFO mapred.JobClient:     HDFS_BYTES
>11/07/27 17:43:09 INFO mapred.JobClient:     FILE_BYTES
>11/07/27 17:43:09 INFO mapred.JobClient:     HDFS_BYTES
>11/07/27 17:43:09 INFO mapred.JobClient:   Map-Reduce F
>11/07/27 17:43:09 INFO mapred.JobClient:     Reduce inp
>11/07/27 17:43:09 INFO mapred.JobClient:     Combine ou
>11/07/27 17:43:09 INFO mapred.JobClient:     Map input
>11/07/27 17:43:09 INFO mapred.JobClient:     Reduce shu
>11/07/27 17:43:09 INFO mapred.JobClient:     Reduce out
>11/07/27 17:43:09 INFO mapred.JobClient:     Spilled Re
>11/07/27 17:43:09 INFO mapred.JobClient:     Map output
>11/07/27 17:43:09 INFO mapred.JobClient:     Map input
>11/07/27 17:43:09 INFO mapred.JobClient:     Combine in
>11/07/27 17:43:09 INFO mapred.JobClient:     Map output
>11/07/27 17:43:09 INFO mapred.JobClient:     Reduce inp
>11/07/27 17:43:09 WARN mapred.JobClient: Use GenericOpt
>e arguments. Applications should implement Tool for the
>11/07/27 17:43:09 INFO mapred.FileInputFormat: Total in
>11/07/27 17:43:09 INFO mapred.JobClient: Running job: j
>11/07/27 17:43:10 INFO mapred.JobClient:  map 0% reduce
>11/07/27 17:43:22 INFO mapred.JobClient:  map 100% redu
>11/07/27 17:43:31 INFO mapred.JobClient:  map 100% redu
>11/07/27 17:43:36 INFO mapred.JobClient:  map 100% redu
>11/07/27 17:43:38 INFO mapred.JobClient: Job complete:
>11/07/27 17:43:39 INFO mapred.JobClient: Counters: 18
>11/07/27 17:43:39 INFO mapred.JobClient:   Job Counters
>11/07/27 17:43:39 INFO mapred.JobClient:     Launched r
>11/07/27 17:43:39 INFO mapred.JobClient:     Launched m
>11/07/27 17:43:39 INFO mapred.JobClient:     Data-local
>11/07/27 17:43:39 INFO mapred.JobClient:   FileSystemCo
>11/07/27 17:43:39 INFO mapred.JobClient:     FILE_BYTES
>11/07/27 17:43:39 INFO mapred.JobClient:     HDFS_BYTES
>11/07/27 17:43:39 INFO mapred.JobClient:     FILE_BYTES
>11/07/27 17:43:39 INFO mapred.JobClient:     HDFS_BYTES
>11/07/27 17:43:39 INFO mapred.JobClient:   Map-Reduce F
>11/07/27 17:43:39 INFO mapred.JobClient:     Reduce inp
>11/07/27 17:43:39 INFO mapred.JobClient:     Combine ou
>11/07/27 17:43:39 INFO mapred.JobClient:     Map input
>11/07/27 17:43:39 INFO mapred.JobClient:     Reduce shu
>11/07/27 17:43:39 INFO mapred.JobClient:     Reduce out
>11/07/27 17:43:39 INFO mapred.JobClient:     Spilled Re
>11/07/27 17:43:39 INFO mapred.JobClient:     Map output
>11/07/27 17:43:39 INFO mapred.JobClient:     Map input
>11/07/27 17:43:39 INFO mapred.JobClient:     Combine in
>11/07/27 17:43:39 INFO mapred.JobClient:     Map output
>11/07/27 17:43:39 INFO mapred.JobClient:     Reduce inp
>
>Williams@TWilliams-LTPC ~/hadoop-0.20.2
>$ bin/hadoop fs -get output output
>
>Williams@TWilliams-LTPC ~/hadoop-0.20.2
>$ cat output/*
>cat: output/_logs: Is a directory
>3       dfs.class
>2       dfs.period
>1       dfs.file
>1       dfs.replication
>1       dfs.servers
>1       dfsadmin
>1       dfsmetrics.log
></CMD>
>
>Regards,
>Uma
>******************************************************************************************
>This email and its attachments contain confidential information from HUAWEI, which is
intended only for the person or entity whose address is listed above. Any use of the information
contained here in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this email in error, please notify the sender by phone or email immediately and delete it!
>*****************************************************************************************
>
>----- Original Message -----
>From: A Df <abbey_dragonforest@yahoo.com>
>Date: Wednesday, July 27, 2011 9:55 pm
>Subject: cygwin not connecting to Hadoop server
>To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
>
>> Hi All:
>> 
>> I am have Hadoop 0.20.2 and I am using cygwin on Windows 7. I 
>> modified the files as shown below for the Hadoop configuration.
>> 
>> conf/core-site.xml:
>> 
>> <configuration>
>>      <property>
>>          <name>fs.default.name</name>
>>          <value>hdfs://localhost:9100</value>
>>      </property>
>> </configuration>
>> 
>> 
>> conf/hdfs-site.xml:
>> 
>> <configuration>
>>      <property>
>>          <name>dfs.replication</name>
>>          <value>1</value>
>>      </property>
>> </configuration>
>> 
>> 
>> conf/mapred-site.xml:
>> 
>> <configuration>
>>      <property>
>>          <name>mapred.job.tracker</name>
>>          <value>localhost:9101</value>
>>      </property>
>> </configuration>
>> 
>> Then I have the PATH variable with
>> $PATH:/cygdrive/c/cygwin/bin:/cygdrive/c/cygwin/usr/bin
>> 
>> I added JAVA_HOME to the file in cygwin\home\Williams\hadoop-
>> 0.20.2\conf\hadoop-env.sh. 
>> My Java home is now at C:\Java\jdk1.6.0_26 so there is not space. I 
>> also turned off my firewall. 
>> However, I get the error from the command line:
>> 
>> <CODE>
>> Williams@TWilliams-LTPC ~
>> $ pwd
>> /home/Williams
>> 
>> Williams@TWilliams-LTPC ~
>> $ cd hadoop-0.20.2
>> 
>> Williams@TWilliams-LTPC ~/hadoop-0.20.2
>> $ bin/start-all.sh
>> starting namenode, logging to /home/Williams/hadoop-
>> 0.20.2/bin/../logs/hadoop-Wi
>> lliams-namenode-TWilliams-LTPC.out
>> localhost: starting datanode, logging to /home/Williams/hadoop-
>> 0.20.2/bin/../logs/hadoop-Williams-datanode-TWilliams-LTPC.out
>> localhost: starting secondarynamenode, logging to 
>> /home/Williams/hadoop-0.20.2/b
>> in/../logs/hadoop-Williams-secondarynamenode-TWilliams-LTPC.out
>> starting jobtracker, logging to /home/Williams/hadoop-
>> 0.20.2/bin/../logs/hadoop-
>> Williams-jobtracker-TWilliams-LTPC.out
>> localhost: starting tasktracker, logging to /home/Williams/hadoop-
>> 0.20.2/bin/../logs/hadoop-Williams-tasktracker-TWilliams-LTPC.out
>> 
>> Williams@TWilliams-LTPC ~/hadoop-0.20.2
>> $ bin/hadoop fs -put conf input
>> 11/07/27 17:11:28 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 0 time(s).
>> 11/07/27 17:11:30 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 1 time(s).
>> 11/07/27 17:11:32 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 2 time(s).
>> 11/07/27 17:11:34 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 3 time(s).
>> 11/07/27 17:11:36 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 4 time(s).
>> 11/07/27 17:11:38 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 5 time(s).
>> 11/07/27 17:11:40 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 6 time(s).
>> 11/07/27 17:11:43 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 7 time(s).
>> 11/07/27 17:11:45 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 8 time(s).
>> 11/07/27 17:11:47 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 9 time(s).
>> Bad connection to FS. command aborted.
>> 
>> Williams@TWilliams-LTPC ~/hadoop-0.20.2
>> $ bin/hadoop fs -put conf input
>> 11/07/27 17:17:29 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 0 time(s).
>> 11/07/27 17:17:31 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 1 time(s).
>> 11/07/27 17:17:33 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 2 time(s).
>> 11/07/27 17:17:35 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 3 time(s).
>> 11/07/27 17:17:37 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 4 time(s).
>> 11/07/27 17:17:39 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 5 time(s).
>> 11/07/27 17:17:41 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 6 time(s).
>> 11/07/27 17:17:44 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 7 time(s).
>> 11/07/27 17:17:46 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 8 time(s).
>> 11/07/27 17:17:48 INFO ipc.Client: Retrying connect to server: 
>> localhost/127.0.0.1:9100. Already tried 9 time(s).
>> Bad connection to FS. command aborted.
>> 
>> Williams@TWilliams-LTPC ~/hadoop-0.20.2
>> $ ping 127.0.0.1:9100
>> Ping request could not find host 127.0.0.1:9100. Please check the 
>> name and try a
>> gain.
>> </CODE>
>> 
>> I am not sure why the ip address seem to have localhost/127.0.0.1 
>> which seems to be repeating itself. The conf files are fine. I also 
>> know that when Hadoop is running there is a web interface to check 
>> but do the default ones work from cygwin which are:
>>     * NameNode - http://localhost:50070/
>>     * JobTracker - http://localhost:50030/
>> 
>> I wanted to give the cygwin a try once more before just switching 
>> to a cloudera hadoop vmware. I was hoping that it would not have so 
>> many problems just to get it working on Windows! Thanks again.
>> 
>> Cheers,
>> A Df
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message