hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Margusja <mar...@roo.ee>
Subject Re: ApplicationMaster link on cluster web page does not work
Date Thu, 28 Aug 2014 18:39:48 GMT
More information

after I started resourcemanager
[root@vm38 ~]# /etc/init.d/hadoop-yarn-resourcemanager start
Starting Hadoop resourcemanager:                           [  OK ]

and I open cluster web interface there is some tcp connections to 8088:

[root@vm38 ~]# netstat -np | grep 8088
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:61120    ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:64412    ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:50139    ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:64407    ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:58817    ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:84.50.21.39:52250    ESTABLISHED 15723/java

this is ok

Now I started new map reduce job and got tracking url 
http://server:8088/proxy/application_1409250808355_0001/

now there are loads of connections and tracking url does not response:
tcp        0      0 ::ffff:90.190.106.33:46910 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:35984 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:37559  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:44154 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:45417 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:57294 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:47949  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55330 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp      432      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:45467  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:58405 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55580 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:51992  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:44578 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:38686  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:39242 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55916 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:48064 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:35148 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:42638 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:50836 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:44789  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55051 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:39207  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:53781 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:54180 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:56344 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:52707 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:52274 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:45417  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:46627  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:44583  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:40928 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:47014 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:60939 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:52274  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:38391 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:47321  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:36186  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:39160 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:54352 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:59425  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:46627 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:57140  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:48788  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:37559 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:55052  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:47104  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:47515 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:45655  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:48064  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:45467 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:47343  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:42678 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:33116  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:40077 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:49100 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:44547  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:40050  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:43073 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:38686 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:41480 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:56344  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:42933 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:45400  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:33023 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:42678  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:43331 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:48524  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:51789 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:44447  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:35984  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:50472 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55326 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:44356 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:48788 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:44560  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:38690  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:57434  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:45282 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:38284 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:44411 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:55712 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:48929 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:49288 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:42505 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:38391  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:54537  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:43331  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:47141  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:60823 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:55712  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:42313  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:45879  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:33412 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:51992 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:37513  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:39260 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:8088 
::ffff:90.190.106.33:46269  ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:46087 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
tcp        0      0 ::ffff:90.190.106.33:47321 
::ffff:90.190.106.33:8088   ESTABLISHED 15723/java
...
...
...

wtf?

Best regards, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
http://ee.linkedin.com/in/margusroo
skype: margusja
ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)"

On 28/08/14 20:40, Margusja wrote:
> Hi
>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
> <property>
>   <name>yarn.app.mapreduce.am.staging-dir</name>
>   <value>/user</value>
> </property>
>
> <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>mapreduce_shuffle</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
> <property>
>         <name>yarn.application.classpath</name>
>         <value>
> /etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*,/home/hduser/mahout-1.0-snapshot/math/target/*

>
>         </value>
> </property>
>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
>         <value>0.0.0.0:8030</value>
> </property>
>
> <property>
>         <name>yarn.web-proxy.address</name>
>         <value>vm38.dbweb.ee:8089</value>
> </property>
>
> </configuration>
>
> when i click job's ApplicationMaster link 
> (http://vm38.dbweb.ee:8089/proxy/application_1409246753441_0001/) on 
> claster all applications list I will get:
>
>
>    HTTP ERROR 404
>
> Problem accessing /proxy/application_1409246753441_0001/mapreduce. 
> Reason:
>
>     NOT_FOUND
>
> ------------------------------------------------------------------------
> /Powered by Jetty://
>
> /I use yarn.web-proxy because without yarn.web-proxy when I click 
> ApplicationMaster link I will have loads of tcp connections with 
> established statuses.
> In yarn-proxy log there is nothing interesting
>


Mime
View raw message