hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Wenzel <mwen...@proheris.de>
Subject AW: WebUI's Server don't work and JobHistoryServer missing
Date Fri, 22 Jul 2016 11:52:33 GMT
Hi Sunil,

first of all thanks for helping me out. I found one reason why I had strange behaviors which
I couldn’t reproduce. I’m running two network interfaces on each node, and my /etc/hosts
only configured all my nodes for a single network interface. Interfaces are in different networks
and therefore no HTTP-Access for me. After cleaning this mess up, my problems seem to be sortable
in a clean order.

I know the yarn.resourcemanager.webapp.address property. Due to my network misconfiguration
a bunch of properties didn’t worked as expected.

You were right regarding JobHistoryServer / Proxyserver / Timelineserver. I read this guide
so often, that I obviously overlooked important parts. After starting my JobHistory Server
I had a bunch of new ways to track things on WebUIs. Proxy- and Timelineserver are two completely
new words for me. I going to research both servers.

I’m going to clean and sort my problems for the first now. After doing this, I believe that
I’ll be able to ask specifically for help on a single problem.
Regards, Mike.

Von: Sunil Govind [mailto:sunil.govind@gmail.com]
Gesendet: Donnerstag, 21. Juli 2016 15:05
An: Mike Wenzel <mwenzel@proheris.de>; user@hadoop.apache.org
Betreff: Re: WebUI's Server don't work and JobHistoryServer missing

Hi Mike

yarn.resourcemanager.webapp.address  is to configure Resource Manager Web UI and you can access
ResourceManager Web UI as "http://<host:port><http://%3chost:port%3e>". It will
run on 8088 as default if you are not configuring this property explicitly. Please refer [https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml]
for more detailed information.

I think you are not running JobHistoryServer. pls refer [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html]
 "Hadoop Startup" section. You can run history server by "[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh
--config $HADOOP_CONF_DIR start historyserver" command.
If you are interested, you can also run "timelineserver" which is for yarn.

Also you might need to run WebAppProxy server to see the MapReduce job application master
ui. Please refer to above link for cluster setup and it has detailed information to start
various services.

Thanks
Sunil


On Tue, Jul 19, 2016 at 1:59 PM Mike Wenzel <mwenzel@proheris.de<mailto:mwenzel@proheris.de>>
wrote:
My cluster looks like:

Node1 - NameNode + ResourceManager
Node2 - SecondaryNameNode
Node3 - DataNode (+NodeManager)
Node4 - DataNode (+NodeManager)
Node5 - DataNode (+NodeManager)

http://node1:8088/cluster works.

My problems:
> SecondaryNamenode WebUI: http://node2:50090 doesn’t work (ERR_EMPTY_RESPONSE).
> Datanode WebUI: http://node3:50075 looks like there is something wrong: https://i.imgur.com/dbTxtkS.png
> Also I’m missing the JobHistory Server. Isn’t the jobhistory server included in yarn?
I tried accessing some URLs I found on the web declared as “default configuration” and
always (ERR_EMPTY_RESPONSE).
> When I run a mapreduce job the output still says: “uri to track the job is httpd://localhost:8080”
and when I try to access any node on this port I get no data (ERR_EMPTY_RESPONSE).

Unfortunately I couldn’t get this problem solved. I only found a few guides setting yarn
up. I tried them all and my situation only got worse when adding more properties on yarn-site.xml.
E.g. the cluster overview-WebUI http://node1:8088/cluster didn’t worked anymore when adding
“yarn.resourcemanager.webapp.address” to yarn-site.xml.
Can anyone help me out configuring yarn correctly please?

My configuration:

core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:///hdfs/tmp</value<file:///\\hdfs\tmp%3c\value>>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://node1:54310</value>
  </property>
</configuration>

hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- NameNode configuration -->
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///hdfs/name</value<file:///\\hdfs\name%3c\value>>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>node1:50070</value>
  </property>
  <property>
    <name>dfs.namenode.checkpoint.dir</name>
    <value>file:///hdfs/checkpoint</value<file:///\\hdfs\checkpoint%3c\value>>
  </property>

<!-- SecondaryNameNode configuration -->
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>node2:50090</value>
  </property>

<!-- DataNode configuration -->
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///hdfs/data</value<file:///\\hdfs\data%3c\value>>
  </property>
</configuration>

yarn-site.xml
<?xml version="1.0"?>
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- RessourceManager configuration -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>node1:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>node1:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>node1:8032</value>
  </property>
</configuration>

masters
node2

slaves
node3
node4
node5
Mime
View raw message