hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: history server for 2 clusters
Date Fri, 21 Feb 2014 05:20:49 GMT
Interesting use-case and setup. We never had this use-case in mind so far - we so far assumed
a history-server per YARN cluster. You may be running into some issues where this assumption
is not valid.

Why do you need two separate YARN clusters for the same underlying data on HDFS? And if that
can't change, why can't you have two history-servers?

+Vinod

On Feb 20, 2014, at 6:08 PM, Anfernee Xu <anfernee.xu@gmail.com> wrote:

> Hi,
> 
> I'm at 2.2.0 release and I have a HDFS cluster which is shared by 2 YARN(MR) cluster,
also I have a single shared history server, what I'm seeing is I can see all job summary for
all jobs from history server UI, I also can see task log for jobs running in one cluster,
but if I want to see log for jobs running in another cluster, it showed me below error
> 
> Logs not available for attempt_1392933787561_0024_m_000000_0. Aggregation may not be
complete, Check back later or try the nodemanager at slc03jvt.mydomain.com:31303 
> 
> Here's my configuration:
> 
> Note: my history server is running on RM node of the MR cluster where I can see the log.
> 
> 
> --------mapred-site.xml
> <property>
>   <name>mapreduce.jobhistory.address</name>
>   <value>slc00dgd:10020</value>
>   <description>MapReduce JobHistory Server IPC host:port</description>
> </property>
> 
> <property>
>   <name>mapreduce.jobhistory.webapp.address</name>
>   <value>slc00dgd:19888</value>
>   <description>MapReduce JobHistory Server Web UI host:port</description>
> </property>
> 
> ----------yarn-site.xml
>   <property>
>      <name>yarn.log-aggregation-enable</name>
>      <value>true</value>
>    </property>
> 
>    <property>
>      <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
>      <value>dc</value>
>    </property>
> 
> Above configuration are almost same for both clusters, the only difference is "yarn.nodemanager.remote-app-log-dir-suffix",
they have different suffix.
> 
> 
> 
> -- 
> --Anfernee


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message