hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SF Hadoop <sfhad...@gmail.com>
Subject Re: Not information in Job History UI
Date Tue, 04 Mar 2014 00:29:42 GMT
Thanks for that info Jian.

You said, "there are no job logs generated on the server that is running
the job.".  So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?

I am quite confused as to where the logs for each specific part of the
ecosystem reside.

If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?

Thanks for your help.


On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jhe@hortonworks.com> wrote:

> Note that node manager will not keep the finished applications and only
> show running apps,  so the UI won't show the finished apps.
>  Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished.  you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sfhadoop@gmail.com> wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI.  I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir.  I get output in regards to the daemons
>> but very little in regards to the job.  All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message