hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SF Hadoop <sfhad...@gmail.com>
Subject Re: Not information in Job History UI
Date Tue, 04 Mar 2014 21:34:53 GMT
That explains a lot.  Thanks for the information.  I appreciate your help.


On Mon, Mar 3, 2014 at 7:47 PM, Jian He <jhe@hortonworks.com> wrote:

> > You said, "there are no job logs generated on the server that is
> running the job.".
> that was quoting your previous sentence and answer your question..
>
> > If I were to run a job and I wanted to tail the job log as it was
> running, where would I find that log?
> 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
> look for logs in local dirs specified by yarn.nodemanager.log-dirs.
> Or
> 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
> to aggregate those NM local logs and upload them to HDFS once application
> is finished.Then you can use yarn logs command  or simply go the history UI
> to see the logs.
> You can find good explanation from
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
>
> Thanks.
>
>
> On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sfhadoop@gmail.com> wrote:
>
>> Thanks for that info Jian.
>>
>> You said, "there are no job logs generated on the server that is running
>> the job.".  So am I correct in assuming the logs will be in the dir
>> specified by yarn.nodemanager.log-dirs on the datanodes?
>>
>> I am quite confused as to where the logs for each specific part of the
>> ecosystem reside.
>>
>> If I were to run a job and I wanted to tail the job log as it was
>> running, where would I find that log?
>>
>> Thanks for your help.
>>
>>
>>  On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jhe@hortonworks.com> wrote:
>>
>>>  Note that node manager will not keep the finished applications and
>>> only show running apps,  so the UI won't show the finished apps.
>>>  Conversely, job history server UI will only show the finished apps but
>>> not the running apps.
>>>
>>> bq. there are no job logs generated on the server that is running the
>>> job.
>>> by default, the local logs will be deleted after job finished.  you can
>>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion
>>> of the logs.
>>>
>>> Jian
>>>
>>>
>>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sfhadoop@gmail.com> wrote:
>>>
>>>> Hadoop 2.2.0
>>>> CentOS 6.4
>>>> Viewing UI in various browsers.
>>>>
>>>> I am having a problem where no information is visible in my Job History
>>>> UI.  I run test jobs, they complete without error, but no information ever
>>>> populates the nodemanager or jobhistory server UI.
>>>>
>>>> Also, there are no job logs generated on the server that is running the
>>>> job.
>>>>
>>>> I have the following settings configured:
>>>> yarn.nodemanager.local-dirs
>>>> yarn.nodemanager.log-dirs
>>>> yarn.log.server.url
>>>>
>>>> ...plus the basic yarn log dir.  I get output in regards to the daemons
>>>> but very little in regards to the job.  All I get that refers to the
>>>> jobhistory server is the following (so it appears to be functioning
>>>> properly):
>>>>
>>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>> bound to port 19888
>>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>>> SelectChannelConnector@server:19888
>>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /jobhistory started at 19888
>>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>>
>>>> I have a feeling this is a misconfiguration but I cannot figure out
>>>> what setting is missing or wrong.
>>>>
>>>> Other than not being able to see any of the jobs in the UIs, everything
>>>> appears to be working correctly so this is quite confusing.
>>>>
>>>> Any help is appreciated.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message