hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Saxena <vsaxena.va...@gmail.com>
Subject Re: Yarn ResourceManager web UI does not show job
Date Mon, 21 Sep 2015 20:35:36 GMT
MR jobs will write history files to path given by config
mapreduce.jobhistory.intermediate-done-dir
History server will then move them to done dir which is given by config m
apreduce.jobhistory.done-dir.

By default these config values
are ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
and ${yarn.app.mapreduce.am.staging-dir}/history/done respectively.

7 days is also configurable(config being mapreduce.jobhistory.max-age-ms).
You can set this value according to your cluster.

I hope this answers your question.

Regards,
Varun Saxena.

On Tue, Sep 22, 2015 at 1:39 AM, Boyu Zhang <boyuzhang35@gmail.com> wrote:

> Thanks a lot for the clarification!
>
> I tried to find the log and history information about finished jobs. But
> they are not in hfs://xxx/user/myusername/output/_SUCCESS (0B). Can you
> please give some pointers on where the statistical/job history files are
> located? The hfs://xxxx/history/done only stores history files up to 7 days.
>
> Thanks,
> Boyu
>
> On Mon, Sep 21, 2015 at 1:23 PM, Varun Saxena <vsaxena.varun@gmail.com>
> wrote:
>
>> No, you cant show them in RM UI then.
>>
>> However if you can start another daemon, you can consider using YARN
>> Application History/Timeline Server or MR Job History Server(only for MR
>> jobs)  to see information about completed jobs.
>> You can look up Hadoop documentation to learn more about them and how to
>> configure them.
>>
>> Just to clarify though, the apps themselves are not lost, as in, the
>> output is not lost. Its just the information about them which is no longer
>> present on RM restart.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Mon, Sep 21, 2015 at 10:31 PM, Boyu Zhang <boyuzhang35@gmail.com>
>> wrote:
>>
>>> Thanks for the answer Varun.
>>>
>>> It is the case that yarn.resourcemanager.recovery.enabled is set to be
>>> false. Is there a way to show the jobs that are submitted before the
>>> restart? We don't want to lose that data.
>>>
>>> Thanks,
>>> Boyu
>>>
>>>
>>> On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena <vsaxena.varun@gmail.com>
>>> wrote:
>>>
>>>> Hi Boyu,
>>>>
>>>> RM stores apps in state store if recovery is enabled. Only then they
>>>> will be available on restart.
>>>> Otherwise they are kept in memory and hence lost on restart.
>>>>
>>>> You may not have it enabled. Check config value for below config. By
>>>> default its false.
>>>> yarn.resourcemanager.recovery.enabled
>>>>
>>>> Regards,
>>>> Varun.
>>>>
>>>> On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang <boyuzhang35@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Everyone,
>>>>>
>>>>> I have a strange error regarding the ResourceManager web UI (
>>>>> http://xx.xx:8088).
>>>>>
>>>>> Someone before me set up the hadoop + yarn cluster using Pivotal HD,
>>>>> it was running fine. Then today, the resource manager and node manager
>>>>> disappeared, the logs did not record this. I restarted them, they are
up
>>>>> and running, but the resource manger web UI does not show any jobs. We
have
>>>>> 700+ jobs in the past, and they were showing before.
>>>>>
>>>>> If I submit MapReduce jobs, the new submitted ones show up. But the
>>>>> disappear again after restart the resource manger and node manager.
>>>>>
>>>>> Can anyone give any hint on where to look?
>>>>>
>>>>> Thanks in advance,
>>>>> Boyu
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message