hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Schmidtke <ro.schmid...@gmail.com>
Subject Re: JobHistory location/configuration
Date Wed, 20 Jan 2016 10:51:41 GMT
Thanks for the hint. I'm currently doing tests which do not involve
long-time deployments. For now everything works as I'd like it to work --
thanks!

On Wed, Jan 20, 2016 at 11:37 AM, Alexander Alten-Lorenz <
wget.null@gmail.com> wrote:

> Robert,
>
> you should also take a look at:
>
> mapreduce.jobhistory.max-age-ms
>
> configures how long the files are held (default one week). And you’re
> right, just wait the configured time to see the logs.
>
>
>
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro.schmidtke@gmail.com>
> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
> that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com
> > wrote:
>
>> Okay so I assumed I could specify paths on my local filesystem, which is
>> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
>> and done_intermediate folders. However only the intermediate folder
>> contains files (even after the job finished). Is there any amount of time I
>> have to wait before they're moved to the done directory? Thanks!
>>
>> Robert
>>
>> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>>> of my jobs after they're finished. My setup does not permit me to view the
>>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>>> <dir>). I have configured:
>>>
>>> mapreduce.jobhistory.address,
>>> mapreduce.jobhistory.webapp.address,
>>> mapreduce.jobhistory.intermediate-done-dir, and
>>> mapreduce.jobhistory.done-dir,
>>>
>>> but there are no history files to be found. Am I missing something here?
>>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>>> bunch in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>


-- 
My GPG Key ID: 336E2680

Mime
View raw message