hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: JobHistory location/configuration
Date Wed, 20 Jan 2016 10:37:30 GMT
Robert,

you should also take a look at:

mapreduce.jobhistory.max-age-ms

configures how long the files are held (default one week). And you’re right, just wait the
configured time to see the logs.


> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro.schmidtke@gmail.com> wrote:
> 
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly that.
> 
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com <mailto:ro.schmidtke@gmail.com>>
wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is not the case.
After doing a hadoop fs -ls -R on my HDFS I found the done and done_intermediate folders.
However only the intermediate folder contains files (even after the job finished). Is there
any amount of time I have to wait before they're moved to the done directory? Thanks!
> 
> Robert
> 
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com <mailto:ro.schmidtke@gmail.com>>
wrote:
> Hi everyone,
> 
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of my jobs after
they're finished. My setup does not permit me to view the history server's UI, so I'm bound
to the CLI (bin/hadoop job -history all <dir>). I have configured:
> 
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
> 
> but there are no history files to be found. Am I missing something here? How can I get
a hold of the files, or make MR produce them at all? Thanks a bunch in advance!
> 
> Robert
> 
> 
> 
> -- 
> My GPG Key ID: 336E2680
> 
> 
> 
> -- 
> My GPG Key ID: 336E2680
> 
> 
> 
> -- 
> My GPG Key ID: 336E2680


Mime
View raw message