hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyu Zhang <boyuzhan...@gmail.com>
Subject Re: Job History files location of 2.0.4
Date Mon, 10 Jun 2013 19:40:32 GMT
I solved the problem, in my case, it is because I did not start the job
history server daemon. After starting it, the history logs are generated in
2 places: the HADOOP_MAPRED_LOG_DIR directory, and in hdfs, /tmp folder.

Hope this is helpful for whoever also has the same problem. By the way, the
documentation for hadoop 2.0.x is out of date, this creates some trouble in
figuring out where the conf/logs are stored and where to set them.

Thanks,
Boyu


On Fri, Jun 7, 2013 at 2:39 PM, Boyu Zhang <boyuzhang35@gmail.com> wrote:

> I used a directory that is local to every slave node: export
> HADOOP_LOG_DIR="/scratch/$USER/$PBS_JOBID/hadoop-$USER/log".
>
> I did not change the " hadoop.job.history.user.location", I thought if I
> don't change this property, the job history is going to be stored in hdfs
> under output/_logs dir.
>
> Then after the job completes, I copied back the logs to the server.
>
> Thanks a lot,
> Boyu
>
>
> On Fri, Jun 7, 2013 at 2:32 PM, Shahab Yunus <shahab.yunus@gmail.com>wrote:
>
>> What value do you have for hadoop.log.dir property?
>>
>>
>> On Fri, Jun 7, 2013 at 5:20 PM, Boyu Zhang <boyuzhang35@gmail.com> wrote:
>>
>>> Hi Shahab,
>>>
>>>
>>> How old were they?
>>>>
>>> They are new, I did the copy automatically right after the job
>>> completed, in a script.
>>>
>>> I am assuming they were from the jobs run on the older version, right?
>>>>
>>> I run the job using the hadoop version 2.0.4 if this is what you mean.
>>>
>>> Or are you looking for new jobs's log that you are running after the
>>>> upgrade?
>>>>
>>> I am looking for new job's log, I did a fresh install of hadoop2.0.4,
>>> then run the job, then copied back the entire hdfs directory.
>>>
>>>
>>>> What about the local file systems? Are the logs there still?
>>>>
>>> Where should I find the logs (job logs, not daemon logs) in the local
>>> file systems?
>>>
>>> Thanks,
>>> Boyu
>>>
>>>
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>>
>>>> On Fri, Jun 7, 2013 at 4:56 PM, Boyu Zhang <boyuzhang35@gmail.com>wrote:
>>>>
>>>>> Thanks Shahab,
>>>>>
>>>>> I saw the link, but it is not the case for me. I copied everything
>>>>> from hdfs ($HADOOP_HOME/bin/hdfs dfs -copyToLocal / $local_dir). But
did
>>>>> not see the logs.
>>>>>
>>>>> Did it work for you?
>>>>>
>>>>> Thanks,
>>>>> Boyu
>>>>>
>>>>>
>>>>> On Fri, Jun 7, 2013 at 1:52 PM, Shahab Yunus <shahab.yunus@gmail.com>wrote:
>>>>>
>>>>>> See this;
>>>>>>
>>>>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201302.mbox/%3C1360184802.61630.YahooMailNeo@web141205.mail.bf1.yahoo.com%3E
>>>>>>
>>>>>> Regards,
>>>>>> Shahab
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 7, 2013 at 4:33 PM, Boyu Zhang <boyuzhang35@gmail.com>wrote:
>>>>>>
>>>>>>> Dear All,
>>>>>>>
>>>>>>> I recently moved from Hadoop0.20.2 to 2.0.4, and I am trying
to find
>>>>>>> the old job history files (used to be in hdfs, output/_logs/history),
it
>>>>>>> records detailed time information for every task attempts.
>>>>>>>
>>>>>>> But now it is not on hdfs anymore, I copied the entire "/" from
hdfs
>>>>>>> to my local dir, but am not able to find this location.
>>>>>>>
>>>>>>> Could anyone give any advise on where are the files?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Boyu
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message