hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyu Zhang <boyuzhan...@gmail.com>
Subject Re: Job History files location of 2.0.4
Date Fri, 07 Jun 2013 21:39:54 GMT
I used a directory that is local to every slave node: export
HADOOP_LOG_DIR="/scratch/$USER/$PBS_JOBID/hadoop-$USER/log".

I did not change the " hadoop.job.history.user.location", I thought if I
don't change this property, the job history is going to be stored in hdfs
under output/_logs dir.

Then after the job completes, I copied back the logs to the server.

Thanks a lot,
Boyu


On Fri, Jun 7, 2013 at 2:32 PM, Shahab Yunus <shahab.yunus@gmail.com> wrote:

> What value do you have for hadoop.log.dir property?
>
>
> On Fri, Jun 7, 2013 at 5:20 PM, Boyu Zhang <boyuzhang35@gmail.com> wrote:
>
>> Hi Shahab,
>>
>>
>> How old were they?
>>>
>> They are new, I did the copy automatically right after the job completed,
>> in a script.
>>
>> I am assuming they were from the jobs run on the older version, right?
>>>
>> I run the job using the hadoop version 2.0.4 if this is what you mean.
>>
>> Or are you looking for new jobs's log that you are running after the
>>> upgrade?
>>>
>> I am looking for new job's log, I did a fresh install of hadoop2.0.4,
>> then run the job, then copied back the entire hdfs directory.
>>
>>
>>> What about the local file systems? Are the logs there still?
>>>
>> Where should I find the logs (job logs, not daemon logs) in the local
>> file systems?
>>
>> Thanks,
>> Boyu
>>
>>
>>>
>>> Regards,
>>> Shahab
>>>
>>>
>>> On Fri, Jun 7, 2013 at 4:56 PM, Boyu Zhang <boyuzhang35@gmail.com>wrote:
>>>
>>>> Thanks Shahab,
>>>>
>>>> I saw the link, but it is not the case for me. I copied everything from
>>>> hdfs ($HADOOP_HOME/bin/hdfs dfs -copyToLocal / $local_dir). But did not see
>>>> the logs.
>>>>
>>>> Did it work for you?
>>>>
>>>> Thanks,
>>>> Boyu
>>>>
>>>>
>>>> On Fri, Jun 7, 2013 at 1:52 PM, Shahab Yunus <shahab.yunus@gmail.com>wrote:
>>>>
>>>>> See this;
>>>>>
>>>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201302.mbox/%3C1360184802.61630.YahooMailNeo@web141205.mail.bf1.yahoo.com%3E
>>>>>
>>>>> Regards,
>>>>> Shahab
>>>>>
>>>>>
>>>>> On Fri, Jun 7, 2013 at 4:33 PM, Boyu Zhang <boyuzhang35@gmail.com>wrote:
>>>>>
>>>>>> Dear All,
>>>>>>
>>>>>> I recently moved from Hadoop0.20.2 to 2.0.4, and I am trying to find
>>>>>> the old job history files (used to be in hdfs, output/_logs/history),
it
>>>>>> records detailed time information for every task attempts.
>>>>>>
>>>>>> But now it is not on hdfs anymore, I copied the entire "/" from hdfs
>>>>>> to my local dir, but am not able to find this location.
>>>>>>
>>>>>> Could anyone give any advise on where are the files?
>>>>>>
>>>>>> Thanks,
>>>>>> Boyu
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message