hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ranadip Chatterjee <ranadi...@gmail.com>
Subject Re: Can't find map or reduce logs when a job ends.
Date Tue, 17 Mar 2015 02:53:28 GMT
Is the job history server up and running on the right host and port? Please
check the job history server logs, if so? A common reason is for the owner
of job history server to not have read permission on the logs or for the
map reduce process owners to not have write permission in the job history
log location. The error should show up in the job history server logs.

Ranadip
On 13 Mar 2015 18:07, "xeonmailinglist-gmail" <xeonmailinglist@gmail.com>
wrote:

>  Hi,
>
> With this configuration in mapreduce (see [1] and [2]), I can’t see the
> map and reduce logs of the job when it ends. When I try to look to the
> history, I get this error Not Found: job_1426267326549_0005. But if I
> list the log dir in the hdfs (see [3]), I have some logs about the job, but
> not logs about the map or reduce tasks.
>
> Why I can’t see the map and reduce logs? Am I missing some configuration?
>
> [1] configuration in mapred-site.xml
>
> <configuration>
>      <property>
>         <name>mapreduce.framework.name</name>
>         <value>yarn</value>
>     </property>
>      <property>
> <name>mapreduce.jobhistory.address </name>
> <value>hadoop-coc-1:10020</value>
>     </property>
>
>      <property>
> <name>mapreduce.jobhistory.webapp.address</name>
> <value>hadoop-coc-1:19888</value>
>     </property>
> <property>
> <name>mapreduce.jobhistory.max-age-ms</name>
> <value>1800000</value>
> </property>
> </configuration>
>
> [2] configuration in yarn-site.xml
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
> <?xml version="1.0"?>
> <configuration>
>
> <!-- Site specific YARN configuration properties -->
>         <property>
>              <name>yarn.nodemanager.aux-services</name>
>              <value>mapreduce_shuffle</value>
>         </property>
>         <property>
>             <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>             <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>         </property>
>   <property>
>     <name>yarn.log-aggregation-enable</name>
>     <value>true</value>
>   </property>
>   <property>
>     <name>yarn.nodemanager.remote-app-log-dir</name>
>     <value>/app-logs</value>
>   </property>
>
> [3] List logs in hadoop
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so
which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>',
or link it with '-z noexecstack'.
> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> Found 4 items
> drwxrwx---   - xeon supergroup          0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
> drwxrwx---   - xeon supergroup          0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
> drwxrwx---   - xeon supergroup          0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
> drwxrwx---   - xeon supergroup          0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
>
> ​
>
> --
> --
>
>

Mime
View raw message