hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeonmailinglist-gmail <xeonmailingl...@gmail.com>
Subject Can't find map or reduce logs when a job ends.
Date Fri, 13 Mar 2015 18:06:56 GMT
Hi,

With this configuration in mapreduce (see [1] and [2]), I can’t see the 
map and reduce logs of the job when it ends. When I try to look to the 
history, I get this error |Not Found: job_1426267326549_0005|. But if I 
list the log dir in the hdfs (see [3]), I have some logs about the job, 
but not logs about the map or reduce tasks.

Why I can’t see the map and reduce logs? Am I missing some configuration?

[1] configuration in mapred-site.xml

|<configuration>
      <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
     </property>
      <property>
<name>mapreduce.jobhistory.address </name>
<value>hadoop-coc-1:10020</value>
     </property>

      <property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-coc-1:19888</value>
     </property>
<property>
<name>mapreduce.jobhistory.max-age-ms</name>
<value>1800000</value>
</property>
</configuration>
|

[2] configuration in yarn-site.xml

|xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>

<!-- Site specific YARN configuration properties -->
         <property>
              <name>yarn.nodemanager.aux-services</name>
              <value>mapreduce_shuffle</value>
         </property>
         <property>
             <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
             <value>org.apache.hadoop.mapred.ShuffleHandler</value>
         </property>
   <property>
     <name>yarn.log-aggregation-enable</name>
     <value>true</value>
   </property>
   <property>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
   </property>
|

[3] List logs in hadoop

|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so
which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link
it with '-z noexecstack'.
15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
Found 4 items
drwxrwx---   - xeon supergroup          0 2015-03-13 13:35 /app-logs/xeon/logs/application_1426267324367_0002
drwxrwx---   - xeon supergroup          0 2015-03-13 13:37 /app-logs/xeon/logs/application_1426267324367_0003
drwxrwx---   - xeon supergroup          0 2015-03-13 13:44 /app-logs/xeon/logs/application_1426267324367_0004
drwxrwx---   - xeon supergroup          0 2015-03-13 13:47 /app-logs/xeon/logs/application_1426267324367_0005
|

​

-- 
--


Mime
View raw message