hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abhiTowson cal <abhishek.dod...@gmail.com>
Subject Re:
Date Sun, 29 Jul 2012 19:02:59 GMT
Hi,

Thanks for reply Harsh, These are my configuration properties

//mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-master-2:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-master-2:19888</value>
</property>
</configuration>


//Yarn-site-xml

<configuration>
  <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $YARN_HOME/*,$YARN_HOME/lib/*
     </value>
  </property>
 <property>
    <name>yarn.resourcemanager.address</name>
      <value>hadoop-master-2:8040</value>
      </property>
       <property>
       <name>yarn.resourcemanager.scheduler.address</name>
       <value>hadoop-master-2:8030</value>
       </property>
       <property>
       <name>yarn.resourcemanager.admin.address</name>
       <value>hadoop-master-2:8141</value>
       </property>
       <property>
       <name>yarn.resourcemanager.webapp.address</name>
       <value>hadoop-master-2:8088</value>
       </property>
       <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce.shuffle</value>
       </property>
 <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
       </property>
       <property>
       <name>yarn.nodemanager.local-dirs</name>
       <value>/srv/storage/yarn/local</value>
       </property>
       <property>
       <name>yarn.nodemanager.log-dirs</name>
       <value>/srv/storage/yarn/log</value>
       </property>
</configuration>


On Sun, Jul 29, 2012 at 2:30 PM, abhiTowson cal
<abhishek.dodda1@gmail.com> wrote:
> Hi All,
>
> I am getting problem that job is running in localrunner rather than
> the cluster enviormnent.
> And when am running the job i would not be able to see the job id in
> the resource manager UI
>
> Can you please go through the issues and let me know ASAP.
>
> sudo -u hdfs hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> 1000000 /benchmark/teragen/input
> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> Generating 1000000 using 1 maps with step of 1000000
> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapred.FileOutputCommitter
> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
> 12/07/29 13:35:59 INFO mapred.Task:  Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> name and  BYTES_READ as counter name instead
> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
> 12/07/29 13:36:00 INFO mapred.JobClient:  map 0% reduce 0%
> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
> is done. And is in the process of commiting
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
> is allowed to commit now
> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
> task 'attempt_local_0001_m_000000_0' to
> hdfs://hadoop-master-1/benchmark/teragen/input
> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
> 12/07/29 13:36:02 INFO mapred.JobClient:  map 100% reduce 0%
> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
> 12/07/29 13:36:02 INFO mapred.JobClient:   File System Counters
> 12/07/29 13:36:02 INFO mapred.JobClient:     FILE: Number of bytes read=142686
> 12/07/29 13:36:02 INFO mapred.JobClient:     FILE: Number of bytes
> written=220956
> 12/07/29 13:36:02 INFO mapred.JobClient:     FILE: Number of read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     FILE: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     FILE: Number of write operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     HDFS: Number of bytes read=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     HDFS: Number of bytes
> written=100000000
> 12/07/29 13:36:02 INFO mapred.JobClient:     HDFS: Number of read operations=1
> 12/07/29 13:36:02 INFO mapred.JobClient:     HDFS: Number of large
> read operations=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     HDFS: Number of write operations=2
> 12/07/29 13:36:02 INFO mapred.JobClient:   Map-Reduce Framework
> 12/07/29 13:36:02 INFO mapred.JobClient:     Map input records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient:     Map output records=1000000
> 12/07/29 13:36:02 INFO mapred.JobClient:     Input split bytes=82
> 12/07/29 13:36:02 INFO mapred.JobClient:     Spilled Records=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     CPU time spent (ms)=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
> 12/07/29 13:36:02 INFO mapred.JobClient:     Total committed heap
> usage (bytes)=124715008
> 12/07/29 13:36:02 INFO mapred.JobClient:
> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>
> Regards
> Abhishek

Mime
View raw message