apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From userguy <rohit.gar...@gmail.com>
Subject Re: Hdfs + apex-core
Date Mon, 15 May 2017 10:50:47 GMT
Hello , 

Are you launching an application using apex-cli?
Ans - Yes 

2) If you are then is it same machine as before which you used WordCount
example to launch from? Is it an example from Apache Malhar ? or Hadoop
example?

Ans - WordCount Example is from Hadoop Test Jar - ( I means Hadoop Map
reduce Job for WordCount works Properly on Same Cluster ) 

3) Is your yarn-site.xml is same on all the nodes?
yes 

4) If possible can you provide yarn-site.xml from all nodes in the email
thread?
Sharing

-----------YARN SITE----------

-->
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
       <name>yarn.resourcemanager.address</name>
          <value>10.0.0.0:8032</value>
     </property>
      <property>
     <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>10.0.0.0:8031</value>
    </property>
      <property>
      <name>yarn.resourcemanager.scheduler.address</name>
       <value>10.0.0.0:8030</value>
      </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///var/data/yarn/local</value>
    </property>

    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>file:///var/data/yarn/logs</value>
    </property>

    <property>
        <name>yarn.log.aggregation-enable</name>
        <value>true</value>
    </property>

    <property>
        <description>Where to aggregate logs</description>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>hdfs://10.0.0.0:9000/var/log/hadoop_yarn/apps</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>40960</value>
    </property>

    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
    </property>

    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>40960</value>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>True</value>
    </property>

    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>

  <property>
        <name>yarn.application.classpath</name>
        <value>
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/etc/hadoop/*,
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/common/*,
           
/var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/common/lib/*,
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/hdfs/*,
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/hdfs/lib/*,
           
/var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/mapreduce/*,
           
/var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/mapreduce/lib/*,
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/yarn/*,
            /var/data/hadoop/hadoop-2.6.0-cdh5.11.0/share/hadoop/yarn/lib/*
        </value>
    </property>

</configuration>

It is excatly same for all the servers 




--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Fwd-Hdfs-apex-core-tp1608p1614.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Mime
View raw message