giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Boyd <db...@data-tactics-corp.com>
Subject Re: Getting Giraph to build/run with CDH4 (cannot initialize cluster - check mapreduce.framework.name)
Date Mon, 25 Feb 2013 23:51:56 GMT
Sandy:
    Thanks that helps a great deal.  I am now at least getting to the 
point that the jobs show up in the job tracker.   However, they all
fail on initialization with the good old:
> java.io.FileNotFoundException: File /tmp/hadoop-mapred/mapred/staging/hdfs/.staging/job_201302211213_0055/job.jar
does not exist
This tells me that maven is either not specifying that the giraph-core 
jar file should be used as the job jar or I am missing
something else in the set up.

Attached is the job.xml file from one of the failed jobs and below is 
the relevant profile out of my pom.xml.
I did upgrade to CDH4.1.3 just to see if that would help.
Also, I have been running all sorts of jobs (benchmarks, and other 
tests) against this cluster for some time so I know that the cluster
works well.

Again, any help is appreciated.

Relevant section of pom.xml:
>     <profile>
>       <id>hadoop_cdh4.1.3mr1</id>
>       <properties>
> <hadoopmr1.version>2.0.0-mr1-cdh4.1.3</hadoopmr1.version>
> <hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
> <munge.symbols>HADOOP_1_SECURITY,HADOOP_1_SECRET_MANAGER</munge.symbols>
>       </properties>
>       <dependencies>
>         <!-- sorted lexicographically -->
>         <dependency>
>           <groupId>commons-net</groupId>
>           <artifactId>commons-net</artifactId>
>         </dependency>
>         <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-client</artifactId>
>           <version>${hadoopmr1.version}</version>
>           <scope>provided</scope>
>         </dependency>
>         <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-common</artifactId>
>           <version>${hadoop.version}</version>
>           <scope>provided</scope>
>         </dependency>
>         <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-hdfs</artifactId>
>           <version>${hadoop.version}</version>
>           <scope>provided</scope>
>         </dependency>
>         <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-test</artifactId>
>           <version>${hadoopmr1.version}</version>
>           <scope>provided</scope>
>         </dependency>
>       </dependencies>
>     </profile>



On 2/25/2013 12:47 PM, Sandy Ryza wrote:
> Hi David,
>
> Moving this to cdh-user, as it is CDH-specific.
>
> CDH4 comes with two versions of mapreduce, MR1, and MR2.  It sounds 
> like you are building against MR2 
> (http://blog.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/). 
>  Do you know whether your cluster runs MR2/YARN or MR1?  If it runs, 
> MR2, you can set mapreduce.framework.name 
> <http://mapreduce.framework.name> to "yarn".  If it runs MR1, you can 
> build against the MR1 jar by setting the version of your hadoop-client 
> to 2.0.0-mr1-cdh4.1.1. 
> (https://ccp.cloudera.com/display/CDH4DOC/Managing+Hadoop+API+Dependencies+in+CDH4)
> Does that help?
>
> -Sandy
>
>
> On Mon, Feb 25, 2013 at 8:26 AM, David Boyd 
> <dboyd@data-tactics-corp.com <mailto:dboyd@data-tactics-corp.com>> wrote:
>
>     All:
>        I am trying to get the Giraph 0.2 snapshot (pulled via GIT on
>     Friday)
>     to build and run with CDH4.
>
>     I modified the pom.xml to provide a profile for my specific
>     version (4.1.1).
>     The build works (mvn -Phadoop_cdh4.1.1 clean package test) and passes
>     all the tests.
>
>     If I try to do the next step and submit to my cluster with the
>     command:
>     mvn -Phadoop_cdh4.1.1 test
>     -Dprop.mapred.job.tracker=10.1.94.53:8021 <http://10.1.94.53:8021>
>     -Dgiraph.zkList=10.1.94.104:2181 <http://10.1.94.104:2181>
>
>      the JSON test in core fails.  If I move that test out of the way
>     a whole bunch of tests in examples
>     fail.  They all fail with:
>
>         java.io.IOException: Cannot initialize Cluster. Please check your
>         configuration for mapreduce.framework.name
>         <http://mapreduce.framework.name> and the correspond server
>         addresses.
>
>
>     I have tried passing mapreduce.framework.name
>     <http://mapreduce.framework.name> as both local and classic.   I
>     have also set those values in my mapreduce-site.xml.
>
>     Interestingly I can run the pagerank benchmark in code with the
>     command:
>
>         hadoop jar
>         ./giraph-core/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-cdh4.1.3-jar-with-dependencies.jar
>         org.apache.giraph.benchmark.PageRankBenchmark
>         -Dmapred.child.java-opts="-Xmx64g -Xms64g XX:+UseConcMarkSweepGC
>         -XX:-UseGCOverheadLimit" -Dgiraph.zkList=10.1.94.104:2181
>         <http://10.1.94.104:2181> -e 1 -s 3 -v
>         -V 50000 -w 83
>
>     And it completes just fine.
>
>     I have searched high and low for documents and examples on how to
>     run the example programs from other
>     than maven but have not found any thing.
>
>     Any help or suggestions  would be greatly appreciated.
>
>     THanks.
>
>
>
>     -- 
>     ========= mailto:dboyd@data-tactics.com
>     <mailto:dboyd@data-tactics.com> ============
>     David W. Boyd
>     Director, Engineering, Research and Development
>     Data Tactics Corporation
>     7901 Jones Branch, Suite 240
>     Mclean, VA 22102
>     office: +1-703-506-3735, ext 308
>     <tel:%2B1-703-506-3735%2C%20ext%20308>
>     fax: +1-703-506-6703 <tel:%2B1-703-506-6703>
>     cell: +1-703-402-7908 <tel:%2B1-703-402-7908>
>     ============== http://www.data-tactics.com/ ============
>
>     The information contained in this message may be privileged
>     and/or confidential and protected from disclosure.
>     If the reader of this message is not the intended recipient
>     or an employee or agent responsible for delivering this message
>     to the intended recipient, you are hereby notified that any
>     dissemination, distribution or copying of this communication
>     is strictly prohibited.  If you have received this communication
>     in error, please notify the sender immediately by replying to
>     this message and deleting the material from any computer.
>
>
>


-- 
========= mailto:dboyd@data-tactics.com ============
David W. Boyd
Director, Engineering, Research and Development
Data Tactics Corporation
7901 Jones Branch, Suite 240
Mclean, VA 22102
office:   +1-703-506-3735, ext 308
fax:     +1-703-506-6703
cell:     +1-703-402-7908
============== http://www.data-tactics.com/ ============
  

The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited.  If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.

  


Mime
View raw message