giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: Getting Giraph to build/run with CDH4 (cannot initialize cluster - check mapreduce.framework.name)
Date Wed, 27 Feb 2013 02:01:22 GMT
Just to throw this out there: it has been noted (by me most recently) on
the Giraph JIRA site that the tests aren't happy when you try to run them
against a running cluster, they like to instantiate their own local
resources (ZK, Hadoop single-node) for the tests. If your example jobs run
with "hadoop jar" on the cluster than thats what matters, you're all set.


On Mon, Feb 25, 2013 at 4:03 PM, David Boyd <dboyd@data-tactics-corp.com>wrote:

>  Sandy:
>    Yes.   Attached is the segment from the job tracker log file that shows
> the error and stack traces.
>
> The maven surefire report for the test shows an assertion failure on the
> following line from
> the test:
>
> assertTrue(job.run(true));
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.giraph.io.TestJsonBase64Format
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.363 sec
> <<< FAILURE!
> testContinue(org.apache.giraph.io.TestJsonBase64Format)  Time elapsed:
> 32.352 sec  <<< FAILURE!
> java.lang.AssertionError:
>         at org.junit.Assert.fail(Assert.java:91)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at org.junit.Assert.assertTrue(Assert.java:54)
>         at
> org.apache.giraph.io.TestJsonBase64Format.testContinue(TestJsonBase64Format.java:74)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>         at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>         at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>         at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>         at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>         at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>         at
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
>         at
> org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
>         at
> org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)
>         at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
>         at
> org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
>
>
> Below is the surefire report stack trace:
>
>
> On 2/25/2013 6:55 PM, Sandy Ryza wrote:
>
> Great to hear it helped.  Are you able to provide the full stack trace for
> that exception?
>
> thanks,
> Sandy
>
>
> On Mon, Feb 25, 2013 at 3:51 PM, David Boyd <dboyd@data-tactics-corp.com>wrote:
>
>>  Sandy:
>>    Thanks that helps a great deal.  I am now at least getting to the
>> point that the jobs show up in the job tracker.   However, they all
>> fail on initialization with the good old:
>>
>> java.io.FileNotFoundException: File /tmp/hadoop-mapred/mapred/staging/hdfs/.staging/job_201302211213_0055/job.jar
does not exist
>>
>>  This tells me that maven is either not specifying that the giraph-core
>> jar file should be used as the job jar or I am missing
>> something else in the set up.
>>
>> Attached is the job.xml file from one of the failed jobs and below is the
>> relevant profile out of my pom.xml.
>> I did upgrade to CDH4.1.3 just to see if that would help.
>> Also, I have been running all sorts of jobs (benchmarks, and other tests)
>> against this cluster for some time so I know that the cluster
>> works well.
>>
>> Again, any help is appreciated.
>>
>> Relevant section of pom.xml:
>>
>>     <profile>
>>       <id>hadoop_cdh4.1.3mr1</id>
>>       <properties>
>>         <hadoopmr1.version>2.0.0-mr1-cdh4.1.3</hadoopmr1.version>
>>         <hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
>>
>> <munge.symbols>HADOOP_1_SECURITY,HADOOP_1_SECRET_MANAGER</munge.symbols>
>>       </properties>
>>       <dependencies>
>>         <!-- sorted lexicographically -->
>>         <dependency>
>>           <groupId>commons-net</groupId>
>>           <artifactId>commons-net</artifactId>
>>         </dependency>
>>         <dependency>
>>           <groupId>org.apache.hadoop</groupId>
>>           <artifactId>hadoop-client</artifactId>
>>           <version>${hadoopmr1.version}</version>
>>           <scope>provided</scope>
>>         </dependency>
>>         <dependency>
>>           <groupId>org.apache.hadoop</groupId>
>>           <artifactId>hadoop-common</artifactId>
>>           <version>${hadoop.version}</version>
>>           <scope>provided</scope>
>>         </dependency>
>>         <dependency>
>>           <groupId>org.apache.hadoop</groupId>
>>           <artifactId>hadoop-hdfs</artifactId>
>>           <version>${hadoop.version}</version>
>>           <scope>provided</scope>
>>         </dependency>
>>         <dependency>
>>           <groupId>org.apache.hadoop</groupId>
>>           <artifactId>hadoop-test</artifactId>
>>           <version>${hadoopmr1.version}</version>
>>           <scope>provided</scope>
>>         </dependency>
>>       </dependencies>
>>     </profile>
>>
>>
>>
>>
>> On 2/25/2013 12:47 PM, Sandy Ryza wrote:
>>
>> Hi David,
>>
>>  Moving this to cdh-user, as it is CDH-specific.
>>
>>  CDH4 comes with two versions of mapreduce, MR1, and MR2.  It sounds
>> like you are building against MR2 (
>> http://blog.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/).
>>  Do you know whether your cluster runs MR2/YARN or MR1?  If it runs, MR2,
>> you can set mapreduce.framework.name to "yarn".  If it runs MR1, you can
>> build against the MR1 jar by setting the version of your hadoop-client to 2.0.0-mr1-cdh4.1.1.
>> (
>> https://ccp.cloudera.com/display/CDH4DOC/Managing+Hadoop+API+Dependencies+in+CDH4
>> )
>> Does that help?
>>
>>  -Sandy
>>
>>
>> On Mon, Feb 25, 2013 at 8:26 AM, David Boyd <dboyd@data-tactics-corp.com>wrote:
>>
>>> All:
>>>    I am trying to get the Giraph 0.2 snapshot (pulled via GIT on Friday)
>>> to build and run with CDH4.
>>>
>>> I modified the pom.xml to provide a profile for my specific version
>>> (4.1.1).
>>> The build works (mvn -Phadoop_cdh4.1.1 clean package test) and passes
>>> all the tests.
>>>
>>> If I try to do the next step and submit to my cluster with the command:
>>> mvn -Phadoop_cdh4.1.1 test -Dprop.mapred.job.tracker=10.1.94.53:8021-Dgiraph.zkList=
>>> 10.1.94.104:2181
>>>
>>>  the JSON test in core fails.  If I move that test out of the way a
>>> whole bunch of tests in examples
>>> fail.  They all fail with:
>>>
>>>> java.io.IOException: Cannot initialize Cluster. Please check your
>>>> configuration for mapreduce.framework.name and the correspond server
>>>> addresses.
>>>>
>>>
>>> I have tried passing mapreduce.framework.name as both local and
>>> classic.   I have also set those values in my mapreduce-site.xml.
>>>
>>> Interestingly I can run the pagerank benchmark in code with the command:
>>>
>>>> hadoop jar
>>>>
>>>> ./giraph-core/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-cdh4.1.3-jar-with-dependencies.jar
>>>> org.apache.giraph.benchmark.PageRankBenchmark
>>>> -Dmapred.child.java-opts="-Xmx64g -Xms64g XX:+UseConcMarkSweepGC
>>>> -XX:-UseGCOverheadLimit" -Dgiraph.zkList=10.1.94.104:2181 -e 1 -s 3 -v
>>>> -V 50000 -w 83
>>>>
>>> And it completes just fine.
>>>
>>> I have searched high and low for documents and examples on how to run
>>> the example programs from other
>>> than maven but have not found any thing.
>>>
>>> Any help or suggestions  would be greatly appreciated.
>>>
>>> THanks.
>>>
>>>
>>>
>>> --
>>> ========= mailto:dboyd@data-tactics.com ============
>>> David W. Boyd
>>> Director, Engineering, Research and Development
>>> Data Tactics Corporation
>>> 7901 Jones Branch, Suite 240
>>> Mclean, VA 22102
>>> office:   +1-703-506-3735, ext 308 <%2B1-703-506-3735%2C%20ext%20308>
>>> fax:     +1-703-506-6703
>>> cell:     +1-703-402-7908
>>> ============== http://www.data-tactics.com/ ============
>>>
>>> The information contained in this message may be privileged
>>> and/or confidential and protected from disclosure.
>>> If the reader of this message is not the intended recipient
>>> or an employee or agent responsible for delivering this message
>>> to the intended recipient, you are hereby notified that any
>>> dissemination, distribution or copying of this communication
>>> is strictly prohibited.  If you have received this communication
>>> in error, please notify the sender immediately by replying to
>>> this message and deleting the material from any computer.
>>>
>>>
>>>
>>
>>
>> --
>> ========= mailto:dboyd@data-tactics.com <dboyd@data-tactics.com> ============
>> David W. Boyd
>> Director, Engineering, Research and Development
>> Data Tactics Corporation
>> 7901 Jones Branch, Suite 240
>> Mclean, VA 22102
>> office:   +1-703-506-3735, ext 308
>> fax:     +1-703-506-6703
>> cell:     +1-703-402-7908
>> ============== http://www.data-tactics.com/ ============
>>
>>
>> The information contained in this message may be privileged
>> and/or confidential and protected from disclosure.
>> If the reader of this message is not the intended recipient
>> or an employee or agent responsible for delivering this message
>> to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication
>> is strictly prohibited.  If you have received this communication
>> in error, please notify the sender immediately by replying to
>> this message and deleting the material from any computer.
>>
>>
>>
>>
>
>
> --
> ========= mailto:dboyd@data-tactics.com <dboyd@data-tactics.com> ============
> David W. Boyd
> Director, Engineering, Research and Development
> Data Tactics Corporation
> 7901 Jones Branch, Suite 240
> Mclean, VA 22102
> office:   +1-703-506-3735, ext 308
> fax:     +1-703-506-6703
> cell:     +1-703-402-7908
> ============== http://www.data-tactics.com/ ============
>
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited.  If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>
>
>

Mime
View raw message