hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: Extending ClusterMapReduceTestCase
Date Tue, 10 Mar 2009 21:28:58 GMT
There are a couple of failures that happen in tests derived from
ClusterMapReduceTestCase that are run outside of the hadoop unit test
framework.

The basic issue is that the unit test doesn't have the benefit of a runtime
environment setup by the bin/hadoop script.

The classpath is usually missing the lib/jetty-ext/*.jar files, and doesn't
get the conf/hadoop-default.xml and conf/hadoop-site.xml.
The *standard* properties are also unset.. hadoop.log.dir, hadoop.log.file,
hadoop.home.dir, hadoop.id.str, hadoop.root.logger

I find that I can get away with just defining hadoop.log.dir.

You can read about this in detail in the chapter on unit testing map/reduce
jobs in my book, out real soon now :)



On Tue, Mar 10, 2009 at 12:08 PM, Brian Forney <bforney@integral7.com>wrote:

> Hi all,
>
> I'm trying to write a JUnit test case that extends ClusterMapReduceTestCase
> to test some code I've written to ease job submission and monitoring
> between
> some existing code. Unfortunately, I see the following problem and cannot
> find the jetty 5.1.4 code anywhere online. Any ideas about why this is
> happening?
>
>    [junit] Testsuite: com.integral7.batch.hadoop.test.TestJobController
>    [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.384 sec
>    [junit]
>    [junit] ------------- Standard Output ---------------
>    [junit] 2009-03-10 12:52:26,303 [main] ERROR
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
> :290) - FSNamesystem initialization failed.
>    [junit] java.io.IOException: Problem starting http server
>    [junit]     at
> org.apache.hadoop.http.HttpServer.start(HttpServer.java:343)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
> java:379)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
> :288)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
> )
>    [junit]     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>    [junit]     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
> :859)
>    [junit]     at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
>    [junit]     at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>    [junit]     at
>
> org.apache.hadoop.mapred.ClusterMapReduceTestCase.startCluster(ClusterMapRed
> uceTestCase.java:81)
>    [junit]     at
>
> org.apache.hadoop.mapred.ClusterMapReduceTestCase.setUp(ClusterMapReduceTest
> Case.java:56)
>    [junit]     at
>
> com.integral7.batch.hadoop.test.TestJobController.setUp(TestJobController.ja
> va:49)
>    [junit]     at junit.framework.TestCase.runBare(TestCase.java:132)
>    [junit]     at junit.framework.TestResult$1.protect(TestResult.java:110)
>    [junit]     at
> junit.framework.TestResult.runProtected(TestResult.java:128)
>    [junit]     at junit.framework.TestResult.run(TestResult.java:113)
>    [junit]     at junit.framework.TestCase.run(TestCase.java:124)
>    [junit]     at junit.framework.TestSuite.runTest(TestSuite.java:232)
>    [junit]     at junit.framework.TestSuite.run(TestSuite.java:227)
>    [junit]     at
>
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81
> )
>    [junit]     at
> junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:36)
>    [junit]     at
>
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
> nner.java:421)
>    [junit]     at
>
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTes
> tRunner.java:912)
>    [junit]     at
>
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
> unner.java:766)
>    [junit] Caused by:
> org.mortbay.util.MultiException[org.xml.sax.SAXParseException: The
> processing instruction target matching "[xX][mM][lL]" is not allowed.,
> org.xml.sax.SAXParseException: The processing instruction target matching
> "[xX][mM][lL]" is not allowed.]
>    [junit]     at org.mortbay.http.HttpServer.doStart(HttpServer.java:731)
>    [junit]     at org.mortbay.util.Container.start(Container.java:72)
>    [junit]     at
> org.apache.hadoop.http.HttpServer.start(HttpServer.java:321)
>    [junit]     ... 23 more
>    [junit] ------------- ---------------- ---------------
>    [junit] Testcase:
> testJobSubmission(com.integral7.batch.hadoop.test.TestJobController):
> Caused an ERROR
>    [junit] Problem starting http server
>    [junit] java.io.IOException: Problem starting http server
>    [junit]     at
> org.apache.hadoop.http.HttpServer.start(HttpServer.java:343)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
> java:379)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
> :288)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
> )
>    [junit]     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>    [junit]     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>    [junit]     at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
> :859)
>    [junit]     at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
>    [junit]     at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>    [junit]     at
>
> org.apache.hadoop.mapred.ClusterMapReduceTestCase.startCluster(ClusterMapRed
> uceTestCase.java:81)
>    [junit]     at
>
> org.apache.hadoop.mapred.ClusterMapReduceTestCase.setUp(ClusterMapReduceTest
> Case.java:56)
>    [junit]     at
>
> com.integral7.batch.hadoop.test.TestJobController.setUp(TestJobController.ja
> va:49)
>    [junit] Caused by:
> org.mortbay.util.MultiException[org.xml.sax.SAXParseException: The
> processing instruction target matching "[xX][mM][lL]" is not allowed.,
> org.xml.sax.SAXParseException: The processing instruction target matching
> "[xX][mM][lL]" is not allowed.]
>    [junit]     at org.mortbay.http.HttpServer.doStart(HttpServer.java:731)
>    [junit]     at org.mortbay.util.Container.start(Container.java:72)
>    [junit]     at
> org.apache.hadoop.http.HttpServer.start(HttpServer.java:321)
>    [junit]
>    [junit]
>    [junit] Test com.integral7.batch.hadoop.test.TestJobController FAILED
>
> Thanks,
> Brian
>
>


-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message