hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: Extending ClusterMapReduceTestCase
Date Tue, 10 Mar 2009 21:32:17 GMT
The other goofy thing is that the  xml parser that is commonly first in the
class path, validates xml in a way that is opposite to what jetty wants.

This line in the preamble before theClusterMapReduceTestCase setup takes
care of the xml errors.

System.setProperty("javax.xml.parsers.SAXParserFactory","org.apache.xerces.jaxp.SAXParserFactoryImpl");


On Tue, Mar 10, 2009 at 2:28 PM, jason hadoop <jason.hadoop@gmail.com>wrote:

> There are a couple of failures that happen in tests derived from
> ClusterMapReduceTestCase that are run outside of the hadoop unit test
> framework.
>
> The basic issue is that the unit test doesn't have the benefit of a runtime
> environment setup by the bin/hadoop script.
>
> The classpath is usually missing the lib/jetty-ext/*.jar files, and doesn't
> get the conf/hadoop-default.xml and conf/hadoop-site.xml.
> The *standard* properties are also unset.. hadoop.log.dir,
> hadoop.log.file, hadoop.home.dir, hadoop.id.str, hadoop.root.logger
>
> I find that I can get away with just defining hadoop.log.dir.


>
> You can read about this in detail in the chapter on unit testing map/reduce
> jobs in my book, out real soon now :)
>
>
>
>
> On Tue, Mar 10, 2009 at 12:08 PM, Brian Forney <bforney@integral7.com>wrote:
>
>> Hi all,
>>
>> I'm trying to write a JUnit test case that extends
>> ClusterMapReduceTestCase
>> to test some code I've written to ease job submission and monitoring
>> between
>> some existing code. Unfortunately, I see the following problem and cannot
>> find the jetty 5.1.4 code anywhere online. Any ideas about why this is
>> happening?
>>
>>    [junit] Testsuite: com.integral7.batch.hadoop.test.TestJobController
>>    [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.384 sec
>>    [junit]
>>    [junit] ------------- Standard Output ---------------
>>    [junit] 2009-03-10 12:52:26,303 [main] ERROR
>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
>> :290) - FSNamesystem initialization failed.
>>    [junit] java.io.IOException: Problem starting http server
>>    [junit]     at
>> org.apache.hadoop.http.HttpServer.start(HttpServer.java:343)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
>> java:379)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
>> :288)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
>> )
>>    [junit]     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>>    [junit]     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
>> :859)
>>    [junit]     at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
>>    [junit]     at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>>    [junit]     at
>>
>> org.apache.hadoop.mapred.ClusterMapReduceTestCase.startCluster(ClusterMapRed
>> uceTestCase.java:81)
>>    [junit]     at
>>
>> org.apache.hadoop.mapred.ClusterMapReduceTestCase.setUp(ClusterMapReduceTest
>> Case.java:56)
>>    [junit]     at
>>
>> com.integral7.batch.hadoop.test.TestJobController.setUp(TestJobController.ja
>> va:49)
>>    [junit]     at junit.framework.TestCase.runBare(TestCase.java:132)
>>    [junit]     at
>> junit.framework.TestResult$1.protect(TestResult.java:110)
>>    [junit]     at
>> junit.framework.TestResult.runProtected(TestResult.java:128)
>>    [junit]     at junit.framework.TestResult.run(TestResult.java:113)
>>    [junit]     at junit.framework.TestCase.run(TestCase.java:124)
>>    [junit]     at junit.framework.TestSuite.runTest(TestSuite.java:232)
>>    [junit]     at junit.framework.TestSuite.run(TestSuite.java:227)
>>    [junit]     at
>>
>> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81
>> )
>>    [junit]     at
>> junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:36)
>>    [junit]     at
>>
>> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
>> nner.java:421)
>>    [junit]     at
>>
>> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTes
>> tRunner.java:912)
>>    [junit]     at
>>
>> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
>> unner.java:766)
>>    [junit] Caused by:
>> org.mortbay.util.MultiException[org.xml.sax.SAXParseException: The
>> processing instruction target matching "[xX][mM][lL]" is not allowed.,
>> org.xml.sax.SAXParseException: The processing instruction target matching
>> "[xX][mM][lL]" is not allowed.]
>>    [junit]     at org.mortbay.http.HttpServer.doStart(HttpServer.java:731)
>>    [junit]     at org.mortbay.util.Container.start(Container.java:72)
>>    [junit]     at
>> org.apache.hadoop.http.HttpServer.start(HttpServer.java:321)
>>    [junit]     ... 23 more
>>    [junit] ------------- ---------------- ---------------
>>    [junit] Testcase:
>> testJobSubmission(com.integral7.batch.hadoop.test.TestJobController):
>> Caused an ERROR
>>    [junit] Problem starting http server
>>    [junit] java.io.IOException: Problem starting http server
>>    [junit]     at
>> org.apache.hadoop.http.HttpServer.start(HttpServer.java:343)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
>> java:379)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
>> :288)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
>> )
>>    [junit]     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>>    [junit]     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>>    [junit]     at
>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
>> :859)
>>    [junit]     at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
>>    [junit]     at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
>>    [junit]     at
>>
>> org.apache.hadoop.mapred.ClusterMapReduceTestCase.startCluster(ClusterMapRed
>> uceTestCase.java:81)
>>    [junit]     at
>>
>> org.apache.hadoop.mapred.ClusterMapReduceTestCase.setUp(ClusterMapReduceTest
>> Case.java:56)
>>    [junit]     at
>>
>> com.integral7.batch.hadoop.test.TestJobController.setUp(TestJobController.ja
>> va:49)
>>    [junit] Caused by:
>> org.mortbay.util.MultiException[org.xml.sax.SAXParseException: The
>> processing instruction target matching "[xX][mM][lL]" is not allowed.,
>> org.xml.sax.SAXParseException: The processing instruction target matching
>> "[xX][mM][lL]" is not allowed.]
>>    [junit]     at org.mortbay.http.HttpServer.doStart(HttpServer.java:731)
>>    [junit]     at org.mortbay.util.Container.start(Container.java:72)
>>    [junit]     at
>> org.apache.hadoop.http.HttpServer.start(HttpServer.java:321)
>>    [junit]
>>    [junit]
>>    [junit] Test com.integral7.batch.hadoop.test.TestJobController FAILED
>>
>> Thanks,
>> Brian
>>
>>
>
>
> --
> Alpha Chapters of my book on Hadoop are available
> http://www.apress.com/book/view/9781430219422
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message