hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject MiniMRCluster not behaving in hadoop 1.0.4
Date Sat, 10 Nov 2012 07:43:49 GMT
i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
unit tests used to run fine but are now misbehaving. i do a simple setup in
the unit test like this:

    private static MiniDFSCluster dfsCluster;
    private static FileSystem fs;
    private static MiniMRCluster mrCluster;

    public static void oneTimeSetUp() throws Exception {
        new File("tmp/logs").mkdirs();
        System.setProperty("hadoop.log.dir", "tmp/logs");

        final Configuration conf = new Configuration();
        conf.set("io.skip.checksum.errors", "true");
        dfsCluster = new MiniDFSCluster(conf, 2, true, null);
        fs = dfsCluster.getFileSystem();
        mrCluster = new MiniMRCluster(2,
dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new

however with hadoop 1.0.4 this results in an exception where the jobtracker

12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
    at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)

not sure what this error is about, but i managed to get around it for now
by removing the 2 hadoop apache jasper dependencies which seem to have
something to do with it: jasper-compiler-5.5.12.jar and
jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
starts, but now all the tasks fail with messages like this:

attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child

am i doing something completely wrong, and was i just lucky that this used
to run with CDH3? any ideas? thanks! koert

View raw message