hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Vivancos <pedro.vivan...@vocali.net>
Subject How to debug a MapReduce application
Date Fri, 16 Jan 2009 18:41:43 GMT
Dear friends,

I am new at Hadoop and at MapReduce techniques. I've developed my first
map-reduce application using hadoop but I can't manage to make it work. I
get the following error at the very beginning of the execution:

java.lang.NullPointerException
    at
org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:504)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:295)
    at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
16-ene-2009 18:29:30 es.vocali.intro.tools.memo.MemoAnnotationMerging main
GRAVE: Se ha producido un error
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)
    at
es.vocali.intro.tools.memo.MemoAnnotationMerging.main(MemoAnnotationMerging.java:160)
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)
    at
es.vocali.intro.tools.memo.MemoAnnotationMerging.main(MemoAnnotationMerging.java:160)

Sorry if I don't give you more information but I don't know where to start
to find the error. My app is quite simple. It just gets some rows from a
postgresql database and try to see which ones can be deleted.

Here you have the configuration I am using:

MemoAnnotationMerging memo = new MemoAnnotationMerging();

        Map<String, String> parametros = memo.checkParams(args);

        memo.initDataStore(parametros.get(DATASTORE_URL));

        JobConf conf = new JobConf(MemoAnnotationMerging.class);
        conf.setJobName("memo - annotation merging");

        conf.setMapperClass(MemoAnnotationMapper.class);
        conf.setCombinerClass(MemoAnnotationReducer.class);
        conf.setReducerClass(MemoAnnotationReducer.class);

        DBConfiguration.configureDB(conf, DRIVER_CLASS,
parametros.get(DATASTORE_URL));

        // ???
        //conf.setInputFormat(DBInputFormat.class);
        //conf.setOutputFormat(TextOutputFormat.class);

        conf.setMapOutputKeyClass(LongWritable.class);
        conf.setMapOutputValueClass(Annotation.class);


        //conf.setOutputKeyClass(Annotation.class);
        //conf.setOutputValueClass(BooleanWritable.class);

        DBInputFormat.setInput(conf, MemoAnnotationDBWritable.class,
GET_ANNOTATIONS_QUERY, COUNT_ANNOTATIONS_QUERY);

        FileOutputFormat.setOutputPath(conf, new Path("eliminar.txt"));

        // ejecutamos el algoritmo map-reduce para mezclar anotaciones
        try {
            JobClient.runJob(conf);

        } catch (IOException e) {
            e.printStackTrace();
            System.exit(-1);
        }

Thanks in advance.

 Pedro Vivancos Vicente
VĂ³cali Sistemas Inteligentes S.L. <http://www.vocali.net>
Edificio CEEIM, Campus de Espinardo
30100, Espinardo, Murcia, Spain
Tel. +34 902 929 644  <http://www.vocali.net>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message