hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Piggott <cpigg...@gmail.com>
Subject Hbase Reducer NPE
Date Fri, 07 Oct 2016 20:03:47 GMT
This question is pertaining to hadoop with core version 1.2.1 and hbase
1.2.3.

I wrote a simple map/reduce job that looks like this:

- The input to the mapper is whole HDFS files at a time, via a custom
InputFormat
- The output of the mapper is <LongWriteable, Text>

The job is configured like this:

    Configuration conf = getConf();

    JobConf cfg = new JobConf(conf, FindMissionTimeJob.class);

    /* Set up mapper */
    Path inputPath = new Path(args[0]);
    WholeFileInputFormat.setInputPaths(cfg, inputPath);
    cfg.setNumMapTasks(1);
    cfg.setMapperClass(TimeMapper.class);
    cfg.setInputFormat(WholeFileInputFormat.class);
    cfg.setMapOutputKeyClass(LongWritable.class);
    cfg.setMapOutputValueClass(Text.class);

    cfg.setNumReduceTasks(1);
    TableMapReduceUtil.initTableReduceJob(tableName, TimeReducer.class,
cfg);


When I run it, I get an NPE here:

16/10/07 15:33:55 INFO mapreduce.Job: Task Id :
attempt_1475608557171_0021_r_000000_2, Status : FAILED

Error: java.lang.NullPointerException
        at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:373)
        at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:478)
        at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)


Tracing the source of that, it really comes down to this bit of code in
ReduceTask.java that is failing:

  if (job.getOutputFormat() instanceof FileOutputFormat) {

      matchedStats = getFsStatistics(FileOutputFormat.getOutputPath(job),
job);

  }


getFsStatistics() throws an NPE becaue there is no output path.  The
problem is, we shoud not even get there, because initTableReduceJob()
already set the output format to this::

    job.setOutputFormat(TableOutputFormat.class);


Looking at the source for that guy:

    public class TableOutputFormat extends
FileOutputFormat<ImmutableBytesWritable, Put>

Ahh hah.  There's the problem.  That extends FileOutputFormat, which causes
everything to blow up because it's a table reducer.  It's not really a File
output, but getFsStatistics is invoked and subsequently blows up.  It seems
likely that getFsStatistics should check for a NULL here, and perhaps that
TableOutputFormat probably should not extend FileOutputFormat - I'm not
sure.


The way I got around this was to create my own OutputFormat called
MyTableOutputFormat that was essentially a copy of the real
TableOutputFormat but instead of extending FIleOutputFormat I did this:

    public class MyTableOutputFormat implements
OutputFormat<ImmutableBytesWritable, Put> { ... }


I don't know if that's a correct or incorrect solution... just that it runs
an empty hbase-outputting reducer without blowing up on an NPE.

I would appreciate any comments/feedback on the problem as well as my
workaround, and whether or not anybody else has encountered this.

--Chris

Mime
View raw message