hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravindra yadav <raviin...@gmail.com>
Subject Fwd: Reduce task failing with MultipleOutputFormat
Date Sat, 10 Aug 2013 18:15:10 GMT
Hi,
I am trying to use mulitpleOutputFormat to generate outputs to multiple
files. But the reducer fails with nullpointer
at MultipleOutputFormat.write().


Following is the Job conf:

public class OutPutFileNameGenerator extends MultipleTextOutputFormat<Text,
Text>{

  protected String generateFileNameForKeyValue(Text key, Text value,
String name) {
 System.out.println( "Key is : " + key );
String str = "partitionFor_"  + key.toString();
  return str ;
}
}


@Override
public int run(String[] args) throws Exception {
Configuration conf = getConf();
 JobConf job = new JobConf(conf, DataPartitioner.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
 FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("Data Partitioner");
 job.setMapperClass(PartitionMapper.class);
job.setReducerClass(PartitionReducer.class);
job.setInputFormat(TextInputFormat.class);
 job.setOutputFormat(OutPutFileNameGenerator.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
// job.setNumReduceTasks(0);
JobClient.runJob(job);
return 0;
 }



Stack Trace:

3/08/10 17:25:11 INFO mapred.JobClient:  map 100% reduce 0%
13/08/10 17:25:21 INFO mapred.JobClient:  map 100% reduce 33%
13/08/10 17:25:24 INFO mapred.JobClient: Task Id :
attempt_201308101504_0009_r_000000_1, Status : FAILED
java.lang.NullPointerException
        at
org.apache.hadoop.mapred.lib.MultipleOutputFormat$1.write(MultipleOutputFormat.java:102)
        at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
        at
org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:498)
        at org.partitioner.PartitionReducer.reduce(PartitionReducer.java:24)
        at org.partitioner.PartitionReducer.reduce(PartitionReducer.java:1)
        at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)


-- 
- Ravi



-- 
- Ravi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message