hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kunal Gupta <ku...@techlead-india.com>
Subject Re: How to write a custom input format and record reader to read multiple lines of text from files
Date Tue, 01 Dec 2009 07:34:42 GMT
I have not implemented a constructor for the class extending the
FileInputFormat class.
I had actually implemented a argument constructor for the class
extending RecordReader. This class was not having a no-arg constructor.
After reading your comment i wrote a no-arg constructor for this class,
but still getting the same error.

Following is the class extending the FileInputFormat class:

	public class MultiLineFileInputFormat
	extends FileInputFormat<LongWritable, Text> {

		/*public MultiLineFileInputFormat()
		{
			super();
		}*/
		
		@Override
		public RecordReader createRecordReader(InputSplit split,
                TaskAttemptContext context)
         throws IOException, InterruptedException 
         {
			
				context.setStatus(split.toString());
				return new MultiLineFileRecordReader((FileSplit) split, context);
         }
		
	}


On Tue, 2009-12-01 at 07:20 +0000, Sean Owen wrote:
> It sounds like you have declared a constructor in
> MultiLineFileInputFormat that needs an argument. By doing so, no
> no-arg constructor is automatically generated. Unless you write one,
> it won't exist. The Hadoop framework instantiates your class by
> calling the no-arg constructor. The error you get says this directly:
> there is no no-arg constructor. Write one to fix it.
> 
> The example you reference has a no-arg constructor, by default, since
> it declares no constructors at all.
> 
> On Tue, Dec 1, 2009 at 6:57 AM, Kunal Gupta <kunal@techlead-india.com> wrote:
> > Can you kindly guide me on what initialisation i need to do in the
> > implemented class constructor - MultiLineFileInputFormat?
> >
> > i was following the sample provided on this yahoo page:
> >
> > http://developer.yahoo.com/hadoop/tutorial/module5.html#fileformat
> >
> >
> >
> >
> > On Tue, 2009-12-01 at 06:45 +0000, Sean Owen wrote:
> >> It sounds like you have no provided a no-arg constructor in
> >> MultiLineFileInputFormat.
> >>
> >> On Tue, Dec 1, 2009 at 6:17 AM, Kunal Gupta <kunal@techlead-india.com>
wrote:
> >> > Can someone explain how to override the "FileInputFormat" and
> >> > "RecordReader" in order to be able to read multiple lines of text from
> >> > input files in a single map task?
> >> >
> >> > Here the key will be the offset of the first line of text and value will
> >> > be the N lines of text.
> >> >
> >> > I have overridden the class FileInputFormat:
> >> >
> >> > public class MultiLineFileInputFormat
> >> >        extends FileInputFormat<LongWritable, Text>{
> >> > ...
> >> > }
> >> >
> >> > and implemented the abstract method:
> >> >
> >> > public RecordReader createRecordReader(InputSplit split,
> >> >                TaskAttemptContext context)
> >> >         throws IOException, InterruptedException {...}
> >> >
> >> > I have also overridden the recordreader class:
> >> >
> >> > public class MultiLineFileRecordReader extends
> >> > RecordReader<LongWritable, Text>
> >> > {...}
> >> >
> >> > and in the job configuration, specified this new InputFormat class:
> >> >
> >> > job.setInputFormatClass(MultiLineFileInputFormat.class);
> >> >
> >> > --------------------------------------------------------------------------
> >> > When I  run this new map/reduce program, i get the following java error:
> >> > --------------------------------------------------------------------------
> >> > Exception in thread "main" java.lang.RuntimeException:
> >> > java.lang.NoSuchMethodException: CustomRecordReader
> >> > $MultiLineFileInputFormat.<init>()
> >> >        at
> >> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> >> >        at
> >> > org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:882)
> >> >        at
> >> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> >> >        at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> >> >        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
> >> >        at CustomRecordReader.main(CustomRecordReader.java:257)
> >> > Caused by: java.lang.NoSuchMethodException: CustomRecordReader
> >> > $MultiLineFileInputFormat.<init>()
> >> >        at java.lang.Class.getConstructor0(Class.java:2706)
> >> >        at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> >> >        at
> >> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> >> >        ... 5 more
> >> >
> >> >
> >>
> >
> >
> 


Mime
View raw message