hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suhail Rehman <suhailreh...@gmail.com>
Subject Re: Using SequenceFiles in Hadoop for an imaging application.
Date Wed, 20 Jan 2010 07:54:03 GMT
This is the console output trying to run this application.  I'm sure the
input sequencefile is a (Text,BytesWritable) pair.

Suhail

10/01/20 10:52:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
10/01/20 10:52:34 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/01/20 10:52:34 INFO mapred.FileInputFormat: Total input paths to process
: 1
10/01/20 10:52:34 INFO mapred.JobClient: Running job: job_201001122200_0019
10/01/20 10:52:35 INFO mapred.JobClient:  map 0% reduce 0%
10/01/20 10:52:44 INFO mapred.JobClient: Task Id :
attempt_201001122200_0019_m_000000_0, Status : FAILED
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable incompatible
with org.apache.hadoop.io.Text
    at BlurMapper.map(BlurMapper.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

On Wed, Jan 20, 2010 at 5:41 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Could you paste your exception message ?
>
>
>
> On Wed, Jan 20, 2010 at 1:15 AM, Suhail Rehman <suhailrehman@gmail.com>wrote:
>
>> I am using hadoop to write some sample code which takes every image and
>> blurs it using a blurring filter.
>>
>> I was able to convert all my input images into a sequence file, and I've
>> written the following hadoop code to perform the blurring operation. (The
>> input sequencefile key-value pairs are Text (filename of the image),
>> BytesWritable (image contents) for each record).
>>
>> 've used the TAR to sequence file creator available here:
>> http://stuartsierra.com/2008/04/24/a-million-little-files
>>
>> But for some reason, I cannot get it working. I've pasted the code here
>> (not including the imports), let me know what I'm doing wrong (I'm new to
>> Hadoop btw).
>>
>> import com.jhlabs.image.BoxBlurFilter;
>>
>> //Mapper Class
>>
>> public class BlurMapper extends MapReduceBase implements
>> Mapper<Text,BytesWritable,Text,BytesWritable> {
>>
>>         public void map(Text key, BytesWritable file,
>>
>>
>>                         OutputCollector<Text, BytesWritable> output, Reporter
reporter) throws
>> IOException {
>>
>>
>>                    //Read Current Image from File.
>>                    BufferedImage img = ImageIO.read(new ByteArrayInputStream
>>
>>
>> (file.getBytes()));
>>                    BufferedImage dest = null;
>>
>>                    //Apply Blur on Filter Operation - External JAR
>>                    BoxBlurFilter BlurOp = new BoxBlurFilter(10,10,2);
>>                    BlurOp.filter(img, dest);
>>
>>
>>                    ByteArrayOutputStream outputbytes = new ByteArrayOutputStream();
>>                    ImageIO.write(dest, "jpeg", outputbytes);
>>                    BytesWritable outfile = new BytesWritable(outputbytes.toByteArray());
>>
>>
>>                    output.collect(key, outfile);
>>         }
>>
>> }
>>
>> //MAIN CLASS
>>
>> public class BlurVideoHadoop {
>>
>>         public static void main(String[] args) {
>>
>>                 if(args.length!=2) {
>>
>>
>>                         System.err.println("Usage: blurvideo input output");
>>                         System.exit(-1);
>>
>>                 }
>>                 JobClient client = new JobClient();
>>                 JobConf conf = new JobConf(BlurVideoHadoop.class);
>>
>>
>>                 conf.setOutputKeyClass(Text.class);
>>                 conf.setOutputValueClass(BytesWritable.class);
>>
>>                 SequenceFileInputFormat.addInputPath(conf, new Path(args[0]));
>>                 SequenceFileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>
>>
>>
>>                 conf.setMapperClass(BlurMapper.class);
>>
>>
>>                 conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);
>>
>>                 client.setConf(conf);
>>                 try {
>>
>>
>>                         JobClient.runJob(conf);
>>                 } catch (Exception e) {
>>                         e.printStackTrace();
>>                 }
>>         }
>>
>>
>>
>> Thanks,
>>
>> Regards,
>>
>> Suhail Rehman
>> MS by Research in Computer Science
>> International Institute of Information Technology - Hyderabad
>> rehman@research.iiit.ac.in
>> ---------------------------------------------------------------------
>> http://research.iiit.ac.in/~rehman <http://research.iiit.ac.in/%7Erehman>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Regards,

Suhail Rehman
MS by Research in Computer Science
International Institute of Information Technology - Hyderabad
rehman@research.iiit.ac.in
---------------------------------------------------------------------
http://research.iiit.ac.in/~rehman

Mime
View raw message