hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: setOutputValueClass(*.class) method
Date Thu, 11 Oct 2012 18:25:03 GMT
Bertand

I am reading my input by declaring it in the driver as follows:

conf.setInputFormat(SequenceFileInputFormat.class);


In the mapper:
public void map(LongWritable key, SomeInterfaceClass value,
                        OutputCollector<Text, SomeInterfaceClass > output, Reporter
reporter)
                                throws IOException {
                SomeInterfaceClass o = null;
                        if ( value instanceof classA) {
                          o = (classA)value;
                  }
                        if ( value instanceof classB) {
                          o = (classB)value;
                  }
                output.collect(new Text(o.get_property_id().toString()), o);
        } // end of map()
     } // end of static class

As u can see "o" can be an instance of classA or classB, where both implement SomeInerfaceClass.

However, when I try to write in the driver:
Conf.setOutputValueClass(SomeIterfaceClass.class);
And run the job I get the following error

Error: java.io.IOException: Type mismatch in value from map: expected SomeInterfaceClass,
received ClassA

I wished I could change the outputClass name depending on the instance of "o" to either ClassA
or ClassB.



From: Bertrand Dechoux [mailto:dechouxb@gmail.com]
Sent: Thursday, October 11, 2012 2:12 PM
To: user@hadoop.apache.org
Subject: Re: setOutputValueClass(*.class) method

Hi Andy,

It should be called within the setup of your job and it should NOT be called within the Mapper.
You could compile the call within the mapper of course. At best it should do nothing. I would
say in practice the job will fail. But that's something that you have to verify if you are
curious.

Back to your original problem, the mapper need to output the same kind of object in order
to be able to unserialize them for the mapper. (More generally the serialization strategy
must be able to figure out what was serialized.) It does not mean your problem is not solvable
but it means that you will need to come up with a wrapper or a common structure.

How do you read different kind of objects with the Mapper? Maybe you could use the same approach.

Regards

Bertrand
On Thu, Oct 11, 2012 at 7:49 PM, Kartashov, Andy <Andy.Kartashov@mpac.ca<mailto:Andy.Kartashov@mpac.ca>>
wrote:
Guys,

Is there another way to set the output from within the Mapper?

My Mapper reads from various serialised files and generates different type of objects for
values depending on the returned value of instanceof. I wanted to change the class name within
the Mapper as opposed to the driver class. Anyway of doing it?

Thanks,
Andy

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel



--
Bertrand Dechoux
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Mime
View raw message