Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D4F25DA9C for ; Thu, 11 Oct 2012 18:25:37 +0000 (UTC) Received: (qmail 28294 invoked by uid 500); 11 Oct 2012 18:25:33 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 28049 invoked by uid 500); 11 Oct 2012 18:25:32 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 28040 invoked by uid 99); 11 Oct 2012 18:25:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Oct 2012 18:25:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [206.47.135.205] (HELO Spam1.prd.mpac.ca) (206.47.135.205) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Oct 2012 18:25:27 +0000 Received: from Spam1.prd.mpac.ca (unknown [127.0.0.1]) by IMSVA80 (Postfix) with ESMTP id 606EE1D805D for ; Thu, 11 Oct 2012 14:25:05 -0400 (EDT) Received: from SMAIL1.prd.mpac.ca (unknown [172.29.2.53]) by Spam1.prd.mpac.ca (Postfix) with ESMTP id 1098D1D8045 for ; Thu, 11 Oct 2012 14:25:05 -0400 (EDT) Received: from SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb]) by SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb%16]) with mapi id 14.02.0318.001; Thu, 11 Oct 2012 14:25:04 -0400 From: "Kartashov, Andy" To: "user@hadoop.apache.org" Subject: RE: setOutputValueClass(*.class) method Thread-Topic: setOutputValueClass(*.class) method Thread-Index: Ac2n2L5oR4ekGJfVQwy778iTFBXg+QAJLiuAAAhJsnA= Date: Thu, 11 Oct 2012 18:25:03 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.29.60.102] Content-Type: multipart/alternative; boundary="_000_BD42F346AE90F544A731516A805D1B8AD713C8SMAIL1prdmpacca_" MIME-Version: 1.0 X-TM-AS-Product-Ver: IMSVA-8.0.0.1304-6.5.0.1024-19264.001 X-TM-AS-Result: No--30.235-5.0-31-10 X-imss-scan-details: No--30.235-5.0-31-10 X-TM-AS-Result-Xfilter: Match text exemption rules:No X-TMASE-MatchedRID: Rp71wniPtoPZfnct5UBzcYa7OiQBC9buuLwbhNl9B5Uj/BgXooqgw0sJ 9gxCreTkLgFWr0XXuuREk5CFGx+q7JShxRaS8Dn14pdq9sdj8LUrHkgIan9a0UUNHQAoZf5cr1m MCv7DNtvA0SZZnp9s5Ivn8CCdcmQih0zWQXCZZ8ibfq32NGAdgYSujFJOXwz9ze5MPJBEaTQ9da gwNh/l7NJKRyQnwzbacgRuMUq2K0h9NgiFro7s/J1v5WrnSRXh0i/hFXziUdPE3grQNcpLWMvUB RfNgv+CjRMwRxWMc3Bb8uIpsgPIHQi0xaFtKyGqlGudLLtRO1vuHZGuwo6K7QbYcy9YQl6epUUB 8yorMCEu5KtAsou+iZ2DdYyTLheONyHKKrfsFDDC+o4YGuB7e7WJGLAWl35tDO+DX+rUwfYtn0n 3FwNXIWECyRBBvKhAMzIweUfxRGafkHSWeTTytMG0UNgaZpYq2U0bfznSI3YepjqUAmO7Xahois /rwN710Nt5/j+aSDEUI4s17Ql301mKiy1F9pgLupDIC9422DrTMQ/93vE8XUbaqUVvLFsiJysUd uYxggjRg+9g+UOo0pJTzjwz+GzorhAB5C8gZ6qKC6Im4I1RF8ogYxpm9PTbHnpmcLWmVc3bs2O+ aM5JK+BkVnaP0Bzb/76CM4Z/MGacSc/2/grB6CSxIFlMYKvCZvU6Giiro5pH1VvnAEaTR4DHqhK zS738v9n4VhRI1otzIlP3TIrDAENlHNJXmf5q X-Virus-Checked: Checked by ClamAV on apache.org --_000_BD42F346AE90F544A731516A805D1B8AD713C8SMAIL1prdmpacca_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Bertand I am reading my input by declaring it in the driver as follows: conf.setInputFormat(SequenceFileInputFormat.class); In the mapper: public void map(LongWritable key, SomeInterfaceClass value, OutputCollector output, = Reporter reporter) throws IOException { SomeInterfaceClass o =3D null; if ( value instanceof classA) { o =3D (classA)value; } if ( value instanceof classB) { o =3D (classB)value; } output.collect(new Text(o.get_property_id().toString()), o)= ; } // end of map() } // end of static class As u can see "o" can be an instance of classA or classB, where both impleme= nt SomeInerfaceClass. However, when I try to write in the driver: Conf.setOutputValueClass(SomeIterfaceClass.class); And run the job I get the following error Error: java.io.IOException: Type mismatch in value from map: expected SomeI= nterfaceClass, received ClassA I wished I could change the outputClass name depending on the instance of "= o" to either ClassA or ClassB. From: Bertrand Dechoux [mailto:dechouxb@gmail.com] Sent: Thursday, October 11, 2012 2:12 PM To: user@hadoop.apache.org Subject: Re: setOutputValueClass(*.class) method Hi Andy, It should be called within the setup of your job and it should NOT be calle= d within the Mapper. You could compile the call within the mapper of course. At best it should d= o nothing. I would say in practice the job will fail. But that's something = that you have to verify if you are curious. Back to your original problem, the mapper need to output the same kind of o= bject in order to be able to unserialize them for the mapper. (More general= ly the serialization strategy must be able to figure out what was serialize= d.) It does not mean your problem is not solvable but it means that you wil= l need to come up with a wrapper or a common structure. How do you read different kind of objects with the Mapper? Maybe you could = use the same approach. Regards Bertrand On Thu, Oct 11, 2012 at 7:49 PM, Kartashov, Andy > wrote: Guys, Is there another way to set the output from within the Mapper? My Mapper reads from various serialised files and generates different type = of objects for values depending on the returned value of instanceof. I want= ed to change the class name within the Mapper as opposed to the driver clas= s. Anyway of doing it? Thanks, Andy NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteur et peu= vent =EAtre couverts par le secret professionnel. Toute utilisation, copie = ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le desti= nataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l= 'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel -- Bertrand Dechoux NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteur et peu= vent =EAtre couverts par le secret professionnel. Toute utilisation, copie = ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le desti= nataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l= 'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD713C8SMAIL1prdmpacca_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Bertand

 

I am reading my input by= declaring it in the driver as follows:

 

conf.setInputFormat(Seq= uenceFileInputFormat.class);

 

 

In the mapper:

public void map(LongWri= table key, SomeInterfaceClass value,

    = ;            &n= bsp;       OutputCollector<Text, SomeInter= faceClass > output, Reporter reporter)

    = ;            &n= bsp;            &nbs= p;  throws IOException {

    = ;            SomeInt= erfaceClass o =3D null;

    = ;             &= nbsp;      if ( value instanceof classA) {

    = ;            &n= bsp;         o =3D (classA)value;

    &nbs= p;            &= nbsp;}

    = ;            &n= bsp;       if ( value instanceof classB) {

    = ;            &n= bsp;         o =3D (classB)value;

    = ;            &n= bsp; }

    = ;            output.= collect(new Text(o.get_property_id().toString()), o);

    = ;    } // end of map()

    = ; } // end of static class

 

As u can see “o&#= 8221; can be an instance of classA or classB, where both implement SomeIner= faceClass.

 

However, when I try to = write in the driver:

Conf.setOutputValueClas= s(SomeIterfaceClass.class);

And run the job I get t= he following error

 

Error: java.io.IOExcept= ion: Type mismatch in value from map: expected SomeInterfaceClass, received= ClassA

 

I wished I could change= the outputClass name depending on the instance of “o” to eithe= r ClassA or ClassB.

 

 

 

From: Bertrand Dechoux [mailto:dechouxb@gmail.com]
Sent: Thursday, October 11, 2012 2:12 PM
To: user@hadoop.apache.org
Subject: Re: setOutputValueClass(*.class) method

 

Hi Andy,

 

It should be called within the setup of your job and= it should NOT be called within the Mapper.

You could compile the call within the mapper of cour= se. At best it should do nothing. I would say in practice the job will fail= . But that's something that you have to verify if you are curious.

 

Back to your original problem, the mapper need to ou= tput the same kind of object in order to be able to unserialize them for th= e mapper. (More generally the serialization strategy must be able to figure= out what was serialized.) It does not mean your problem is not solvable but it means that you will need to c= ome up with a wrapper or a common structure.

 

How do you read different kind of objects with the M= apper? Maybe you could use the same approach.

 

Regards

 

Bertrand

On Thu, Oct 11, 2012 at 7:49 PM, Kartashov, Andy <= ;Andy.Kartashov= @mpac.ca> wrote:

Guys,=

 

Is there ano= ther way to set the output from within the Mapper?

 

My Mapper re= ads from various serialised files and generates different type of objects f= or values depending on the returned value of instanceof. I wanted to change the class name within the Mapper as opposed to the driver= class. Anyway of doing it?

 

Thanks,

Andy<= /p>

 

NOTICE: This e-mail message and any attachments are = confidential, subject to copyright and may be privileged. Any unauthorized = use, copying or disclosure is prohibited. If you are not the intended recip= ient, please delete and contact the sender immediately. Please consider the environment before printing this e= -mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagn= e sont confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre = couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n= '=EAtes pas le destinataire pr=E9vu de ce courriel, supprimez-le et contact= ez imm=E9diatement l'exp=E9diteur. Veuillez penser =E0 l'environnement avan= t d'imprimer le pr=E9sent courriel



 

--
Bertrand Dechoux

NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confidentiel= s, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts par le sec= ret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le dest= inataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement = l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD713C8SMAIL1prdmpacca_--