Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A3C5E18487 for ; Mon, 20 Jul 2015 22:56:54 +0000 (UTC) Received: (qmail 43348 invoked by uid 500); 20 Jul 2015 22:56:49 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 43217 invoked by uid 500); 20 Jul 2015 22:56:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 43207 invoked by uid 99); 20 Jul 2015 22:56:49 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2015 22:56:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id BEAE218A10E for ; Mon, 20 Jul 2015 22:56:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.041 X-Spam-Level: ** X-Spam-Status: No, score=2.041 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-1.108, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id PizieHFSpb36 for ; Mon, 20 Jul 2015 22:56:47 +0000 (UTC) Received: from mail-ie0-f174.google.com (mail-ie0-f174.google.com [209.85.223.174]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 2A8FA20D14 for ; Mon, 20 Jul 2015 22:56:47 +0000 (UTC) Received: by ietj16 with SMTP id j16so128800966iet.0 for ; Mon, 20 Jul 2015 15:56:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=RZ2v9uz6mjOKOL12/yC5FKhjznbSM16wuLZqWLaef+M=; b=KK7rAr7GFbIbIAQGsWEXQC1ARxIQGNZku0UY+I98Cgxwkq9tTeTKe+75LrWBTi94GV 44/EXSJUlvurbOu9HAheq21bbPR37eobeHurR7bMVJ8U1vcQ00MIR5khxlPDgrcOOx4a BQnCijhxgYfowe5T/YJ2ZRESt8UmHJORDHiy9YdQx9ldyTe11VXyKLSMkZkwjfEob9zy Az5V9X42yGtK1Dfa3LgWlbMd64a1Dlpn1Hh7NpE69B3NlcnQTLb94uGsN5nTt9a1A1cj 9XNhzpfCdvUAKEUtS/G8DwxV5x5hFtvoAl9RU6x5wEfIdEAph6fkyUWB5uNCSA26b+JF cvcA== MIME-Version: 1.0 X-Received: by 10.50.103.73 with SMTP id fu9mr19027782igb.46.1437432961016; Mon, 20 Jul 2015 15:56:01 -0700 (PDT) Received: by 10.107.170.223 with HTTP; Mon, 20 Jul 2015 15:56:00 -0700 (PDT) In-Reply-To: References: Date: Mon, 20 Jul 2015 16:56:00 -0600 Message-ID: Subject: Fwd: Avro Map Reduce for Multiple Schemas From: Nishanth S To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b2e12d307af48051b5671a3 --047d7b2e12d307af48051b5671a3 Content-Type: text/plain; charset=UTF-8 Hello, I have to output multiple avro files with different schemas as the output of a mapreduce job.Currently I am achieving this by doing a union of all the schemas in the driver and then by using Avromultipleoutputs to output two files. AvroMultipleOutputs.addNamedOutput(job, "a", AvroKeyValueOutputFormat.class, Schema.create(Schema.Type.NULL),A.getClassSchema()); AvroMultipleOutputs.addNamedOutput(job, "b", AvroKeyValueOutputFormat.class, Schema.create(Schema.Type.NULL),B.getClassSchema()); List schemas = new ArrayList(); schemas.add(C.getClassSchema()); schemas.add(D.getClassSchema()); AvroKeyValueOutputFormat.class, Schema.create(Schema.Type.NULL),A.getClassSchema()); AvroJob.setOutputKeySchema(job, Schema.create(Schema.Type.NULL)); AvroJob.setOutputValueSchema(job,B.getClassSchema().createUnion(schemas) ); Is there a better way to do this?.Request help. Thanks, Nishan --047d7b2e12d307af48051b5671a3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hello,
I have to output multiple avro =C2=A0files with different s= chemas as the =C2=A0output of a =C2=A0mapreduce =C2=A0job.Currently I am ac= hieving this by doing a union of all the schemas in the driver and then =C2= =A0by using Avromultipleoutputs =C2=A0to =C2=A0output two files.
=

AvroMultipleOutputs.addNamedOutput(job, = "a",
Av= roKeyValueOutputFormat.class, =C2=A0Schema.create(Schema.Type.NULL),A.getCl= assSchema());
=C2=A0 =C2=A0 =C2=A0 =C2=A0 AvroMultipleOutputs.add= NamedOutput(job, "b",
AvroKeyValueOutputFormat.class, =C2=A0Schema.create(Schema.T= ype.NULL),B.getClassSchema());
List<Schema> sche= mas =3D new ArrayList<Schema>();
schemas.add(C.getClassSchema());
schemas.add(D.getClassSchema());
AvroKeyValueOut= putFormat.class, =C2=A0 =C2=A0 =C2=A0 =C2=A0Schema.create(Schema.Type.NULL)= ,A.getClassSchema());
=C2=A0 =C2=A0 =C2=A0 =C2=A0 AvroJob.setOutp= utKeySchema(job, Schema.create(Schema.Type.NULL));
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 AvroJob.setOutputValueSchema(job,B.getClassSchema().createUni= on(schemas) );

Is there a better way to do t= his?.Request help.

Thanks,
Nishan
<= /div>

--047d7b2e12d307af48051b5671a3--