avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Jain <ankur.j...@yash.com>
Subject RE: AVRO MultipleOutput
Date Tue, 13 Jan 2015 06:47:02 GMT
Thanks Deepak,

I tried using below but that also not helping in changing output file name…

In Job file
MultipleOutputs.addNamedOutput(job, "clean", AvroKeyOutputFormat.class, Measurement.class
, NullWritable.class);

In Reducer
Setup:
MultipleOutputs<Measurement, NullWritable> mos;

       @Override
       protected void setup(Context context) throws IOException, InterruptedException {
              mos = new MultipleOutputs<Measurement, NullWritable>(context);
       }

And in reducer method
if (isClean) {
              mos.write("clean", new Text("clean"), value.datum(), "clean");
       } else {
              mos.write("discarded", new Text("discarded"), value.datum(), "discarded");
       }


Can you please point on the what I am doing wrong?

Thank you
Ankur



From: Deepak Sharma [mailto:deepakmca05@gmail.com]
Sent: Tuesday, January 13, 2015 11:38 AM
To: user; Ankur Jain
Subject: Re: AVRO MultipleOutput

Hi Ankur
Not sure on the avro part , but MR provides addNamedOutputs() that can be set in Driver class
to rename the output files from part-*** to something else.
HTH.
Thanks
Deepak

On Tue, Jan 13, 2015 at 11:29 AM, Ankur Jain <ankur.jain@yash.com<mailto:ankur.jain@yash.com>>
wrote:
Hi Team,

My usecase is to create files with different names.
I tried using AVRO mapreduce api for MultipleOutputs using org.apache.avro.mapreduce.AvroMultipleOutputs.
I followed Javadoc mentioned for this class and found below issues…


1.       We cannot assign AvroMultipleOutputs object to MultipleOutputs.

2.       Even if I followed all steps mentioned in Javadoc I am still getting file with name
part-0000

Can someone please guide me on how I can create files with different name using AVRO mapreduce
api?
Any working example will be great help….

I am using HDP 2.2 , avro (1.7.4), JDK 1.7, reading and writing in avro files.

Thanks and Best Regards,
Ankur
Information transmitted by this e-mail is proprietary to YASH Technologies and/ or its Customers
and is intended for use only by the individual or entity to which it is addressed, and may
contain information that is privileged, confidential or exempt from disclosure under applicable
law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information
in any manner is strictly prohibited. In such cases, please notify us immediately at info@yash.com<mailto:info@yash.com>
and delete this mail from your records.



--
Thanks
Deepak
www.bigdatabig.com<http://www.bigdatabig.com>
www.keosha.net<http://www.keosha.net>
Information transmitted by this e-mail is proprietary to YASH Technologies and/ or its Customers
and is intended for use only by the individual or entity to which it is addressed, and may
contain information that is privileged, confidential or exempt from disclosure under applicable
law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information
in any manner is strictly prohibited. In such cases, please notify us immediately at info@yash.com
and delete this mail from your records.
Mime
View raw message