avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Максим Стаценко <m.statse...@corp.mail.ru>
Subject Problems with writing avro using map reduce.
Date Wed, 24 Aug 2016 15:38:00 GMT

Hello!
Thank you for your software! I have one problem with my attempts to write avrofile using mapreduce.
Can you help me, please. I asked my friends and stackoverflow, but nobody can find an answer.
On the input I have file with data for some hive tables. And on the output of the MapReduce
I want to have some groups of avro files with tables schemes for each table. I'm writing data
to Generic record and after that I'm trying to write it with AvroMultipleOutputs.

But I get a very strange error from reducer part :
>16/08/23 16:08:08 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_0,
Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
16/08/23 16:08:15 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_1, Status
: FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
16/08/23 16:08:23 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_2, Status
: FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
I simplified my code in attach

My input file:
1234567
12345566
0987654321
 
I add link to stackoverflow, because there is a good highlighting in it.
http://stackoverflow.com/questions/39102348/writing-genericrecord-to-multiple-avro-output
 

Thank you!
Max Statsenko
----------------------------------------------------------------------
Mime
View raw message