avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maulik Gandhi <mmg...@gmail.com>
Subject Re: Not able to resolve union for array type
Date Mon, 06 Jun 2016 13:17:09 GMT
Hi Giri,

Can you share a simplified implementation code and the model you are trying
to emit.

Thanks.
- Maulik

On Sat, Jun 4, 2016 at 3:04 PM, Giri P <gpatcham@gmail.com> wrote:

> I need the other way either null or array of records. I was able to insert
> not null records but when I try inserting nulls it throws me error
>
> I'm using spark to write avro files
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Failed
> to serialize task 16, not attempting to retry it. Exception during
> serialization: java.io.NotSerializableException:
> org.apache.avro.Schema$UnionSchema
> Serialization stack:
>         - object not serializable (class:
> org.apache.avro.Schema$UnionSchema, value:
> ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute
> id"},{"name":"updateDate","type":"long","doc":"Update date of the
> attribute"},{"name":"value","type":"string","doc":"Value of the
> attribute"}]}}])
>         - writeObject data (class: java.lang.Throwable)
>         - object (class org.apache.avro.UnresolvedUnionException,
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute
> id"},{"name":"updateDate","type":"long","doc":"Update date of the
> attribute"},{"name":"value","type":"string","doc":"Value of the
> attribute"}]}}]: )
>         - writeObject data (class: java.lang.Throwable)
>         - object (class java.io.IOException, java.io.IOException:
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"array","items":{"type":"record","name":"MatchedAttrTest","namespace":"com.conversantmedia.data.cp.avro","fields":[{"name":"id","type":"long","doc":"Attribute
> id"},{"name":"updateDate","type":"long","doc":"Update date of the
> attribute"},{"name":"value","type":"string","doc":"Value of the
> attribute"}]}}]: )
>         - writeObject data (class:
> org.apache.spark.rdd.ParallelCollectionPartition)
>         - object (class org.apache.spark.rdd.ParallelCollectionPartition,
> org.apache.spark.rdd.ParallelCollectionPartition@7b0)
>         - element of array (index: 0)
>         - array (class [Ljava.lang.Object;, size 10)
>         - field (class: scala.collection.mutable.ArraySeq, name: array,
> type: class [Ljava.lang.Object;)
>         - object (class scala.collection.mutable.ArraySeq,
> ArraySeq(org.apache.spark.rdd.ParallelCollectionPartition@7b0,
> org.apache.spark.rdd.ParallelCollectionPartition@7b1,
> org.apache.spark.rdd.ParallelCollectionPartition@7b2,
> org.apache.spark.rdd.ParallelCollectionPartition@7b3,
> org.apache.spark.rdd.ParallelCollectionPartition@7b4,
> org.apache.spark.rdd.ParallelCollectionPartition@7b5,
> org.apache.spark.rdd.ParallelCollectionPartition@7b6,
> org.apache.spark.rdd.ParallelCollectionPartition@7b7,
> org.apache.spark.rdd.ParallelCollectionPartition@7b8,
> org.apache.spark.rdd.ParallelCollectionPartition@7b9))
>         - writeObject data (class:
> org.apache.spark.rdd.CoalescedRDDPartition)
>         - object (class org.apache.spark.rdd.CoalescedRDDPartition,
> CoalescedRDDPartition(0,MapPartitionsRDD[8] at map at
> <console>:81,[I@19cc3c95,None))
>         - field (class: org.apache.spark.scheduler.ResultTask, name:
> partition, type: interface org.apache.spark.Partition)
>         - object (class org.apache.spark.scheduler.ResultTask,
> ResultTask(4, 0))
>
>
> On Fri, Jun 3, 2016 at 10:53 PM, Doug Cutting <cutting@gmail.com> wrote:
>
>> Your schema permits null or an array of records. I suspect you want an
>> array containing nulls or records, e.g.,
>>
>> {"type":"array","items":["null",{"type":"record"...
>> On Jun 3, 2016 5:54 PM, "Giri P" <gpatcham@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm getting below error when I try to insert null into union of array
>>>
>>> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
>>> ["null",{"type":"array","items":{"type":"record","name":"DeMatchedAttr","namespace":"cp","fields":[{"name":"id","type":"long","doc":"Attribute
>>> id"},{"name":"updateDate","type":"long","doc":"Update date of the
>>> attribute"},{"name":"value","type":"string","doc":"Value of the
>>> attribute"}]}}]: null
>>>
>>> Is there any issue with the schema ?
>>>
>>> Thanks for the help
>>>
>>> -Giri
>>>
>>
>

Mime
View raw message