flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Fwd: Re: AVRO Union type support in Flink
Date Wed, 19 Jul 2017 14:57:16 GMT
We have similar checks in our KafkaAvroTableSource, but I could not find 
such a check in AvroTypeInfo. The field should have a generic type, so 
you can work with it. If you want to use it as key, you might have to 
use a mapper before and convert it into a valid key type.

Timo



-------- Weitergeleitete Nachricht --------
Betreff: 	Re: AVRO Union type support in Flink
Datum: 	Wed, 19 Jul 2017 10:26:24 -0400
Von: 	Vishnu Viswanath <vishnu.viswanath25@gmail.com>
An: 	Timo Walther <twalthr@apache.org>



Hi Timo,

Thanks for checking that. I did not try yet. My current application uses 
Cascading and it has the limitation that Union cannot contain two 
concrete types - link 
<https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>,

so was wondering if I can use Flink. Will give it a try.

Hi Martin,
The documentation is here 
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink>
I use it to create AVRO files from source files in S3 and write to Kafka.

Thanks,
Vishnu


On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <twalthr@apache.org 
<mailto:twalthr@apache.org>> wrote:

    Hi Vishnu,

    I took a look into the code. Actually, we should support it.
    However, those types might be mapped to Java Objects that will be
    serialized with our generic Kryo serializer. Have you tested it?

    Regards,
    Timo


    Am 19.07.17 um 06:30 schrieb Martin Eden:
>     Hey Vishnu,
>
>     For those of us on the list that are not very familiar with Flink
>     and Avro can you give a pointed to the docs you are referring to
>     and how you intend to use it? Just so we gain understanding as well.
>
>     Thanks,
>     Martin
>
>     On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath
>     <vishnu.viswanath25@gmail.com
>     <mailto:vishnu.viswanath25@gmail.com>> wrote:
>
>         Hi All,
>
>         Does Flink support AVRO union types - Documentation says it
>         supports nullable types: {"name": "type_double_test", "type":
>         ["null", "double"]}
>
>         But my schema has something like : {"name": "union_field",
>         "type": ["string", "double"]}
>
>         Thanks
>         Vishnu
>
>
>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message