flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: RawSchema as deserialization schema
Date Mon, 26 Sep 2016 09:38:04 GMT
The RawSchema was once part of Flink's Kafka connector. I've removed it
because its implementation is trivial and I didn't expect that there are
many people who need the schema (also, I think I saw people using a map()
operator after the consumer to deserialize the byte[] into their formats).

As Stephan mentioned, I'm going to add some tooling so that people can
directly use Kafka's Serialization classes, and they provide a
"ByteArraySerializer" which is doing exactly the same as the "RawSchema"


On Mon, Sep 26, 2016 at 11:27 AM, Swapnil Chougule <the.swapnil09@gmail.com>
wrote:

> Hi Robert, May I know your inputs on same ?
>
> Thanks,
> Swapnil
>
> On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <sewen@apache.org> wrote:
>
>> /cc Robert, he is looking into extending the Kafka Connectors to support
>> more of Kafka's direct utilities
>>
>> On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <
>> the.swapnil09@gmail.com> wrote:
>>
>>> It will be good to have RawSchema as one of the deserialization schema
>>> in streaming framework (like SimpleStringSchema).
>>> Many use cases needs data in byte array format after reading from source
>>> like kafka.
>>>
>>> Any inputs for same ?
>>>
>>> On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <
>>> the.swapnil09@gmail.com> wrote:
>>>
>>>> Thanks Maximilian. I implemented same & it worked for me. I was under
>>>> impression that RawSchema is available from flink.
>>>>
>>>> Regards,
>>>> Swapnil
>>>>
>>>> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mxm@apache.org>
>>>> wrote:
>>>>
>>>>> Just implement DeserializationSchema and return the byte array from
>>>>> Kafka. Byte array serialization poses no problem to the Flink
>>>>> serialization.
>>>>>
>>>>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>>>>> <the.swapnil09@gmail.com> wrote:
>>>>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want
to
>>>>> read
>>>>> > byte array as it is in datastream. I tried to use RawSchema as
>>>>> > desrialization schema but couldn't find same 1.1.1.
>>>>> > I want to know whether I have to write my custom  implementation
for
>>>>> same ?
>>>>> > Can somebody help me to sort out same ?
>>>>> >
>>>>> > Also passing byte[] to next operator is supported as far as
>>>>> serialization is
>>>>> > concerned ?
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message