nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <philippe.gib...@orange.com>
Subject RE: Nifi kafka ser/deser problem
Date Tue, 05 Sep 2017 14:47:27 GMT
Hello Bryan,

Schema  Write Strategy :  Confluent Schema Registry Reference
Schema Access strategy :  Use 'Schema Name' Property 
Schema registry :  ConfluentSchemaRegistry 
Schema Name : topic7
Schema Text : ${avro.schema}
Compression Format : NONE

Thanks
Phil


-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com] 
Sent: mardi 5 septembre 2017 16:39
To: users@nifi.apache.org
Subject: Re: Nifi kafka ser/deser problem

Phil,

What is the "Schema Write Strategy" set to on the AvroRecordSetWriter?

-Bryan

On Tue, Sep 5, 2017 at 10:23 AM,  <philippe.gibert@orange.com> wrote:
> Hello Joe
>
> This  json content 
> {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","
> surface":223,"norme":"NF"}}
>
> is the input of the PublishKafkaRecord_0_10 processor. This processor have among others
 the following properties set :
>
> Record Reader : JsonTreeReader  --> pointing to the 
> ConfluentSchemaRegistry Record Writer :  AvroRecordSetWriter -> 
> pointing to the confluentSchemaRegistry
>
>
> I trace the call to  confluent registry and  I see the get  on the 
> subject  ( Confluent registry side) And I also  see  the  binary 
> content on a  consumer shell console (kafka-console-consumer )
> ---------------
> room
>            room1147.100,3.246▒NF
> -------------------------------
>
> Thanks
> Phil
>
> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: mardi 5 septembre 2017 15:26
> To: users@nifi.apache.org
> Subject: Re: Nifi kafka ser/deser problem
>
> Hello
>
> Can you share the details of how you're serializing the data?  We'll need to understand
the configuration of the record writer of the publish kafka process to help discuss about
reading it back (be that in nifi or otherwise).
>
> Since we support multiple modes of encoding the schema information this is the most likely
source of difficulty.
>
> Thanks
> Joe
>
> On Tue, Sep 5, 2017 at 8:53 AM,  <philippe.gibert@orange.com> wrote:
>> Hello, My SW environment  à   Nifi  1.4 compiled from sources , on Ubuntu
>> 14.04  and a running Confluent 3.3  Platform (running Confluent 
>> Registry and kafkaStreams )
>>
>> I have a process group with a PublishKafkaRecord_0_10 that serialize 
>> a record ( with 3 fields)
>>
>> And another process group  with a ConsumeKafkaRecord_0_10 that 
>> deserialise the record . that works nicely with reader/writer and  with the  Confluent
>> Registry   storing the Avro MetaData.
>>
>> I do not have  ser/deser problems because I think Nifi does both jobs 
>> ……
>>
>>
>>
>> But I got problems of deserialization  if I put some  java Kafka client in
>> the game … ( try to consume  from  Confluent   Kafka Streams )
>>
>> I got  some deserialization  exception :
>>
>> org.apache.kafka.common.errors.SerializationException: Error 
>> deserializing Avro message for id -1
>>
>>
>>
>> Any trick to  force another serialization  ( nifi side) or 
>> deserialization ( confluent side )  ?
>>
>>
>>
>> Best
>>
>> Phil
>>
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi 
>> que les pieces jointes. Les messages electroniques etant susceptibles 
>> d'alteration, Orange decline toute responsabilite si ce message a ete 
>> altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not 
>> be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender 
>> and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have 
>> been modified, changed or falsified.
>> Thank you.
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes.
Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite
si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed,
used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message
and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified,
changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees
et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par
erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant
susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may
be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message
and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed
or falsified.
Thank you.

Mime
View raw message