nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Jattiot <mjatt...@opensense.fr>
Subject Re: Issue with ValidateRecord processor and Avro map data type
Date Wed, 10 Oct 2018 16:08:27 GMT
I believe it's a bug too. Thank you Matt for opening the issue !

On Wed, Oct 10, 2018 at 5:15 PM Matt Burgess <mattyb149@apache.org> wrote:

> Maxime,
>
> I believe this is a bug, specifically at [1]. It should support
> MapRecord entries as well. I have written NIFI-5678 [2] to address
> this. The workaround is the one you and Bryan mentioned, using a
> record instead of a map.
>
> Regards,
> Matt
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java#L199
> [2] https://issues.apache.org/jira/browse/NIFI-5678
>
> On Wed, Oct 10, 2018 at 10:19 AM Maxime Jattiot <mjattiot@opensense.fr>
> wrote:
> >
> > Yes I made it worked with record type as well but I can't use a record
> > because my field names are dynamic.
> >
> > In many JSON we use collections of keys that will be named at runtime. I
> > thought the map data type of Avro was specially designed for it ....
> >
> > Any other ideas ?
> >
> >
> > On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende <bbende@gmail.com> wrote:
> >
> > > I was able to make it work using a nested record instead of a map....
> > >
> > > {
> > >     "type": "record",
> > >     "name": "example",
> > >     "fields" : [
> > >       {
> > >         "name": "field1",
> > >         "type": {
> > >           "name" : "field1",
> > >           "type" : "record",
> > >           "fields" : [
> > >             { "name": "toto", "type": "string" },
> > >             { "name": "titi", "type": "string" }
> > >           ]
> > >         }
> > >       }
> > >     ]
> > > }
> > > On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot <mjattiot@opensense.fr
> >
> > > wrote:
> > > >
> > > > Sure, here is the schema :
> > > > {
> > > >     "name" : "test",
> > > >     "type" : "record",
> > > >     "fields" : [ {
> > > >         "name" : "field1",
> > > >         "type" : {
> > > >             "type" : "map",
> > > >             "values" : "string"
> > > >         }
> > > >     } ]
> > > > }
> > > >
> > > > And here is the JSON message :
> > > > [{
> > > >     "field1": {
> > > >         "toto" : "v1",
> > > >         "titi" : "v2"
> > > >     }
> > > > }]
> > > >
> > > > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <bbende@gmail.com>
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > The screenshots did not come through, maybe just paste the schema
> and
> > > JSON
> > > > > as text in the email.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <
> mjattiot@opensense.fr>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I am having hard time to validate a simple JSON message with
an
> Avro
> > > > > > schema. My schema works with avro-tools but doesn't with
> > > ValidateRecord
> > > > > > processor. I wish you could help me.
> > > > > >
> > > > > >    - I am having the following debug log within Nifi :
> > > > > >
> > > > > > Record #1 is invalid due to:
> > > > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> > > Value is
> > > > > > of type org.apache.nifi.serialization.record.MapRecord but was
> > > expected
> > > > > to
> > > > > > be of type MAP[STRING]
> > > > > >
> > > > > >
> > > > > >    - Here is my schema :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > > > >
> > > > > >    - Here is the JSON content sent :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > > > >
> > > > > > Does someone has any clue ? I have been struggling 2 days with
> this
> > > map
> > > > > > data type ... :(
> > > > > >
> > > > > > Thank you !
> > > > > >
> > > > > > Maxime
> > > > > >
> > > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message