nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Floran Stuijt (Jira)" <j...@apache.org>
Subject [jira] [Updated] (NIFI-6918) Impossible to fill AVRO fixed type from JSON
Date Fri, 29 Nov 2019 21:22:00 GMT

     [ https://issues.apache.org/jira/browse/NIFI-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Floran Stuijt updated NIFI-6918:
--------------------------------
    Description: 
Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:

 
{noformat}
{
   "type":"record",
   "name":"Message",
   "namespace":"com.example",
   "fields":[
      {
         "name":"MAC",
         "type":{
            "type":"fixed",
            "name":"MY_FIXED_TYPE",
            "size":6
         }
      }
   ]
}{noformat}
 

The input JSON-forms I tried are:

 
{noformat}
{ "MAC": [ 0, 1, 2, 3, 4, 5] }

{ "MAC": "012345" }

{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}

{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
{noformat}
 

Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead
of the ConvertJsonToAvro-processor. Also without any luck.
  
 It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_*
be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
 # The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro
conversion is _not_ the same as Avro JSON encoding from the specification.
 # This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
 # {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}} is
true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
 # As far as I can tell, parsing a JSON string with Jackson _never_ generates such a JSON
node.

  was:
Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:
 
{{{"type" : "record","name" : "Message","namespace" : "com.example","fields" : [ {"name" :
"MAC","type" : {"type" : "fixed","name" : "MY_FIXED_TYPE","size" : 6}}]}}}

The input JSON-forms I tried are
 
{{{ "MAC": [ 0, 1, 2, 3, 4, 5] }{ "MAC": "012345" }{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}{"MAC":{"MY_FIXED_TYPE":
"\u0000\u0001\u0002\u0003\u0004\u0005"}}}}

Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead
of the ConvertJsonToAvro-processor. Also without any luck.
 
It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_*
be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
 # The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro
conversion is _not_ the same as Avro JSON encoding from the specification.
 # This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
 # {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}} is
true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
 # As far as I can tell, parsing a JSON string with Jackson _never_ generates such a JSON
node.


> Impossible to fill AVRO fixed type from JSON
> --------------------------------------------
>
>                 Key: NIFI-6918
>                 URL: https://issues.apache.org/jira/browse/NIFI-6918
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 1.9.2
>            Reporter: Floran Stuijt
>            Priority: Major
>
> Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:
>  
> {noformat}
> {
>    "type":"record",
>    "name":"Message",
>    "namespace":"com.example",
>    "fields":[
>       {
>          "name":"MAC",
>          "type":{
>             "type":"fixed",
>             "name":"MY_FIXED_TYPE",
>             "size":6
>          }
>       }
>    ]
> }{noformat}
>  
> The input JSON-forms I tried are:
>  
> {noformat}
> { "MAC": [ 0, 1, 2, 3, 4, 5] }
> { "MAC": "012345" }
> {"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}
> {"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
> {noformat}
>  
> Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead
of the ConvertJsonToAvro-processor. Also without any luck.
>   
>  It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_*
be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
>  # The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This
JSON-to-Avro conversion is _not_ the same as Avro JSON encoding from the specification.
>  # This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
>  # {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}}
is true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
>  # As far as I can tell, parsing a JSON string with Jackson _never_ generates such a
JSON node.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message