nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dirk Arends (Jira)" <j...@apache.org>
Subject [jira] [Updated] (NIFI-7601) QueryRecord failing to handle arrays in 1.11.4
Date Mon, 06 Jul 2020 07:25:00 GMT

     [ https://issues.apache.org/jira/browse/NIFI-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dirk Arends updated NIFI-7601:
------------------------------
    Description: 
I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not an issue on
1.11.0. If a record contains arrays of “complex” data I am not able to perform any queries
on the record, even if the queries don’t require data from within the array.
  
{code:java}
2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, section=1], offset=51521,
length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to java.lang.IllegalStateException:
Unhandled ARRAY component type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled
ARRAY component type: OBJECT, id: 1111
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60)
at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
[https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]

 

I believe this to be a regression, possibly introduced by this commit (though I haven't been
able to confirm):

 

[https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]

https://issues.apache.org/jira/browse/NIFI-7095

 

Example 1: Succeeds when performing a query in a QueryRecord

 

Data:
{code:java}
{"total":5,"payload":["Test"]}{code}
 

Schema:
{code:java}
{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
  {"name":"total","type":"int"},
  {"name":"payload","type":[
    {"type":"array","items":"string"}
  ]}
]}
{code}
 

Example 2: Fails when performing a query in a QueryRecord

 

Data:
{code:java}
{"total":5,"payload":[{"type":"Test"}]}{code}
 

Schema:
{code:java}
{"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[
  {"name":"total","type":"int"},
  {"name":"payload","type":[
    "null",
    {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
      {"name":"type","type":"string"}
    ]}}
  ]}
]}
{code}
 

I have set up a simple flow.xml which demonstrates the issue. It is reproducible with a fresh
copy of the convenience binaries, with no special steps.

  was:
I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not an issue on
1.11.0. If a record contains arrays of “complex” data I am not able to perform any queries
on the record, even if the queries don’t require data from within the array.
 
{code:java}
2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, section=1], offset=51521,
length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to java.lang.IllegalStateException:
Unhandled ARRAY component type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled
ARRAY component type: OBJECT, id: 1111
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60)
at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
[https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]

 

I believe this to be a regression, possibly introduced by this commit (though I haven't been
able to confirm):

 

[https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]

https://issues.apache.org/jira/browse/NIFI-7095

 

Example 1: Succeeds when performing a query in a QueryRecord

 

Data: 
{code:java}
{"total":5,"payload":["Test"]}{code}
 

 

Schema:

 
{code:java}
{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
  {"name":"total","type":"int"},
  {"name":"payload","type":[
    {"type":"array","items":"string"}
  ]}
]}
{code}
 

 

Example 2: Fails when performing a query in a QueryRecord

 

Data:
{code:java}
{"total":5,"payload":[{"type":"Test"}]}{code}
 

Schema:
{code:java}
{"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[
  {"name":"total","type":"int"},
  {"name":"payload","type":[
    "null",
    {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
      {"name":"type","type":"string"}
    ]}}
  ]}
]}
{code}
 

I have set up a simple flow.xml which demonstrates the issue. It is reproducible with a fresh
copy of the convenience binaries, with no special steps.


> QueryRecord failing to handle arrays in 1.11.4
> ----------------------------------------------
>
>                 Key: NIFI-7601
>                 URL: https://issues.apache.org/jira/browse/NIFI-7601
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 1.11.4
>            Reporter: Dirk Arends
>            Priority: Major
>         Attachments: flow.xml
>
>
> I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not an issue
on 1.11.0. If a record contains arrays of “complex” data I am not able to perform any
queries on the record, even if the queries don’t require data from within the array.
>   
> {code:java}
> 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, section=1], offset=51521,
length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to java.lang.IllegalStateException:
Unhandled ARRAY component type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled
ARRAY component type: OBJECT, id: 1111
> java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111
> at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
> at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
> at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
> at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
> at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
> at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
> at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60)
> at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
> at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
> at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]
>  
> I believe this to be a regression, possibly introduced by this commit (though I haven't
been able to confirm):
>  
> [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]
> https://issues.apache.org/jira/browse/NIFI-7095
>  
> Example 1: Succeeds when performing a query in a QueryRecord
>  
> Data:
> {code:java}
> {"total":5,"payload":["Test"]}{code}
>  
> Schema:
> {code:java}
> {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
>   {"name":"total","type":"int"},
>   {"name":"payload","type":[
>     {"type":"array","items":"string"}
>   ]}
> ]}
> {code}
>  
> Example 2: Fails when performing a query in a QueryRecord
>  
> Data:
> {code:java}
> {"total":5,"payload":[{"type":"Test"}]}{code}
>  
> Schema:
> {code:java}
> {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[
>   {"name":"total","type":"int"},
>   {"name":"payload","type":[
>     "null",
>     {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
>       {"name":"type","type":"string"}
>     ]}}
>   ]}
> ]}
> {code}
>  
> I have set up a simple flow.xml which demonstrates the issue. It is reproducible with
a fresh copy of the convenience binaries, with no special steps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message