From issues-return-99655-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Mon Jul 6 07:25:05 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8F3E5180652 for ; Mon, 6 Jul 2020 09:25:05 +0200 (CEST) Received: (qmail 3644 invoked by uid 500); 6 Jul 2020 07:25:01 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 3580 invoked by uid 99); 6 Jul 2020 07:25:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2020 07:25:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id AAD0041074 for ; Mon, 6 Jul 2020 07:25:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 1E8D87802B1 for ; Mon, 6 Jul 2020 07:25:00 +0000 (UTC) Date: Mon, 6 Jul 2020 07:25:00 +0000 (UTC) From: "Dirk Arends (Jira)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (NIFI-7601) QueryRecord failing to handle arrays in 1.11.4 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-7601?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-7601: ------------------------------ Description:=20 I am running into an issue QueryRecord processors on Nifi 1.11.4 which was = not an issue on 1.11.0. If a record contains arrays of =E2=80=9Ccomplex=E2= =80=9D data I am not able to perform any queries on the record, even if the= queries don=E2=80=99t require data from within the array. =C2=A0 {code:java} 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.proc= essors.standard.QueryRecord QueryRecord[id=3D01731003-15bb-12a6-7e03-8c418a= b7fb97] Unable to query StandardFlowFileRecord[uuid=3D04ed077b-0556-4dda-83= 6e-62c5f9a5c772,claim=3DStandardContentClaim [resourceClaim=3DStandardResou= rceClaim[id=3D1594014641476-1, container=3Ddefault, section=3D1], offset=3D= 51521, length=3D334],offset=3D0,name=3D04ed077b-0556-4dda-836e-62c5f9a5c772= ,size=3D334] due to java.lang.IllegalStateException: Unhandled ARRAY compon= ent type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled ARRA= Y component type: OBJECT, id: 1111 java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id= : 1111 at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValu= e(AbstractCursor.java:1346) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(A= bstractCursor.java:1299) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(Ab= stractCursor.java:1352) at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.ja= va:729) at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(Resu= ltSetRecordSet.java:186) at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(Res= ultSetRecordSet.java:156) at org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultSet= RecordSet.java:60) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.ja= va:332) at org.apache.nifi.controller.repository.StandardProcessSession.write(Stand= ardProcessSession.java:2746) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.ja= va:325) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.= java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProce= ssorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.= java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(T= imerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:30= 5) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) {code} [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/o= rg/apache/calcite/avatica/util/AbstractCursor.java#L1346] =C2=A0 I believe this to be a regression, possibly introduced by this commit (thou= gh I haven't been able to confirm): =C2=A0 [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077= d297] https://issues.apache.org/jira/browse/NIFI-7095 =C2=A0 Example 1: Succeeds when performing a query in a QueryRecord =C2=A0 Data: {code:java} {"total":5,"payload":["Test"]}{code} =C2=A0 Schema: {code:java} {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fi= elds":[ =C2=A0=C2=A0{"name":"total","type":"int"}, =C2=A0=C2=A0{"name":"payload","type":[ =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":"string"} =C2=A0=C2=A0]} ]} {code} =C2=A0 Example 2: Fails when performing a query in a QueryRecord =C2=A0 Data: {code:java} {"total":5,"payload":[{"type":"Test"}]}{code} =C2=A0 Schema: {code:java} {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields= ":[ =C2=A0=C2=A0{"name":"total","type":"int"}, =C2=A0=C2=A0{"name":"payload","type":[ =C2=A0=C2=A0=C2=A0=C2=A0"null", =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":{"type":"record","name":"Ap= iResponseData","namespace":"com.example.api","fields":[ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{"name":"type","type":"string"} =C2=A0=C2=A0=C2=A0=C2=A0]}} =C2=A0=C2=A0]} ]} {code} =C2=A0 I have set up a simple flow.xml which demonstrates the issue. It is reprodu= cible with a fresh copy of the convenience binaries, with no special steps. was: I am running into an issue QueryRecord processors on Nifi 1.11.4 which was = not an issue on 1.11.0. If a record contains arrays of =E2=80=9Ccomplex=E2= =80=9D data I am not able to perform any queries on the record, even if the= queries don=E2=80=99t require data from within the array. =C2=A0 {code:java} 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.proc= essors.standard.QueryRecord QueryRecord[id=3D01731003-15bb-12a6-7e03-8c418a= b7fb97] Unable to query StandardFlowFileRecord[uuid=3D04ed077b-0556-4dda-83= 6e-62c5f9a5c772,claim=3DStandardContentClaim [resourceClaim=3DStandardResou= rceClaim[id=3D1594014641476-1, container=3Ddefault, section=3D1], offset=3D= 51521, length=3D334],offset=3D0,name=3D04ed077b-0556-4dda-836e-62c5f9a5c772= ,size=3D334] due to java.lang.IllegalStateException: Unhandled ARRAY compon= ent type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled ARRA= Y component type: OBJECT, id: 1111 java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id= : 1111 at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValu= e(AbstractCursor.java:1346) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(A= bstractCursor.java:1299) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(Ab= stractCursor.java:1352) at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.ja= va:729) at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(Resu= ltSetRecordSet.java:186) at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(Res= ultSetRecordSet.java:156) at org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultSet= RecordSet.java:60) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.ja= va:332) at org.apache.nifi.controller.repository.StandardProcessSession.write(Stand= ardProcessSession.java:2746) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.ja= va:325) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.= java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProce= ssorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.= java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(T= imerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:30= 5) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) {code} [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/o= rg/apache/calcite/avatica/util/AbstractCursor.java#L1346] =C2=A0 I believe this to be a regression, possibly introduced by this commit (thou= gh I haven't been able to confirm): =C2=A0 [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077= d297] https://issues.apache.org/jira/browse/NIFI-7095 =C2=A0 Example 1: Succeeds when performing a query in a QueryRecord =C2=A0 Data:=20 {code:java} {"total":5,"payload":["Test"]}{code} =C2=A0 =C2=A0 Schema: =C2=A0 {code:java} {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fi= elds":[ =C2=A0=C2=A0{"name":"total","type":"int"}, =C2=A0=C2=A0{"name":"payload","type":[ =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":"string"} =C2=A0=C2=A0]} ]} {code} =C2=A0 =C2=A0 Example 2: Fails when performing a query in a QueryRecord =C2=A0 Data: {code:java} {"total":5,"payload":[{"type":"Test"}]}{code} =C2=A0 Schema: {code:java} {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields= ":[ =C2=A0=C2=A0{"name":"total","type":"int"}, =C2=A0=C2=A0{"name":"payload","type":[ =C2=A0=C2=A0=C2=A0=C2=A0"null", =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":{"type":"record","name":"Ap= iResponseData","namespace":"com.example.api","fields":[ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{"name":"type","type":"string"} =C2=A0=C2=A0=C2=A0=C2=A0]}} =C2=A0=C2=A0]} ]} {code} =C2=A0 I have set up a simple flow.xml which demonstrates the issue. It is reprodu= cible with a fresh copy of the convenience binaries, with no special steps. > QueryRecord failing to handle arrays in 1.11.4 > ---------------------------------------------- > > Key: NIFI-7601 > URL: https://issues.apache.org/jira/browse/NIFI-7601 > Project: Apache NiFi > Issue Type: Bug > Affects Versions: 1.11.4 > Reporter: Dirk Arends > Priority: Major > Attachments: flow.xml > > > I am running into an issue QueryRecord processors on Nifi 1.11.4 which wa= s not an issue on 1.11.0. If a record contains arrays of =E2=80=9Ccomplex= =E2=80=9D data I am not able to perform any queries on the record, even if = the queries don=E2=80=99t require data from within the array. > =C2=A0 > {code:java} > 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.pr= ocessors.standard.QueryRecord QueryRecord[id=3D01731003-15bb-12a6-7e03-8c41= 8ab7fb97] Unable to query StandardFlowFileRecord[uuid=3D04ed077b-0556-4dda-= 836e-62c5f9a5c772,claim=3DStandardContentClaim [resourceClaim=3DStandardRes= ourceClaim[id=3D1594014641476-1, container=3Ddefault, section=3D1], offset= =3D51521, length=3D334],offset=3D0,name=3D04ed077b-0556-4dda-836e-62c5f9a5c= 772,size=3D334] due to java.lang.IllegalStateException: Unhandled ARRAY com= ponent type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled A= RRAY component type: OBJECT, id: 1111 > java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, = id: 1111 > at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertVa= lue(AbstractCursor.java:1346) > at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject= (AbstractCursor.java:1299) > at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(= AbstractCursor.java:1352) > at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.= java:729) > at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(Re= sultSetRecordSet.java:186) > at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(R= esultSetRecordSet.java:156) > at org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultS= etRecordSet.java:60) > at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.= java:332) > at org.apache.nifi.controller.repository.StandardProcessSession.write(Sta= ndardProcessSession.java:2746) > at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.= java:325) > at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcesso= r.java:27) > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardPro= cessorNode.java:1176) > at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTas= k.java:213) > at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run= (TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor= s.java:515) > at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:= 305) > at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu= tureTask.run(ScheduledThreadPoolExecutor.java:305) > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool= Executor.java:1128) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo= lExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > {code} > [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java= /org/apache/calcite/avatica/util/AbstractCursor.java#L1346] > =C2=A0 > I believe this to be a regression, possibly introduced by this commit (th= ough I haven't been able to confirm): > =C2=A0 > [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f631077030= 77d297] > https://issues.apache.org/jira/browse/NIFI-7095 > =C2=A0 > Example 1: Succeeds when performing a query in a QueryRecord > =C2=A0 > Data: > {code:java} > {"total":5,"payload":["Test"]}{code} > =C2=A0 > Schema: > {code:java} > {"type":"record","name":"ApiResponseData","namespace":"com.example.api","= fields":[ > =C2=A0=C2=A0{"name":"total","type":"int"}, > =C2=A0=C2=A0{"name":"payload","type":[ > =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":"string"} > =C2=A0=C2=A0]} > ]} > {code} > =C2=A0 > Example 2: Fails when performing a query in a QueryRecord > =C2=A0 > Data: > {code:java} > {"total":5,"payload":[{"type":"Test"}]}{code} > =C2=A0 > Schema: > {code:java} > {"type":"record","name":"ApiResponse","namespace":"com.example.api","fiel= ds":[ > =C2=A0=C2=A0{"name":"total","type":"int"}, > =C2=A0=C2=A0{"name":"payload","type":[ > =C2=A0=C2=A0=C2=A0=C2=A0"null", > =C2=A0=C2=A0=C2=A0=C2=A0{"type":"array","items":{"type":"record","name":"= ApiResponseData","namespace":"com.example.api","fields":[ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{"name":"type","type":"string"} > =C2=A0=C2=A0=C2=A0=C2=A0]}} > =C2=A0=C2=A0]} > ]} > {code} > =C2=A0 > I have set up a simple flow.xml which demonstrates the issue. It is repro= ducible with a fresh copy of the convenience binaries, with no special step= s. -- This message was sent by Atlassian Jira (v8.3.4#803005)