Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4AA55181C1 for ; Wed, 6 May 2015 21:53:00 +0000 (UTC) Received: (qmail 9681 invoked by uid 500); 6 May 2015 21:53:00 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 9652 invoked by uid 500); 6 May 2015 21:53:00 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 9642 invoked by uid 99); 6 May 2015 21:53:00 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2015 21:53:00 +0000 Date: Wed, 6 May 2015 21:53:00 +0000 (UTC) From: "Mehant Baid (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-1673) Flatten function can not work well with nested arrays. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DRILL-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531506#comment-14531506 ] Mehant Baid commented on DRILL-1673: ------------------------------------ Assigning it to [~jaltekruse], he has a partial fix for this already. > Flatten function can not work well with nested arrays. > ------------------------------------------------------ > > Key: DRILL-1673 > URL: https://issues.apache.org/jira/browse/DRILL-1673 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill > Affects Versions: 0.7.0 > Environment: 0.7.0 > Reporter: Hao Zhu > Assignee: Jason Altekruse > Priority: Blocker > Fix For: 1.0.0 > > Attachments: DRILL-1673.patch, error.log > > > Flatten function failed to scan nested arrays , for example something like ""[[ ]]"". > The only difference of JSON files between below 2 tests is > "num":[1,2,3] > VS > "num":[[1,2,3]] > ==============Test 1 (Works well):============== > file: > {code} > {"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num": [1,2,3]}, {"id1":"2","name":"hao", "num": [4,5,6]} ]} > {code} > SQL: > {code} > 0: jdbc:drill:zk=local> select t.`fixed_column` as fixed_column, flatten(t.`list_column`) from dfs.root.`/Users/hzu/Documents/sharefolder/hp/n2.json` as t; > +--------------+------------+ > | fixed_column | EXPR$1 | > +--------------+------------+ > | abc | {"id1":"1","name":"zhu","num":[1,2,3]} | > | abc | {"id1":"2","name":"hao","num":[4,5,6]} | > +--------------+------------+ > 2 rows selected (0.154 seconds) > {code} > ==============Test 2 (Failed):============== > file: > {code} > {"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num": [[1,2,3]]}, {"id1":"2","name":"hao", "num": [[4,5,6]]} ]} > {code} > SQL: > {code} > 0: jdbc:drill:zk=local> select t.`fixed_column` as fixed_column, flatten(t.`list_column`) from dfs.root.`/Users/hzu/Documents/sharefolder/hp/n3.json` as t; > +--------------+------------+ > | fixed_column | EXPR$1 | > +--------------+------------+ > Query failed: Failure while running fragment.[ df28347b-fac1-497d-b9c5-a313ba77aa4d on 10.250.0.115:31010 ] > (java.lang.UnsupportedOperationException) > org.apache.drill.exec.vector.complex.RepeatedListVector$RepeatedListTransferPair.splitAndTransfer():339 > org.apache.drill.exec.vector.complex.RepeatedMapVector$SingleMapTransferPair.splitAndTransfer():305 > org.apache.drill.exec.test.generated.FlattenerGen22.flattenRecords():93 > org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork():152 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():89 > org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext():118 > org.apache.drill.exec.record.AbstractRecordBatch.next():106 > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124 > org.apache.drill.exec.record.AbstractRecordBatch.next():86 > org.apache.drill.exec.record.AbstractRecordBatch.next():76 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():52 > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129 > org.apache.drill.exec.record.AbstractRecordBatch.next():106 > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124 > org.apache.drill.exec.physical.impl.BaseRootExec.next():67 > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():122 > org.apache.drill.exec.physical.impl.BaseRootExec.next():57 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():105 > org.apache.drill.exec.work.WorkManager$RunnableWrapper.run():249 > .......():0 > java.lang.RuntimeException: java.sql.SQLException: Failure while executing query. > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514) > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148) > at sqlline.SqlLine.print(SqlLine.java:1809) > at sqlline.SqlLine$Commands.execute(SqlLine.java:3766) > at sqlline.SqlLine$Commands.sql(SqlLine.java:3663) > at sqlline.SqlLine.dispatch(SqlLine.java:889) > at sqlline.SqlLine.begin(SqlLine.java:763) > at sqlline.SqlLine.start(SqlLine.java:498) > at sqlline.SqlLine.main(SqlLine.java:460) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)