drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jean-claude (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-4859) Missing function implementation: [repeated_count(INT-OPTIONAL)
Date Mon, 22 Aug 2016 18:25:21 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431360#comment-15431360
] 

jean-claude edited comment on DRILL-4859 at 8/22/16 6:24 PM:
-------------------------------------------------------------

I looks like the JSON reader emits INT fields when it does not know anything about the selected
field.

As such I have added repeated_count functions for the types INT-REQUIRED, INT-OPTIONAL, BIGINT-REQUIRED,
BIGINT-OPTIONAL. I then run the query above using enable_union_type enabled.

It now computes the count of zero for the few thousands rows until it reaches a map which
has the given field. At that point I get this error
{code}
| 0      |
| 0      |
| 0      |
Error: SYSTEM ERROR: DrillRuntimeException: Unable to cast union to LIST

Fragment 0:0

[Error Id: d03e7bb0-365e-40d6-aa79-4c826754c675 on 192.168.1.3:31010]

  (org.apache.drill.common.exceptions.DrillRuntimeException) Unable to cast union to LIST
    org.apache.drill.exec.expr.fn.ExceptionFunction.throwException():51
    org.apache.drill.exec.test.generated.ProjectorGen2.doEval():113
    org.apache.drill.exec.test.generated.ProjectorGen2.projectRecords():62
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():199
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
    org.apache.drill.exec.record.AbstractRecordBatch.next():162
    org.apache.drill.exec.physical.impl.BaseRootExec.next():104
    org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
    org.apache.drill.exec.physical.impl.BaseRootExec.next():94
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1657
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)
{code}



was (Author: jccote):
I looks like the JSON reader emits INT fields when it does not know anything about the selected
field.

As such I have added repeated_count functions for INT, INT-OPTIONAL, BIGINT, BIGINT-OPTIONAL.
I then run the query above using enable_union_type enabled.

It now computes the count of zero for the few thousands rows until it reaches a map which
has the given field. At that point I get this error
{code}
| 0      |
| 0      |
| 0      |
Error: SYSTEM ERROR: DrillRuntimeException: Unable to cast union to LIST

Fragment 0:0

[Error Id: d03e7bb0-365e-40d6-aa79-4c826754c675 on 192.168.1.3:31010]

  (org.apache.drill.common.exceptions.DrillRuntimeException) Unable to cast union to LIST
    org.apache.drill.exec.expr.fn.ExceptionFunction.throwException():51
    org.apache.drill.exec.test.generated.ProjectorGen2.doEval():113
    org.apache.drill.exec.test.generated.ProjectorGen2.projectRecords():62
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():199
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
    org.apache.drill.exec.record.AbstractRecordBatch.next():162
    org.apache.drill.exec.physical.impl.BaseRootExec.next():104
    org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
    org.apache.drill.exec.physical.impl.BaseRootExec.next():94
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1657
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)
{code}


> Missing function implementation: [repeated_count(INT-OPTIONAL)
> --------------------------------------------------------------
>
>                 Key: DRILL-4859
>                 URL: https://issues.apache.org/jira/browse/DRILL-4859
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: jean-claude
>            Priority: Minor
>
> Given a JSON file with many thousands of empty maps {} and a few lines like this at the
end of the file
> {code}
> {}
> {}
> {}
> {}
> {}
> {"listArray":[], "intArray": [1,2,3,4], "mapArray": [{"name": "foo"},{"name": "foo"}],
"label": "foo"}
> {"listArray":[[1,2],[3,4],[5,6]], "intArray": [1,2,3,4], "mapArray": [{"name": "foo"}],
"label": "foo"}
> {"listArray":[[1,2]], "intArray": [1,2,3,4], "mapArray": [], "label": "foo"}
> {"listArray":[[1,2],[3,4]], "intArray": [1,2,3,4], "mapArray": [{"name": "foo"},{"name":
"foo"},{"name": "foo"}], "label": "foo"}
> {"listArray":[[1,2],[3,4],[5,6]], "intArray": [1,2,3,4], "mapArray": [{"name": "foo"},{"name":
"foo"}], "label": "foo"}
> {}
> {"intArray": [1,2,3,4], "mapArray": [{"name": "foo"},{"name": "foo"},{"name": "foo"},{"name":
"foo"}], "label": "foo"}
> {"intArray": [1,2,3,4], "mapArray": [{"name": "foo"},{"name": "foo"}], "label": "foo"}
> {code}
> If you perform a repeated_count on the intArray field. The JSON reader will assume a
plain INT column. You can do this with any type of array even the mapArray above, you get
the same error.
> {code}
> 0: jdbc:drill:zk=local> select repeated_count(intArray) from dfs.`/Users/jccote/repeated_count.json`;
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming
schema.  Errors:
>  
> Error in expression at index -1.  Error: Missing function implementation: [repeated_count(INT-OPTIONAL)].
 Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: f2e25b81-a53c-4fc6-9bf3-2f7b2fa68d60 on 192.168.1.3:31010]
>   (org.apache.drill.exec.exception.SchemaChangeException) Failure while trying to materialize
incoming schema.  Errors:
>  
> Error in expression at index -1.  Error: Missing function implementation: [repeated_count(INT-OPTIONAL)].
 Full expression: --UNKNOWN EXPRESSION--..
>     org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema():424
>     org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
>     org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
>     org.apache.drill.exec.record.AbstractRecordBatch.next():162
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():104
>     org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():94
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
>     java.security.AccessController.doPrivileged():-2
>     javax.security.auth.Subject.doAs():422
>     org.apache.hadoop.security.UserGroupInformation.doAs():1657
>     org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
>     org.apache.drill.common.SelfCleaningRunnable.run():38
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1142
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():617
>     java.lang.Thread.run():745 (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message