drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinfeng Ni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5140) Fix CompileException in run-time generated code when record batch has large number of fields.
Date Fri, 28 Apr 2017 01:04:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987967#comment-15987967
] 

Jinfeng Ni commented on DRILL-5140:
-----------------------------------

[~vvysotskyi],  seems to me DRILL-5164 falls into similar pattern as the query in DRILL-5140;
 it's using select * from T1 join T2, where both T1 and T2 have more than 5k columns.  I believe
such query will fail on SQL server or Oracle.  Can you try run a similar query on Hive to
see if it can support such query?

Also, with the patch, if I increase the number of fields from 5k to 10k in TestLargeFileCompilation.java,
and run testProject(), I still hit same "too many constant" error.  That is, the patch seems
not to solve the root cause of the problem completely; we still have some restriction in terms
of # of columns used in the query. 

{code}
2017-04-27 17:44:22,322 [BitServer-1] DEBUG o.a.d.exec.rpc.control.ControlServer - Sending
response with Sender 236355166
2017-04-27 17:44:23,852 [26fd6fb9-8c44-ce48-293e-2622045d08e9:frag:0:0] ERROR o.a.d.e.c.DrillDiagnosticListener
- ProjectorGen0.java:230380: error: too many constants
    private class ProjectorGen00 {
            ^ (compiler.err.limit.pool)
{code}

> Fix CompileException in run-time generated code when record batch has large number of
fields.
> ---------------------------------------------------------------------------------------------
>
>                 Key: DRILL-5140
>                 URL: https://issues.apache.org/jira/browse/DRILL-5140
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.9.0
>            Reporter: Khurram Faraaz
>            Assignee: Volodymyr Vysotskyi
>            Priority: Critical
>         Attachments: drill_5117.q, manyColumns.csv
>
>
> CTAS that does SELECT over 5003 columns fails with CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject...
> Drill 1.9.0 git commit ID : 4c1b420b
> CTAS statement and CSV data file are attached.
> I ran test with and without setting the below system option, test failed in both cases.
> alter system set `exec.java_compiler`='JDK';
> sqlline session just closes with below message, after the failing CTAS is executed.
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> Stack trace from drillbit.log
> {noformat}
> 2016-12-20 12:02:16,016 [27a6e241-99b1-1f2a-8a91-394f8166e969:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor
- SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]',
Line 11, Column 8: ProjectorGen45.java:11: error: too many constants
> public class ProjectorGen45 {
>        ^ (compiler.err.limit.pool)
> Fragment 0:0
> [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: CompileException: File
'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]', Line 11, Column
8: ProjectorGen45.java:11: error: too many constants
> public class ProjectorGen45 {
>        ^ (compiler.err.limit.pool)
> Fragment 0:0
> [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
>         at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.9.0.jar:1.9.0]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting
to load generated class
>         at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:487)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
~[drill-java-exec-1.9.0.jar:1.9.0]
>         at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91]
>         at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_91]
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
~[hadoop-common-2.7.0-mapr-1607.jar:na]
>         at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
[drill-java-exec-1.9.0.jar:1.9.0]
>         ... 4 common frames omitted
> Caused by: org.apache.drill.exec.exception.ClassTransformationException: java.util.concurrent.ExecutionException:
org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation
classes for value:
> ...
> {noformat}
> Thanks,
> Khurram



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message