spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kunal Goswami (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-25059) Exception while executing an action on DataFrame that read Json
Date Wed, 08 Aug 2018 20:18:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kunal Goswami updated SPARK-25059:
----------------------------------
    External issue URL: https://issues.apache.org/jira/browse/SPARK-16845

> Exception while executing an action on DataFrame that read Json
> ---------------------------------------------------------------
>
>                 Key: SPARK-25059
>                 URL: https://issues.apache.org/jira/browse/SPARK-25059
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Shell
>    Affects Versions: 2.2.0
>         Environment: AWS EMR 5.8.0 
> Spark 2.2.0 
>  
>            Reporter: Kunal Goswami
>            Priority: Major
>              Labels: Spark-SQL
>
> When I try to read ~9600 Json files using
> {noformat}
> val test = spark.read.option("header", true).option("inferSchema", true).json(paths:
_*) {noformat}
>  
> Any action on the above created data frame results in: 
> {noformat}
> Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "apply2_1$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
of class "org.apache.spark.sql.catalyst.expressions.Generat[73/1850]
> pecificUnsafeProjection" grows beyond 64 KB
>   at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:949)
>   at org.codehaus.janino.CodeContext.write(CodeContext.java:839)
>   at org.codehaus.janino.UnitCompiler.writeOpcode(UnitCompiler.java:11081)
>   at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4546)
>   at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774)
>   at org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762)
>   at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180)
>   at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151)
>   at org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112)
>   at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377)
>   at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220)
>   at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220)
>   at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1538)
>   at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1379)
>   at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$ForStatement.accept(Java.java:2660)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1538)
>   at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1379)
>   at org.codehaus.janino.UnitCompiler$6.visitForStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$ForStatement.accept(Java.java:2660)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220)
>   at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
>   at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
>   at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$Block.accept(Java.java:2471)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2220)
>   at org.codehaus.janino.UnitCompiler.access$1800(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1378)
>   at org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$IfStatement.accept(Java.java:2621)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811)
>   at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262)
>   at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894)
>   at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377)
>   at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369)
>   at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420)
>   at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206)
>   at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374)
>   at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369)
>   at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345)
>   at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396)
>   at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311)
>   at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229)
>   at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196)
>   at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91)
>   at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:960)
>   ... 36 more
> {noformat}
> https://issues.apache.org/jira/browse/SPARK-16845 refers that it was fixed and applied
to 2.2.0, but I'm still observing this on AWS EMR 5.8.0. Can you please help?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message