spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuaki Ishizaki (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error
Date Tue, 15 Aug 2017 16:27:02 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127477#comment-16127477
] 

Kazuaki Ishizaki edited comment on SPARK-21720 at 8/15/17 4:26 PM:
-------------------------------------------------------------------

In this case, to add JVM option {{-Xss512m}} eliminates this exception and this works well.

However, when the number of fields is 1024, I got the following exception:
{code}
08:41:40.022 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed
to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "apply(Lorg/apache/spark/sql/catalyst/InternalRow;)Lorg/apache/spark/sql/catalyst/expressions/UnsafeRow;"
of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
grows beyond 64 KB
...
{code}

I am working for solving this 64KB problem.


was (Author: kiszk):
In this case, to add JVM option {{-Xss512m}} eliminates this exception and this works well.

When the number of fields is 1024, I got the following exception:
{code}
08:41:40.022 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed
to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "apply(Lorg/apache/spark/sql/catalyst/InternalRow;)Lorg/apache/spark/sql/catalyst/expressions/UnsafeRow;"
of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
grows beyond 64 KB
...
{code}

I am working for solving this 64KB problem.

> Filter predicate with many conditions throw stackoverflow error
> ---------------------------------------------------------------
>
>                 Key: SPARK-21720
>                 URL: https://issues.apache.org/jira/browse/SPARK-21720
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: srinivasan
>
> When trying to filter on dataset with many predicate conditions on both spark sql and
dataset filter transformation as described below, spark throws a stackoverflow exception
> Case 1: Filter Transformation on Data
>         Dataset<Row> filter = sourceDataset.filter(String.format("not(%s)", buildQuery()));
>         filter.show();
> where buildQuery() returns
> Field1 = "" and  Field2 = "" and  Field3 = "" and  Field4 = "" and  Field5 = "" and 
BLANK_5 = "" and  Field7 = "" and  Field8 = "" and  Field9 = "" and  Field10 = "" and  Field11
= "" and  Field12 = "" and  Field13 = "" and  Field14 = "" and  Field15 = "" and  Field16
= "" and  Field17 = "" and  Field18 = "" and  Field19 = "" and  Field20 = "" and  Field21
= "" and  Field22 = "" and  Field23 = "" and  Field24 = "" and  Field25 = "" and  Field26
= "" and  Field27 = "" and  Field28 = "" and  Field29 = "" and  Field30 = "" and  Field31
= "" and  Field32 = "" and  Field33 = "" and  Field34 = "" and  Field35 = "" and  Field36
= "" and  Field37 = "" and  Field38 = "" and  Field39 = "" and  Field40 = "" and  Field41
= "" and  Field42 = "" and  Field43 = "" and  Field44 = "" and  Field45 = "" and  Field46
= "" and  Field47 = "" and  Field48 = "" and  Field49 = "" and  Field50 = "" and  Field51
= "" and  Field52 = "" and  Field53 = "" and  Field54 = "" and  Field55 = "" and  Field56
= "" and  Field57 = "" and  Field58 = "" and  Field59 = "" and  Field60 = "" and  Field61
= "" and  Field62 = "" and  Field63 = "" and  Field64 = "" and  Field65 = "" and  Field66
= "" and  Field67 = "" and  Field68 = "" and  Field69 = "" and  Field70 = "" and  Field71
= "" and  Field72 = "" and  Field73 = "" and  Field74 = "" and  Field75 = "" and  Field76
= "" and  Field77 = "" and  Field78 = "" and  Field79 = "" and  Field80 = "" and  Field81
= "" and  Field82 = "" and  Field83 = "" and  Field84 = "" and  Field85 = "" and  Field86
= "" and  Field87 = "" and  Field88 = "" and  Field89 = "" and  Field90 = "" and  Field91
= "" and  Field92 = "" and  Field93 = "" and  Field94 = "" and  Field95 = "" and  Field96
= "" and  Field97 = "" and  Field98 = "" and  Field99 = "" and  Field100 = "" and  Field101
= "" and  Field102 = "" and  Field103 = "" and  Field104 = "" and  Field105 = "" and  Field106
= "" and  Field107 = "" and  Field108 = "" and  Field109 = "" and  Field110 = "" and  Field111
= "" and  Field112 = "" and  Field113 = "" and  Field114 = "" and  Field115 = "" and  Field116
= "" and  Field117 = "" and  Field118 = "" and  Field119 = "" and  Field120 = "" and  Field121
= "" and  Field122 = "" and  Field123 = "" and  Field124 = "" and  Field125 = "" and  Field126
= "" and  Field127 = "" and  Field128 = "" and  Field129 = "" and  Field130 = "" and  Field131
= "" and  Field132 = "" and  Field133 = "" and  Field134 = "" and  Field135 = "" and  Field136
= "" and  Field137 = "" and  Field138 = "" and  Field139 = "" and  Field140 = "" and  Field141
= "" and  Field142 = "" and  Field143 = "" and  Field144 = "" and  Field145 = "" and  Field146
= "" and  Field147 = "" and  Field148 = "" and  Field149 = "" and  Field150 = "" and  Field151
= "" and  Field152 = "" and  Field153 = "" and  Field154 = "" and  Field155 = "" and  Field156
= "" and  Field157 = "" and  Field158 = "" and  Field159 = "" and  Field160 = "" and  Field161
= "" and  Field162 = "" and  Field163 = "" and  Field164 = "" and  Field165 = "" and  Field166
= "" and  Field167 = "" and  Field168 = "" and  Field169 = "" and  Field170 = "" and  Field171
= "" and  Field172 = "" and  Field173 = "" and  Field174 = "" and  Field175 = "" and  Field176
= "" and  Field177 = "" and  Field178 = "" and  Field179 = "" and  Field180 = "" and  Field181
= "" and  Field182 = "" and  Field183 = "" and  Field184 = "" and  Field185 = "" and  Field186
= "" and  Field187 = "" and  Field188 = "" and  Field189 = "" and  Field190 = "" and  Field191
= "" and  Field192 = "" and  Field193 = "" and  Field194 = "" and  Field195 = "" and  Field196
= "" and  Field197 = "" and  Field198 = "" and  Field199 = "" and  Field200 = "" and  Field201
= "" and  Field202 = "" and  Field203 = "" and  Field204 = "" and  Field205 = "" and  Field206
= "" and  Field207 = "" and  Field208 = "" and  Field209 = "" and  Field210 = "" and  Field211
= "" and  Field212 = "" and  Field213 = "" and  Field214 = "" and  Field215 = "" and  Field216
= "" and  Field217 = "" and  Field218 = "" and  Field219 = "" and  Field220 = "" and  Field221
= "" and  Field222 = "" and  Field223 = "" and  Field224 = "" and  Field225 = "" and  Field226
= "" and  Field227 = "" and  Field228 = "" and  Field229 = "" and  Field230 = "" and  Field231
= "" and  Field232 = "" and  Field233 = "" and  Field234 = "" and  Field235 = "" and  Field236
= "" and  Field237 = "" and  Field238 = "" and  Field239 = "" and  Field240 = "" and  Field241
= "" and  Field242 = "" and  Field243 = "" and  Field244 = "" and  Field245 = "" and  Field246
= "" and  Field247 = "" and  Field248 = "" and  Field249 = "" and  Field250 = "" and  Field251
= "" and  Field252 = "" and  Field253 = "" and  Field254 = "" and  Field255 = "" and  Field256
= "" and  Field257 = "" and  Field258 = "" and  Field259 = "" and  Field260 = "" and  Field261
= "" and  Field262 = "" and  Field263 = "" and  Field264 = "" and  Field265 = "" and  Field266
= "" and  Field267 = "" and  Field268 = "" and  Field269 = "" and  Field270 = "" and  Field271
= "" and  Field272 = "" and  Field273 = "" and  Field274 = "" and  Field275 = "" and  Field276
= "" and  Field277 = "" and  Field278 = "" and  Field279 = "" and  Field280 = "" and  Field281
= "" and  Field282 = "" and  Field283 = "" and  Field284 = "" and  Field285 = "" and  Field286
= "" and  Field287 = "" and  Field288 = "" and  Field289 = "" and  Field290 = "" and  Field291
= "" and  Field292 = "" and  Field293 = "" and  Field294 = "" and  Field295 = "" and  Field296
= "" and  Field297 = "" and  Field298 = "" and  Field299 = "" and  Field300 = "" and  Field301
= "" and  Field302 = "" and  Field303 = "" and  Field304 = "" and  Field305 = "" and  Field306
= "" and  Field307 = "" and  Field308 = "" and  Field309 = "" and  Field310 = "" and  Field311
= "" and  Field312 = "" and  Field313 = "" and  Field314 = "" and  Field315 = "" and  Field316
= "" and  Field317 = "" and  Field318 = "" and  Field319 = "" and  Field320 = "" and  Field321
= "" and  Field322 = "" and  Field323 = "" and  Field324 = "" and  Field325 = "" and  Field326
= "" and  Field327 = "" and  Field328 = "" and  Field329 = "" and  Field330 = "" and  Field331
= "" and  Field332 = "" and  Field333 = "" and  Field334 = ""
> Case 2 : Using Spark SQL on temp view
> select * from temp where not( Field1 = '' and  Field2 = '' and  Field3 = '' and  Field4
= '' and  Field5 = '' and  BLANK_5 = '' and  Field7 = '' and  Field8 = '' and  Field9 = ''
and  Field10 = '' and  Field11 = '' and  Field12 = '' and  Field13 = '' and  Field14 = ''
and  Field15 = '' and  Field16 = '' and  Field17 = '' and  Field18 = '' and  Field19 = ''
and  Field20 = '' and  Field21 = '' and  Field22 = '' and  Field23 = '' and  Field24 = ''
and  Field25 = '' and  Field26 = '' and  Field27 = '' and  Field28 = '' and  Field29 = ''
and  Field30 = '' and  Field31 = '' and  Field32 = '' and  Field33 = '' and  Field34 = ''
and  Field35 = '' and  Field36 = '' and  Field37 = '' and  Field38 = '' and  Field39 = ''
and  Field40 = '' and  Field41 = '' and  Field42 = '' and  Field43 = '' and  Field44 = ''
and  Field45 = '' and  Field46 = '' and  Field47 = '' and  Field48 = '' and  Field49 = ''
and  Field50 = '' and  Field51 = '' and  Field52 = '' and  Field53 = '' and  Field54 = ''
and  Field55 = '' and  Field56 = '' and  Field57 = '' and  Field58 = '' and  Field59 = ''
and  Field60 = '' and  Field61 = '' and  Field62 = '' and  Field63 = '' and  Field64 = ''
and  Field65 = '' and  Field66 = '' and  Field67 = '' and  Field68 = '' and  Field69 = ''
and  Field70 = '' and  Field71 = '' and  Field72 = '' and  Field73 = '' and  Field74 = ''
and  Field75 = '' and  Field76 = '' and  Field77 = '' and  Field78 = '' and  Field79 = ''
and  Field80 = '' and  Field81 = '' and  Field82 = '' and  Field83 = '' and  Field84 = ''
and  Field85 = '' and  Field86 = '' and  Field87 = '' and  Field88 = '' and  Field89 = ''
and  Field90 = '' and  Field91 = '' and  Field92 = '' and  Field93 = '' and  Field94 = ''
and  Field95 = '' and  Field96 = '' and  Field97 = '' and  Field98 = '' and  Field99 = ''
and  Field100 = '' and  Field101 = '' and  Field102 = '' and  Field103 = '' and  Field104
= '' and  Field105 = '' and  Field106 = '' and  Field107 = '' and  Field108 = '' and  Field109
= '' and  Field110 = '' and  Field111 = '' and  Field112 = '' and  Field113 = '' and  Field114
= '' and  Field115 = '' and  Field116 = '' and  Field117 = '' and  Field118 = '' and  Field119
= '' and  Field120 = '' and  Field121 = '' and  Field122 = '' and  Field123 = '' and  Field124
= '' and  Field125 = '' and  Field126 = '' and  Field127 = '' and  Field128 = '' and  Field129
= '' and  Field130 = '' and  Field131 = '' and  Field132 = '' and  Field133 = '' and  Field134
= '' and  Field135 = '' and  Field136 = '' and  Field137 = '' and  Field138 = '' and  Field139
= '' and  Field140 = '' and  Field141 = '' and  Field142 = '' and  Field143 = '' and  Field144
= '' and  Field145 = '' and  Field146 = '' and  Field147 = '' and  Field148 = '' and  Field149
= '' and  Field150 = '' and  Field151 = '' and  Field152 = '' and  Field153 = '' and  Field154
= '' and  Field155 = '' and  Field156 = '' and  Field157 = '' and  Field158 = '' and  Field159
= '' and  Field160 = '' and  Field161 = '' and  Field162 = '' and  Field163 = '' and  Field164
= '' and  Field165 = '' and  Field166 = '' and  Field167 = '' and  Field168 = '' and  Field169
= '' and  Field170 = '' and  Field171 = '' and  Field172 = '' and  Field173 = '' and  Field174
= '' and  Field175 = '' and  Field176 = '' and  Field177 = '' and  Field178 = '' and  Field179
= '' and  Field180 = '' and  Field181 = '' and  Field182 = '' and  Field183 = '' and  Field184
= '' and  Field185 = '' and  Field186 = '' and  Field187 = '' and  Field188 = '' and  Field189
= '' and  Field190 = '' and  Field191 = '' and  Field192 = '' and  Field193 = '' and  Field194
= '' and  Field195 = '' and  Field196 = '' and  Field197 = '' and  Field198 = '' and  Field199
= '' and  Field200 = '' and  Field201 = '' and  Field202 = '' and  Field203 = '' and  Field204
= '' and  Field205 = '' and  Field206 = '' and  Field207 = '' and  Field208 = '' and  Field209
= '' and  Field210 = '' and  Field211 = '' and  Field212 = '' and  Field213 = '' and  Field214
= '' and  Field215 = '' and  Field216 = '' and  Field217 = '' and  Field218 = '' and  Field219
= '' and  Field220 = '' and  Field221 = '' and  Field222 = '' and  Field223 = '' and  Field224
= '' and  Field225 = '' and  Field226 = '' and  Field227 = '' and  Field228 = '' and  Field229
= '' and  Field230 = '' and  Field231 = '' and  Field232 = '' and  Field233 = '' and  Field234
= '' and  Field235 = '' and  Field236 = '' and  Field237 = '' and  Field238 = '' and  Field239
= '' and  Field240 = '' and  Field241 = '' and  Field242 = '' and  Field243 = '' and  Field244
= '' and  Field245 = '' and  Field246 = '' and  Field247 = '' and  Field248 = '' and  Field249
= '' and  Field250 = '' and  Field251 = '' and  Field252 = '' and  Field253 = '' and  Field254
= '' and  Field255 = '' and  Field256 = '' and  Field257 = '' and  Field258 = '' and  Field259
= '' and  Field260 = '' and  Field261 = '' and  Field262 = '' and  Field263 = '' and  Field264
= '' and  Field265 = '' and  Field266 = '' and  Field267 = '' and  Field268 = '' and  Field269
= '' and  Field270 = '' and  Field271 = '' and  Field272 = '' and  Field273 = '' and  Field274
= '' and  Field275 = '' and  Field276 = '' and  Field277 = '' and  Field278 = '' and  Field279
= '' and  Field280 = '' and  Field281 = '' and  Field282 = '' and  Field283 = '' and  Field284
= '' and  Field285 = '' and  Field286 = '' and  Field287 = '' and  Field288 = '' and  Field289
= '' and  Field290 = '' and  Field291 = '' and  Field292 = '' and  Field293 = '' and  Field294
= '' and  Field295 = '' and  Field296 = '' and  Field297 = '' and  Field298 = '' and  Field299
= '' and  Field300 = '' and  Field301 = '' and  Field302 = '' and  Field303 = '' and  Field304
= '' and  Field305 = '' and  Field306 = '' and  Field307 = '' and  Field308 = '' and  Field309
= '' and  Field310 = '' and  Field311 = '' and  Field312 = '' and  Field313 = '' and  Field314
= '' and  Field315 = '' and  Field316 = '' and  Field317 = '' and  Field318 = '' and  Field319
= '' and  Field320 = '' and  Field321 = '' and  Field322 = '' and  Field323 = '' and  Field324
= '' and  Field325 = '' and  Field326 = '' and  Field327 = '' and  Field328 = '' and  Field329
= '' and  Field330 = '' and  Field331 = '' and  Field332 = '' and  Field333 = '' and  Field334
= '')
> in both cases, the error thrown is
> java.lang.StackOverflowError
> 	at org.codehaus.janino.CodeContext.extract16BitValue(CodeContext.java:679)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:545)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> 	at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> ......
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message