Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id ED4D7200CF6 for ; Mon, 14 Aug 2017 04:11:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id EBF21163C34; Mon, 14 Aug 2017 02:11:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DEB4B163C32 for ; Mon, 14 Aug 2017 04:11:05 +0200 (CEST) Received: (qmail 85382 invoked by uid 500); 14 Aug 2017 02:11:05 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 85252 invoked by uid 99); 14 Aug 2017 02:11:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Aug 2017 02:11:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 905C31A0A82 for ; Mon, 14 Aug 2017 02:11:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id M_AXHHLgF9CO for ; Mon, 14 Aug 2017 02:11:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id B866960D92 for ; Mon, 14 Aug 2017 02:11:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 20BC7E00A7 for ; Mon, 14 Aug 2017 02:11:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 387152140D for ; Mon, 14 Aug 2017 02:11:00 +0000 (UTC) Date: Mon, 14 Aug 2017 02:11:00 +0000 (UTC) From: "srinivasan (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (SPARK-21720) FIlter predicate with many conditions throw stackoverflow error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 14 Aug 2017 02:11:07 -0000 srinivasan created SPARK-21720: ---------------------------------- Summary: FIlter predicate with many conditions throw stackover= flow error Key: SPARK-21720 URL: https://issues.apache.org/jira/browse/SPARK-21720 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0 Reporter: srinivasan When trying to filter on dataset with many predicate conditions on both spa= rk sql and dataset filter transformation as described below, spark throws a= stackoverflow exception Case 1: Filter Transformation on Data Dataset filter =3D sourceDataset.filter(String.format("not(%s)= ", buildQuery())); filter.show(); where buildQuery() returns Field1 =3D "" and Field2 =3D "" and Field3 =3D "" and Field4 =3D "" and = Field5 =3D "" and BLANK_5 =3D "" and Field7 =3D "" and Field8 =3D "" an= d Field9 =3D "" and Field10 =3D "" and Field11 =3D "" and Field12 =3D "= " and Field13 =3D "" and Field14 =3D "" and Field15 =3D "" and Field16 = =3D "" and Field17 =3D "" and Field18 =3D "" and Field19 =3D "" and Fie= ld20 =3D "" and Field21 =3D "" and Field22 =3D "" and Field23 =3D "" and= Field24 =3D "" and Field25 =3D "" and Field26 =3D "" and Field27 =3D "= " and Field28 =3D "" and Field29 =3D "" and Field30 =3D "" and Field31 = =3D "" and Field32 =3D "" and Field33 =3D "" and Field34 =3D "" and Fie= ld35 =3D "" and Field36 =3D "" and Field37 =3D "" and Field38 =3D "" and= Field39 =3D "" and Field40 =3D "" and Field41 =3D "" and Field42 =3D "= " and Field43 =3D "" and Field44 =3D "" and Field45 =3D "" and Field46 = =3D "" and Field47 =3D "" and Field48 =3D "" and Field49 =3D "" and Fie= ld50 =3D "" and Field51 =3D "" and Field52 =3D "" and Field53 =3D "" and= Field54 =3D "" and Field55 =3D "" and Field56 =3D "" and Field57 =3D "= " and Field58 =3D "" and Field59 =3D "" and Field60 =3D "" and Field61 = =3D "" and Field62 =3D "" and Field63 =3D "" and Field64 =3D "" and Fie= ld65 =3D "" and Field66 =3D "" and Field67 =3D "" and Field68 =3D "" and= Field69 =3D "" and Field70 =3D "" and Field71 =3D "" and Field72 =3D "= " and Field73 =3D "" and Field74 =3D "" and Field75 =3D "" and Field76 = =3D "" and Field77 =3D "" and Field78 =3D "" and Field79 =3D "" and Fie= ld80 =3D "" and Field81 =3D "" and Field82 =3D "" and Field83 =3D "" and= Field84 =3D "" and Field85 =3D "" and Field86 =3D "" and Field87 =3D "= " and Field88 =3D "" and Field89 =3D "" and Field90 =3D "" and Field91 = =3D "" and Field92 =3D "" and Field93 =3D "" and Field94 =3D "" and Fie= ld95 =3D "" and Field96 =3D "" and Field97 =3D "" and Field98 =3D "" and= Field99 =3D "" and Field100 =3D "" and Field101 =3D "" and Field102 = =3D "" and Field103 =3D "" and Field104 =3D "" and Field105 =3D "" and = Field106 =3D "" and Field107 =3D "" and Field108 =3D "" and Field109 =3D= "" and Field110 =3D "" and Field111 =3D "" and Field112 =3D "" and Fie= ld113 =3D "" and Field114 =3D "" and Field115 =3D "" and Field116 =3D ""= and Field117 =3D "" and Field118 =3D "" and Field119 =3D "" and Field1= 20 =3D "" and Field121 =3D "" and Field122 =3D "" and Field123 =3D "" an= d Field124 =3D "" and Field125 =3D "" and Field126 =3D "" and Field127 = =3D "" and Field128 =3D "" and Field129 =3D "" and Field130 =3D "" and = Field131 =3D "" and Field132 =3D "" and Field133 =3D "" and Field134 =3D= "" and Field135 =3D "" and Field136 =3D "" and Field137 =3D "" and Fie= ld138 =3D "" and Field139 =3D "" and Field140 =3D "" and Field141 =3D ""= and Field142 =3D "" and Field143 =3D "" and Field144 =3D "" and Field1= 45 =3D "" and Field146 =3D "" and Field147 =3D "" and Field148 =3D "" an= d Field149 =3D "" and Field150 =3D "" and Field151 =3D "" and Field152 = =3D "" and Field153 =3D "" and Field154 =3D "" and Field155 =3D "" and = Field156 =3D "" and Field157 =3D "" and Field158 =3D "" and Field159 =3D= "" and Field160 =3D "" and Field161 =3D "" and Field162 =3D "" and Fie= ld163 =3D "" and Field164 =3D "" and Field165 =3D "" and Field166 =3D ""= and Field167 =3D "" and Field168 =3D "" and Field169 =3D "" and Field1= 70 =3D "" and Field171 =3D "" and Field172 =3D "" and Field173 =3D "" an= d Field174 =3D "" and Field175 =3D "" and Field176 =3D "" and Field177 = =3D "" and Field178 =3D "" and Field179 =3D "" and Field180 =3D "" and = Field181 =3D "" and Field182 =3D "" and Field183 =3D "" and Field184 =3D= "" and Field185 =3D "" and Field186 =3D "" and Field187 =3D "" and Fie= ld188 =3D "" and Field189 =3D "" and Field190 =3D "" and Field191 =3D ""= and Field192 =3D "" and Field193 =3D "" and Field194 =3D "" and Field1= 95 =3D "" and Field196 =3D "" and Field197 =3D "" and Field198 =3D "" an= d Field199 =3D "" and Field200 =3D "" and Field201 =3D "" and Field202 = =3D "" and Field203 =3D "" and Field204 =3D "" and Field205 =3D "" and = Field206 =3D "" and Field207 =3D "" and Field208 =3D "" and Field209 =3D= "" and Field210 =3D "" and Field211 =3D "" and Field212 =3D "" and Fie= ld213 =3D "" and Field214 =3D "" and Field215 =3D "" and Field216 =3D ""= and Field217 =3D "" and Field218 =3D "" and Field219 =3D "" and Field2= 20 =3D "" and Field221 =3D "" and Field222 =3D "" and Field223 =3D "" an= d Field224 =3D "" and Field225 =3D "" and Field226 =3D "" and Field227 = =3D "" and Field228 =3D "" and Field229 =3D "" and Field230 =3D "" and = Field231 =3D "" and Field232 =3D "" and Field233 =3D "" and Field234 =3D= "" and Field235 =3D "" and Field236 =3D "" and Field237 =3D "" and Fie= ld238 =3D "" and Field239 =3D "" and Field240 =3D "" and Field241 =3D ""= and Field242 =3D "" and Field243 =3D "" and Field244 =3D "" and Field2= 45 =3D "" and Field246 =3D "" and Field247 =3D "" and Field248 =3D "" an= d Field249 =3D "" and Field250 =3D "" and Field251 =3D "" and Field252 = =3D "" and Field253 =3D "" and Field254 =3D "" and Field255 =3D "" and = Field256 =3D "" and Field257 =3D "" and Field258 =3D "" and Field259 =3D= "" and Field260 =3D "" and Field261 =3D "" and Field262 =3D "" and Fie= ld263 =3D "" and Field264 =3D "" and Field265 =3D "" and Field266 =3D ""= and Field267 =3D "" and Field268 =3D "" and Field269 =3D "" and Field2= 70 =3D "" and Field271 =3D "" and Field272 =3D "" and Field273 =3D "" an= d Field274 =3D "" and Field275 =3D "" and Field276 =3D "" and Field277 = =3D "" and Field278 =3D "" and Field279 =3D "" and Field280 =3D "" and = Field281 =3D "" and Field282 =3D "" and Field283 =3D "" and Field284 =3D= "" and Field285 =3D "" and Field286 =3D "" and Field287 =3D "" and Fie= ld288 =3D "" and Field289 =3D "" and Field290 =3D "" and Field291 =3D ""= and Field292 =3D "" and Field293 =3D "" and Field294 =3D "" and Field2= 95 =3D "" and Field296 =3D "" and Field297 =3D "" and Field298 =3D "" an= d Field299 =3D "" and Field300 =3D "" and Field301 =3D "" and Field302 = =3D "" and Field303 =3D "" and Field304 =3D "" and Field305 =3D "" and = Field306 =3D "" and Field307 =3D "" and Field308 =3D "" and Field309 =3D= "" and Field310 =3D "" and Field311 =3D "" and Field312 =3D "" and Fie= ld313 =3D "" and Field314 =3D "" and Field315 =3D "" and Field316 =3D ""= and Field317 =3D "" and Field318 =3D "" and Field319 =3D "" and Field3= 20 =3D "" and Field321 =3D "" and Field322 =3D "" and Field323 =3D "" an= d Field324 =3D "" and Field325 =3D "" and Field326 =3D "" and Field327 = =3D "" and Field328 =3D "" and Field329 =3D "" and Field330 =3D "" and = Field331 =3D "" and Field332 =3D "" and Field333 =3D "" and Field334 =3D= "" Case 2 : Using Spark SQL on temp view select * from temp where not( Field1 =3D '' and Field2 =3D '' and Field3 = =3D '' and Field4 =3D '' and Field5 =3D '' and BLANK_5 =3D '' and Field= 7 =3D '' and Field8 =3D '' and Field9 =3D '' and Field10 =3D '' and Fie= ld11 =3D '' and Field12 =3D '' and Field13 =3D '' and Field14 =3D '' and= Field15 =3D '' and Field16 =3D '' and Field17 =3D '' and Field18 =3D '= ' and Field19 =3D '' and Field20 =3D '' and Field21 =3D '' and Field22 = =3D '' and Field23 =3D '' and Field24 =3D '' and Field25 =3D '' and Fie= ld26 =3D '' and Field27 =3D '' and Field28 =3D '' and Field29 =3D '' and= Field30 =3D '' and Field31 =3D '' and Field32 =3D '' and Field33 =3D '= ' and Field34 =3D '' and Field35 =3D '' and Field36 =3D '' and Field37 = =3D '' and Field38 =3D '' and Field39 =3D '' and Field40 =3D '' and Fie= ld41 =3D '' and Field42 =3D '' and Field43 =3D '' and Field44 =3D '' and= Field45 =3D '' and Field46 =3D '' and Field47 =3D '' and Field48 =3D '= ' and Field49 =3D '' and Field50 =3D '' and Field51 =3D '' and Field52 = =3D '' and Field53 =3D '' and Field54 =3D '' and Field55 =3D '' and Fie= ld56 =3D '' and Field57 =3D '' and Field58 =3D '' and Field59 =3D '' and= Field60 =3D '' and Field61 =3D '' and Field62 =3D '' and Field63 =3D '= ' and Field64 =3D '' and Field65 =3D '' and Field66 =3D '' and Field67 = =3D '' and Field68 =3D '' and Field69 =3D '' and Field70 =3D '' and Fie= ld71 =3D '' and Field72 =3D '' and Field73 =3D '' and Field74 =3D '' and= Field75 =3D '' and Field76 =3D '' and Field77 =3D '' and Field78 =3D '= ' and Field79 =3D '' and Field80 =3D '' and Field81 =3D '' and Field82 = =3D '' and Field83 =3D '' and Field84 =3D '' and Field85 =3D '' and Fie= ld86 =3D '' and Field87 =3D '' and Field88 =3D '' and Field89 =3D '' and= Field90 =3D '' and Field91 =3D '' and Field92 =3D '' and Field93 =3D '= ' and Field94 =3D '' and Field95 =3D '' and Field96 =3D '' and Field97 = =3D '' and Field98 =3D '' and Field99 =3D '' and Field100 =3D '' and Fi= eld101 =3D '' and Field102 =3D '' and Field103 =3D '' and Field104 =3D '= ' and Field105 =3D '' and Field106 =3D '' and Field107 =3D '' and Field= 108 =3D '' and Field109 =3D '' and Field110 =3D '' and Field111 =3D '' a= nd Field112 =3D '' and Field113 =3D '' and Field114 =3D '' and Field115= =3D '' and Field116 =3D '' and Field117 =3D '' and Field118 =3D '' and = Field119 =3D '' and Field120 =3D '' and Field121 =3D '' and Field122 = =3D '' and Field123 =3D '' and Field124 =3D '' and Field125 =3D '' and = Field126 =3D '' and Field127 =3D '' and Field128 =3D '' and Field129 =3D= '' and Field130 =3D '' and Field131 =3D '' and Field132 =3D '' and Fie= ld133 =3D '' and Field134 =3D '' and Field135 =3D '' and Field136 =3D ''= and Field137 =3D '' and Field138 =3D '' and Field139 =3D '' and Field1= 40 =3D '' and Field141 =3D '' and Field142 =3D '' and Field143 =3D '' an= d Field144 =3D '' and Field145 =3D '' and Field146 =3D '' and Field147 = =3D '' and Field148 =3D '' and Field149 =3D '' and Field150 =3D '' and = Field151 =3D '' and Field152 =3D '' and Field153 =3D '' and Field154 =3D= '' and Field155 =3D '' and Field156 =3D '' and Field157 =3D '' and Fie= ld158 =3D '' and Field159 =3D '' and Field160 =3D '' and Field161 =3D ''= and Field162 =3D '' and Field163 =3D '' and Field164 =3D '' and Field1= 65 =3D '' and Field166 =3D '' and Field167 =3D '' and Field168 =3D '' an= d Field169 =3D '' and Field170 =3D '' and Field171 =3D '' and Field172 = =3D '' and Field173 =3D '' and Field174 =3D '' and Field175 =3D '' and = Field176 =3D '' and Field177 =3D '' and Field178 =3D '' and Field179 =3D= '' and Field180 =3D '' and Field181 =3D '' and Field182 =3D '' and Fie= ld183 =3D '' and Field184 =3D '' and Field185 =3D '' and Field186 =3D ''= and Field187 =3D '' and Field188 =3D '' and Field189 =3D '' and Field1= 90 =3D '' and Field191 =3D '' and Field192 =3D '' and Field193 =3D '' an= d Field194 =3D '' and Field195 =3D '' and Field196 =3D '' and Field197 = =3D '' and Field198 =3D '' and Field199 =3D '' and Field200 =3D '' and = Field201 =3D '' and Field202 =3D '' and Field203 =3D '' and Field204 =3D= '' and Field205 =3D '' and Field206 =3D '' and Field207 =3D '' and Fie= ld208 =3D '' and Field209 =3D '' and Field210 =3D '' and Field211 =3D ''= and Field212 =3D '' and Field213 =3D '' and Field214 =3D '' and Field2= 15 =3D '' and Field216 =3D '' and Field217 =3D '' and Field218 =3D '' an= d Field219 =3D '' and Field220 =3D '' and Field221 =3D '' and Field222 = =3D '' and Field223 =3D '' and Field224 =3D '' and Field225 =3D '' and = Field226 =3D '' and Field227 =3D '' and Field228 =3D '' and Field229 =3D= '' and Field230 =3D '' and Field231 =3D '' and Field232 =3D '' and Fie= ld233 =3D '' and Field234 =3D '' and Field235 =3D '' and Field236 =3D ''= and Field237 =3D '' and Field238 =3D '' and Field239 =3D '' and Field2= 40 =3D '' and Field241 =3D '' and Field242 =3D '' and Field243 =3D '' an= d Field244 =3D '' and Field245 =3D '' and Field246 =3D '' and Field247 = =3D '' and Field248 =3D '' and Field249 =3D '' and Field250 =3D '' and = Field251 =3D '' and Field252 =3D '' and Field253 =3D '' and Field254 =3D= '' and Field255 =3D '' and Field256 =3D '' and Field257 =3D '' and Fie= ld258 =3D '' and Field259 =3D '' and Field260 =3D '' and Field261 =3D ''= and Field262 =3D '' and Field263 =3D '' and Field264 =3D '' and Field2= 65 =3D '' and Field266 =3D '' and Field267 =3D '' and Field268 =3D '' an= d Field269 =3D '' and Field270 =3D '' and Field271 =3D '' and Field272 = =3D '' and Field273 =3D '' and Field274 =3D '' and Field275 =3D '' and = Field276 =3D '' and Field277 =3D '' and Field278 =3D '' and Field279 =3D= '' and Field280 =3D '' and Field281 =3D '' and Field282 =3D '' and Fie= ld283 =3D '' and Field284 =3D '' and Field285 =3D '' and Field286 =3D ''= and Field287 =3D '' and Field288 =3D '' and Field289 =3D '' and Field2= 90 =3D '' and Field291 =3D '' and Field292 =3D '' and Field293 =3D '' an= d Field294 =3D '' and Field295 =3D '' and Field296 =3D '' and Field297 = =3D '' and Field298 =3D '' and Field299 =3D '' and Field300 =3D '' and = Field301 =3D '' and Field302 =3D '' and Field303 =3D '' and Field304 =3D= '' and Field305 =3D '' and Field306 =3D '' and Field307 =3D '' and Fie= ld308 =3D '' and Field309 =3D '' and Field310 =3D '' and Field311 =3D ''= and Field312 =3D '' and Field313 =3D '' and Field314 =3D '' and Field3= 15 =3D '' and Field316 =3D '' and Field317 =3D '' and Field318 =3D '' an= d Field319 =3D '' and Field320 =3D '' and Field321 =3D '' and Field322 = =3D '' and Field323 =3D '' and Field324 =3D '' and Field325 =3D '' and = Field326 =3D '' and Field327 =3D '' and Field328 =3D '' and Field329 =3D= '' and Field330 =3D '' and Field331 =3D '' and Field332 =3D '' and Fie= ld333 =3D '' and Field334 =3D '') in both cases, the error thrown is java.lang.StackOverflowError =09at org.codehaus.janino.CodeContext.extract16BitValue(CodeContext.java:67= 9) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:545) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) =09at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541) ...... Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org