From issues-return-184225-archive-asf-public=cust-asf.ponee.io@spark.apache.org Wed Feb 7 02:40:12 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 483F3180657 for ; Wed, 7 Feb 2018 02:40:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 384A4160C5B; Wed, 7 Feb 2018 01:40:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7E872160C46 for ; Wed, 7 Feb 2018 02:40:11 +0100 (CET) Received: (qmail 61418 invoked by uid 500); 7 Feb 2018 01:40:10 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 61338 invoked by uid 99); 7 Feb 2018 01:40:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Feb 2018 01:40:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8325918032C for ; Wed, 7 Feb 2018 01:40:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id gtop_xt4-NHL for ; Wed, 7 Feb 2018 01:40:08 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 24DF55F296 for ; Wed, 7 Feb 2018 01:40:08 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1899CE02AA for ; Wed, 7 Feb 2018 01:40:07 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0C28224106 for ; Wed, 7 Feb 2018 01:40:02 +0000 (UTC) Date: Wed, 7 Feb 2018 01:40:02 +0000 (UTC) From: "Xiao Li (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SPARK-22226) splitExpression can create too many method calls (generating a Constant Pool limit error) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-22226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22226: ---------------------------- Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > splitExpression can create too many method calls (generating a Constant Pool limit error) > ----------------------------------------------------------------------------------------- > > Key: SPARK-22226 > URL: https://issues.apache.org/jira/browse/SPARK-22226 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 2.2.0 > Reporter: Marco Gaido > Assignee: Marco Gaido > Priority: Major > Fix For: 2.3.0 > > > Code generation for very wide datasets can fail because of the Constant Pool limit reached. > This can be caused by many reasons. One of them is that we are currently splitting the definition of the generated methods among several {{NestedClass}} but all these methods are called in the main class. Since we have entries added to the constant pool for each method invocation, this is limiting the number of rows and is leading for very wide dataset to: > {noformat} > org.codehaus.janino.JaninoRuntimeException: Constant pool for class org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection has grown past JVM limit of 0xFFFF > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org