From issues-return-206796-archive-asf-public=cust-asf.ponee.io@spark.apache.org Sun Nov 4 20:21:08 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B3E98180677 for ; Sun, 4 Nov 2018 20:21:07 +0100 (CET) Received: (qmail 69945 invoked by uid 500); 4 Nov 2018 19:21:06 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 69818 invoked by uid 99); 4 Nov 2018 19:21:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Nov 2018 19:21:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 69A90C0417 for ; Sun, 4 Nov 2018 19:21:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.251 X-Spam-Level: X-Spam-Status: No, score=-109.251 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id f3dhJM88ojHZ for ; Sun, 4 Nov 2018 19:21:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 722F45F430 for ; Sun, 4 Nov 2018 19:21:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 9E2B2E12FE for ; Sun, 4 Nov 2018 19:21:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2D46127763 for ; Sun, 4 Nov 2018 19:21:00 +0000 (UTC) Date: Sun, 4 Nov 2018 19:21:00 +0000 (UTC) From: "Benedikt Beckermann (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SPARK-25938) Action on cached dataset causes WARN java.lang.AssertionError in log4j logs. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-25938?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedikt Beckermann updated SPARK-25938: ---------------------------------------- Description:=20 Caching and counting any DataSet causes an AssertionError in the log4j logs= . The results are still valid and no stderr is shown.=C2=A0 Example code:=20 {code:scala} var df =3D Seq(100).toDF("count") df.cache() df.count() {code} log4j Output:=20 {code} 18/11/02 15:39:57 WARN ExecutionListenerManager: Error executing query exec= ution listener java.lang.AssertionError: assertion failed: InMemoryRelation fields: output= , useCompression, batchSize, storageLevel, child, tableName, _cachedColumnB= uffers, rowCountStats, sizeInBytesStats, statsOfPlanToCache, outputOrdering= , values: List(count#124), true, 10000, StorageLevel(disk, memory, deserial= ized, 1 replicas), LocalTableScan [count#47] , None, LocalTableScan [count#47] MapPartitionsRDD[2] at cache at command-1842411053765017:3, LongAccumulato= r(id: 0, name: None, value: 1), LongAccumulator(id: 1, name: None, value: 4= ), Statistics(sizeInBytes=3D12.0 B, hints=3Dnone) =09at scala.Predef$.assert(Predef.scala:170) =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.scal= a:638) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:626) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala= :631) =09at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:61= 7) =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger$.getSQLQueryP= lanBlob(SQLQueryPlanLogger.scala:70) =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger.onSuccess(SQL= QueryPlanLogger.scala:42) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:124) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apach= e$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryE= xecutionListener.scala:145) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apach= e$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryE= xecutionListener.scala:143) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at scala.collection.generic.TraversableForwarder$class.foreach(Traversab= leForwarder.scala:35) =09at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) =09at org.apache.spark.sql.util.ExecutionListenerManager.org$apache$spark$s= ql$util$ExecutionListenerManager$$withErrorHandling(QueryExecutionListener.= scala:143) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply$mcV$sp(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager.readLock(QueryExec= utionListener.scala:156) =09at org.apache.spark.sql.util.ExecutionListenerManager.onSuccess(QueryExe= cutionListener.scala:122) =09at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3330) =09at org.apache.spark.sql.Dataset.count(Dataset.scala:2804) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw$$iw.= (command-1842411053765017:4) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw.(command-1842411053765017:51) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw.(c= ommand-1842411053765017:53) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw.(comma= nd-1842411053765017:55) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw.(command-1= 842411053765017:57) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw.(command-18424= 11053765017:59) =09at line587cf41c5b6a4080a07e49150407fa2732.$read.(command-184241105= 3765017:61) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-18424110= 53765017:65) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-184241= 1053765017) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print$lzycompute(:7) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print(:6) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval.$print() =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:62) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) =09at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047= ) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunR= eq$1.apply(IMain.scala:638) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunR= eq$1.apply(IMain.scala:637) =09at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaCla= ssLoader.scala:31) =09at scala.reflect.internal.util.AbstractFileClassLoader.asContext(Abstrac= tFileClassLoader.scala:19) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.= scala:637) =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) =09at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.= scala:199) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply$mcV$sp(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.tr= apExit(DriverLocal.scala:493) =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(Driv= erLocal.scala:448) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDrive= rLocal.scala:189) =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3.a= pply(DriverLocal.scala:248) =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3.a= pply(DriverLocal.scala:228) =09at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1= .apply(UsageLogging.scala:188) =09at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) =09at com.databricks.logging.UsageLogging$class.withAttributionContext(Usag= eLogging.scala:183) =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionConte= xt(DriverLocal.scala:40) =09at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLo= gging.scala:221) =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(= DriverLocal.scala:40) =09at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.= scala:228) =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecut= ingCommand$2.apply(DriverWrapper.scala:595) =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecut= ingCommand$2.apply(DriverWrapper.scala:595) =09at scala.util.Try$.apply(Try.scala:192) =09at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingComman= d(DriverWrapper.scala:590) =09at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAn= dError(DriverWrapper.scala:474) =09at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(Dri= verWrapper.scala:548) =09at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(Drive= rWrapper.scala:380) =09at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWra= pper.scala:327) =09at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.= scala:215) =09at java.lang.Thread.run(Thread.java:748){code} was: Caching and counting any DataSet causes an AssertionError in the log4j logs= . The results are still valid and no stderr is shown.=C2=A0 Example code:=20 {code:scala} var df =3D Seq(100).toDF("count") df.cache() df.count() {code} log4j Output:=20 {code:log} 18/11/02 15:39:57 WARN ExecutionListenerManager: Error executing query exec= ution listener java.lang.AssertionError: assertion failed: InMemoryRelation fields: output= , useCompression, batchSize, storageLevel, child, tableName, _cachedColumnB= uffers, rowCountStats, sizeInBytesStats, statsOfPlanToCache, outputOrdering= , values: List(count#124), true, 10000, StorageLevel(disk, memory, deserial= ized, 1 replicas), LocalTableScan [count#47] , None, LocalTableScan [count#47] MapPartitionsRDD[2] at cache at command-1842411053765017:3, LongAccumulato= r(id: 0, name: None, value: 1), LongAccumulator(id: 1, name: None, value: 4= ), Statistics(sizeInBytes=3D12.0 B, hints=3Dnone) =09at scala.Predef$.assert(Predef.scala:170) =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.scal= a:638) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:626) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$spar= k$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:62= 8) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$cat= alyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala= :631) =09at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:61= 7) =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger$.getSQLQueryP= lanBlob(SQLQueryPlanLogger.scala:70) =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger.onSuccess(SQL= QueryPlanLogger.scala:42) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:124) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apach= e$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryE= xecutionListener.scala:145) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apach= e$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(QueryE= xecutionListener.scala:143) =09at scala.collection.immutable.List.foreach(List.scala:381) =09at scala.collection.generic.TraversableForwarder$class.foreach(Traversab= leForwarder.scala:35) =09at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) =09at org.apache.spark.sql.util.ExecutionListenerManager.org$apache$spark$s= ql$util$ExecutionListenerManager$$withErrorHandling(QueryExecutionListener.= scala:143) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply$mcV$sp(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSuccess= $1.apply(QueryExecutionListener.scala:123) =09at org.apache.spark.sql.util.ExecutionListenerManager.readLock(QueryExec= utionListener.scala:156) =09at org.apache.spark.sql.util.ExecutionListenerManager.onSuccess(QueryExe= cutionListener.scala:122) =09at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3330) =09at org.apache.spark.sql.Dataset.count(Dataset.scala:2804) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw$$iw.= (command-1842411053765017:4) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw.(command-1842411053765017:51) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw.(c= ommand-1842411053765017:53) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw.(comma= nd-1842411053765017:55) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw.(command-1= 842411053765017:57) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw.(command-18424= 11053765017:59) =09at line587cf41c5b6a4080a07e49150407fa2732.$read.(command-184241105= 3765017:61) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-18424110= 53765017:65) =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-184241= 1053765017) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print$lzycompute(:7) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print(:6) =09at line587cf41c5b6a4080a07e49150407fa2732.$eval.$print() =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:62) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) =09at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047= ) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunR= eq$1.apply(IMain.scala:638) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunR= eq$1.apply(IMain.scala:637) =09at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaCla= ssLoader.scala:31) =09at scala.reflect.internal.util.AbstractFileClassLoader.asContext(Abstrac= tFileClassLoader.scala:19) =09at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.= scala:637) =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) =09at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.= scala:199) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply$mcV$sp(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1= .apply(ScalaDriverLocal.scala:189) =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.tr= apExit(DriverLocal.scala:493) =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(Driv= erLocal.scala:448) =09at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDrive= rLocal.scala:189) =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3.a= pply(DriverLocal.scala:248) =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3.a= pply(DriverLocal.scala:228) =09at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1= .apply(UsageLogging.scala:188) =09at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) =09at com.databricks.logging.UsageLogging$class.withAttributionContext(Usag= eLogging.scala:183) =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionConte= xt(DriverLocal.scala:40) =09at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLo= gging.scala:221) =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(= DriverLocal.scala:40) =09at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.= scala:228) =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecut= ingCommand$2.apply(DriverWrapper.scala:595) =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecut= ingCommand$2.apply(DriverWrapper.scala:595) =09at scala.util.Try$.apply(Try.scala:192) =09at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingComman= d(DriverWrapper.scala:590) =09at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAn= dError(DriverWrapper.scala:474) =09at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(Dri= verWrapper.scala:548) =09at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(Drive= rWrapper.scala:380) =09at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWra= pper.scala:327) =09at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.= scala:215) =09at java.lang.Thread.run(Thread.java:748){code} > Action on cached dataset causes WARN java.lang.AssertionError in log4j lo= gs. > -------------------------------------------------------------------------= --- > > Key: SPARK-25938 > URL: https://issues.apache.org/jira/browse/SPARK-25938 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 2.3.1 > Environment: We're running=C2=A0Apache Spark 2.3.1 on Databricks = 4.2 while developing in Scala > Reporter: Benedikt Beckermann > Priority: Minor > > Caching and counting any DataSet causes an AssertionError in the log4j lo= gs. The results are still valid and no stderr is shown.=C2=A0 > Example code:=20 > {code:scala} > var df =3D Seq(100).toDF("count") > df.cache() > df.count() > {code} > log4j Output:=20 > {code} > 18/11/02 15:39:57 WARN ExecutionListenerManager: Error executing query ex= ecution listener > java.lang.AssertionError: assertion failed: InMemoryRelation fields: outp= ut, useCompression, batchSize, storageLevel, child, tableName, _cachedColum= nBuffers, rowCountStats, sizeInBytesStats, statsOfPlanToCache, outputOrderi= ng, values: List(count#124), true, 10000, StorageLevel(disk, memory, deseri= alized, 1 replicas), LocalTableScan [count#47] > , None, LocalTableScan [count#47] > MapPartitionsRDD[2] at cache at command-1842411053765017:3, LongAccumula= tor(id: 0, name: None, value: 1), LongAccumulator(id: 1, name: None, value:= 4), Statistics(sizeInBytes=3D12.0 B, hints=3Dnone) > =09at scala.Predef$.assert(Predef.scala:170) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.sc= ala:638) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$c= atalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:626) > =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$sp= ark$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:= 628) > =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$sp= ark$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:= 628) > =09at scala.collection.immutable.List.foreach(List.scala:381) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$c= atalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) > =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$sp= ark$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:= 628) > =09at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$org$apache$sp= ark$sql$catalyst$trees$TreeNode$$collectJsonValue$1$1.apply(TreeNode.scala:= 628) > =09at scala.collection.immutable.List.foreach(List.scala:381) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$c= atalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:628) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.sca= la:631) > =09at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:= 617) > =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger$.getSQLQuer= yPlanBlob(SQLQueryPlanLogger.scala:70) > =09at com.databricks.backend.daemon.driver.SQLQueryPlanLogger.onSuccess(S= QLQueryPlanLogger.scala:42) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSucce= ss$1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:124) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSucce= ss$1$$anonfun$apply$mcV$sp$1.apply(QueryExecutionListener.scala:123) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apa= che$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(Quer= yExecutionListener.scala:145) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$org$apa= che$spark$sql$util$ExecutionListenerManager$$withErrorHandling$1.apply(Quer= yExecutionListener.scala:143) > =09at scala.collection.immutable.List.foreach(List.scala:381) > =09at scala.collection.generic.TraversableForwarder$class.foreach(Travers= ableForwarder.scala:35) > =09at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) > =09at org.apache.spark.sql.util.ExecutionListenerManager.org$apache$spark= $sql$util$ExecutionListenerManager$$withErrorHandling(QueryExecutionListene= r.scala:143) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSucce= ss$1.apply$mcV$sp(QueryExecutionListener.scala:123) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSucce= ss$1.apply(QueryExecutionListener.scala:123) > =09at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$onSucce= ss$1.apply(QueryExecutionListener.scala:123) > =09at org.apache.spark.sql.util.ExecutionListenerManager.readLock(QueryEx= ecutionListener.scala:156) > =09at org.apache.spark.sql.util.ExecutionListenerManager.onSuccess(QueryE= xecutionListener.scala:122) > =09at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3330) > =09at org.apache.spark.sql.Dataset.count(Dataset.scala:2804) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw$$i= w.(command-1842411053765017:4) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw$$iw.(command-1842411053765017:51) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw$$iw.= (command-1842411053765017:53) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw$$iw.(com= mand-1842411053765017:55) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw$$iw.(command= -1842411053765017:57) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$$iw.(command-184= 2411053765017:59) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read.(command-1842411= 053765017:61) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-184241= 1053765017:65) > =09at line587cf41c5b6a4080a07e49150407fa2732.$read$.(command-1842= 411053765017) > =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print$lzycompute(:7) > =09at line587cf41c5b6a4080a07e49150407fa2732.$eval$.$print(:6) > =09at line587cf41c5b6a4080a07e49150407fa2732.$eval.$print() > =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp= l.java:62) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:498) > =09at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:78= 6) > =09at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:10= 47) > =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRu= nReq$1.apply(IMain.scala:638) > =09at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRu= nReq$1.apply(IMain.scala:637) > =09at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaC= lassLoader.scala:31) > =09at scala.reflect.internal.util.AbstractFileClassLoader.asContext(Abstr= actFileClassLoader.scala:19) > =09at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMai= n.scala:637) > =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) > =09at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) > =09at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoo= p.scala:199) > =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl= $1.apply$mcV$sp(ScalaDriverLocal.scala:189) > =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl= $1.apply(ScalaDriverLocal.scala:189) > =09at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl= $1.apply(ScalaDriverLocal.scala:189) > =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.= trapExit(DriverLocal.scala:493) > =09at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(Dr= iverLocal.scala:448) > =09at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDri= verLocal.scala:189) > =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3= .apply(DriverLocal.scala:248) > =09at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$3= .apply(DriverLocal.scala:228) > =09at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext= $1.apply(UsageLogging.scala:188) > =09at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > =09at com.databricks.logging.UsageLogging$class.withAttributionContext(Us= ageLogging.scala:183) > =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionCon= text(DriverLocal.scala:40) > =09at com.databricks.logging.UsageLogging$class.withAttributionTags(Usage= Logging.scala:221) > =09at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTag= s(DriverLocal.scala:40) > =09at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLoca= l.scala:228) > =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExec= utingCommand$2.apply(DriverWrapper.scala:595) > =09at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExec= utingCommand$2.apply(DriverWrapper.scala:595) > =09at scala.util.Try$.apply(Try.scala:192) > =09at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingComm= and(DriverWrapper.scala:590) > =09at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutput= AndError(DriverWrapper.scala:474) > =09at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(D= riverWrapper.scala:548) > =09at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(Dri= verWrapper.scala:380) > =09at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverW= rapper.scala:327) > =09at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrappe= r.scala:215) > =09at java.lang.Thread.run(Thread.java:748){code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org