Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 13 Sep 2017 07:45:01 +0000 (UTC)
From: "Wenchen Fan (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13101764.1505247162000.99333.1505288701459@Atlassian.JIRA>
In-Reply-To: <JIRA.13101764.1505247162000@Atlassian.JIRA>
References: <JIRA.13101764.1505247162000@Atlassian.JIRA> <JIRA.13101764.1505247162863@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-21987) Spark 2.3 cannot read 2.2 event
 logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 13 Sep 2017 07:45:07 -0000


    [ https://issues.apache.org/jira/browse/SPARK-21987?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D161=
64260#comment-16164260 ]=20

Wenchen Fan commented on SPARK-21987:
-------------------------------------

Adding back the `SparkPlanGraph.metadata` is one solution, and we can also =
annotate `SparkPlanGraph` with {{@JsonIgnoreProperties(ignoreUnknown =3D tr=
ue)}}. IMO keeping the {{metadata}} field is a little ugly, since `SparkPla=
nGraph` is a developer API, and we should be able to remove some fields if =
necessary.

> Spark 2.3 cannot read 2.2 event logs
> ------------------------------------
>
>                 Key: SPARK-21987
>                 URL: https://issues.apache.org/jira/browse/SPARK-21987
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Marcelo Vanzin
>            Priority: Blocker
>
> Reported by [~jincheng] in a comment in SPARK-18085:
> {noformat}
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecog=
nized field "metadata" (class org.apache.spark.sql.execution.SparkPlanInfo)=
, not marked as ignorable (4 known properties: "simpleString", "nodeName", =
"children", "metrics"])
>  at [Source: {"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQL=
ExecutionStart","executionId":0,"description":"json at NativeMethodAccessor=
Impl.java:0","details":"org.apache.spark.sql.DataFrameWriter.json(DataFrame=
Writer.scala:487)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Meth=
od)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j=
ava:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc=
cessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\npy4j=
.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\npy4j.reflection.R=
eflectionEngine.invoke(ReflectionEngine.java:357)\npy4j.Gateway.invoke(Gate=
way.java:280)\npy4j.commands.AbstractCommand.invokeMethod(AbstractCommand.j=
ava:132)\npy4j.commands.CallCommand.execute(CallCommand.java:79)\npy4j.Gate=
wayConnection.run(GatewayConnection.java:214)\njava.lang.Thread.run(Thread.=
java:748)","physicalPlanDescription":"=3D=3D Parsed Logical Plan =3D=3D\nRe=
partition 200, true\n+- LogicalRDD [uid#327L, gids#328]\n\n=3D=3D Analyzed =
Logical Plan =3D=3D\nuid: bigint, gids: array<bigint>\nRepartition 200, tru=
e\n+- LogicalRDD [uid#327L, gids#328]\n\n=3D=3D Optimized Logical Plan =3D=
=3D\nRepartition 200, true\n+- LogicalRDD [uid#327L, gids#328]\n\n=3D=3D Ph=
ysical Plan =3D=3D\nExchange RoundRobinPartitioning(200)\n+- Scan ExistingR=
DD[uid#327L,gids#328]","sparkPlanInfo":{"nodeName":"Exchange","simpleString=
":"Exchange RoundRobinPartitioning(200)","children":[{"nodeName":"ExistingR=
DD","simpleString":"Scan ExistingRDD[uid#327L,gids#328]","children":[],"met=
adata":{},"metrics":[{"name":"number of output rows","accumulatorId":140,"m=
etricType":"sum"}]}],"metadata":{},"metrics":[{"name":"data size total (min=
, med, max)","accumulatorId":139,"metricType":"size"}]},"time":150483705294=
8}; line: 1, column: 1622] (through reference chain: org.apache.spark.sql.e=
xecution.ui.SparkListenerSQLExecutionStart["sparkPlanInfo"]->org.apache.spa=
rk.sql.execution.SparkPlanInfo["children"]->com.fasterxml.jackson.module.sc=
ala.deser.BuilderWrapper[0]->org.apache.spark.sql.execution.SparkPlanInfo["=
metadata"])
> =09at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.fr=
om(UnrecognizedPropertyException.java:51)
> {noformat}
> This was caused by SPARK-17701 (which at this moment is still open even t=
hough the patch has been committed).


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org