spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mario Briggs (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-16545) Structured Streaming : foreachSink creates the Physical Plan multiple times per TriggerInterval
Date Thu, 14 Jul 2016 10:54:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-16545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376758#comment-15376758
] 

Mario Briggs edited comment on SPARK-16545 at 7/14/16 10:53 AM:
----------------------------------------------------------------

While looking at the performance of Structured streaming, found some excessive time being
spent in the driver. 

Further looking into this, found the time spent in multiple (3 to be exact) initialisations
of [QueryExecution.executedPlan|https://github.com/mariobriggs/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala#L85]
due to multiple instances of QueryExecution created in the forEachSink.addBatch. 

Creation of physical plan involves more time and hence shouldn't be done more than once


was (Author: mariobriggs):
While looking at the performance of Structured streaming, found some excessive time being
spent in the driver. 

Further looking into this, found the time spent in multiple (3 to be exact) initialisations
of QueryExecution.executedPlan due to multiple instances of QueryExecution created in the
forEachSink.addBatch. 

Creation of physical plan involves more time and hence shouldn't be done more than once

> Structured Streaming : foreachSink creates the Physical Plan multiple times per TriggerInterval

> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16545
>                 URL: https://issues.apache.org/jira/browse/SPARK-16545
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL, Streaming
>    Affects Versions: 2.0.0
>            Reporter: Mario Briggs
>             Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message