spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Burak Yavuz (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-22238) EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed
Date Tue, 10 Oct 2017 22:07:00 GMT
Burak Yavuz created SPARK-22238:
-----------------------------------

             Summary: EnsureStatefulOpPartitioning shouldn't ask for the child RDD before
planning is completed
                 Key: SPARK-22238
                 URL: https://issues.apache.org/jira/browse/SPARK-22238
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.3.0
            Reporter: Burak Yavuz
            Assignee: Burak Yavuz


In EnsureStatefulOpPartitioning, we check that the inputRDD to a SparkPlan has the expected
partitioning for Streaming Stateful Operators. The problem is that we are not allowed to access
this information during planning.

The reason we added that check was because CoalesceExec could actually create RDDs with 0
partitions. We should fix it such that when CoalesceExec says that there is a SinglePartition,
there is in fact an inputRDD of 1 partition instead of 0 partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message