spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <>
Subject [jira] [Resolved] (SPARK-23565) Improved error message for when the number of sources for a query changes
Date Fri, 27 Apr 2018 15:06:00 GMT


Shixiong Zhu resolved SPARK-23565.
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 20946

> Improved error message for when the number of sources for a query changes
> -------------------------------------------------------------------------
>                 Key: SPARK-23565
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.3.0
>            Reporter: Patrick McGloin
>            Priority: Minor
>             Fix For: 2.4.0
> If you change the number of sources for a Structured Streaming query then you will get
an assertion error as the number of sources in the checkpoint does not match the number of
sources in the query that is starting.  This can happen if, for example, you add a union
to the input of the query.  This is of course correct but the error is a bit cryptic and
requires investigation.
> Suggestion for a more informative error message =>
> The number of sources for this query has changed.  There are [x] sources in the checkpoint offsets
and now there are [y] sources requested by the query.  Cannot continue.
> This is the current message.
> 02-03-2018 13:14:22 ERROR StreamExecution:91 - Query ORPositionsState to Kafka [id =
35f71e63-dbd0-49e9-98b2-a4c72a7da80e, runId = d4439aca-549c-4ef6-872e-29fbfde1df78] terminated
with error java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:156)
at org.apache.spark.sql.execution.streaming.OffsetSeq.toStreamProgress(OffsetSeq.scala:38)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(StreamExecution.scala:297)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1$$anonfun$apply$mcZ$sp$1.apply(StreamExecution.scala:294)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1$$anonfun$apply$mcZ$sp$1.apply(StreamExecution.scala:294)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:279)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1.apply$mcZ$sp(StreamExecution.scala:294)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message