spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aakash Basu <aakash.spark....@gmail.com>
Subject Wait for 30 seconds before terminating Spark Streaming
Date Wed, 21 Mar 2018 10:41:13 GMT
Hi,

Using: *Spark 2.3 + Kafka 0.10*


How to wait for 30 seconds after the latest stream and if there's no more
streaming data, gracefully exit.

Is it running -

query.awaitTermination(30)


Or is it something else?

I tried with this, keeping -

option("startingOffsets", "latest")

for both my input streams be joined.

Am first running the Spark job, and then pushing both the csv file data
into the two respective Kafka topics, but getting the following error -


ERROR MicroBatchExecution:91 - Query [id =
21c96e5c-770d-4d59-8893-4401217120b6, runId =
e03d7ac9-97b6-442e-adf4-dd232f9ed616] terminated with error
org.apache.spark.SparkException: Writing job aborted.


When I keep -

option("startingOffsets", "earliest")

The first batch output works perfectly and then terminates after the given
time.

Please help!

Thanks,
Aakash.

Mime
View raw message