spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject Why does Spark SQL use custom local property not SparkContext.setJobGroup?
Date Thu, 22 Jun 2017 01:16:39 GMT

Just noticed that Spark SQL uses local property
(via SQLExecution.withNewExecutionId [1]) to group Spark jobs
logically together while Structured Streaming uses
SparkContext.setJobGroup [2] to do the same.

I think Structured Streaming is more correct as it uses what Spark
Core introduced and uses in web UI (without introducing a custom

Why does Spark SQL introduce a custom solution based on local property? What's wrong with


Jacek Laskowski
Mastering Apache Spark 2
Follow me at

To unsubscribe e-mail:

View raw message