spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Toupin <>
Subject Concurrency issue in SQLExecution.withNewExecutionId
Date Thu, 10 Sep 2015 16:09:18 GMT
Look at this code:


This exception is there to prevent "nested `withNewExecutionId`" but what if
there is two concurrent commands that happens to run on the same thread?
Then the thread local getLocalProperty will returns an execution id,
triggering that exception.

This is not hypothetical,  one of our spark job crash randomly with the
following stack trace (Using Spark 1.5, it ran without problem in Spark

java.lang.IllegalArgumentException: is already set
	at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904)
	at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385)

Also imagine the following:

future { df1.count() }
future { df2.count() }

Could we double check this if this an issue?

View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message