spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <>
Subject Re: context.runJob() was suspended in getPreferredLocations() function
Date Sun, 01 Jan 2017 12:34:41 GMT


Simply said, you submit another Job in the event thread which will be
blocked and can't receive the this job submission event. So your second job
submission is never processed, and the getPreferredLocations method is never

Fei Hu wrote
> Dear all,
> I tried to customize my own RDD. In the getPreferredLocations() function,
> I
> used the following code to query anonter RDD, which was used as an input
> to
> initialize this customized RDD:
>                    * val results: Array[Array[DataChunkPartition]] =
> context.runJob(partitionsRDD, (context: TaskContext, partIter:
> Iterator[DataChunkPartition]) => partIter.toArray, partitions, allowLocal
> =
> true)*
> The problem is that when executing the above code, the task seemed to be
> suspended. I mean the job just stopped at this code, but no errors and no
> outputs.
> What is the reason for it?
> Thanks,
> Fei

Liang-Chi Hsieh | @viirya 
Spark Technology Center 
View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe e-mail:

View raw message