spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: getting a list of executors for use in getPreferredLocations
Date Thu, 03 Mar 2016 23:10:36 GMT
What do you mean by consistent? Throughout the life cycle of an app, the
executors can come and go and as a result really has no consistency. Do you
just need it for a specific job?



On Thu, Mar 3, 2016 at 3:08 PM, Cody Koeninger <cody@koeninger.org> wrote:

> I need getPreferredLocations to choose a consistent executor for a
> given partition in a stream.  In order to do that, I need to know what
> the current executors are.
>
> I'm currently grabbing them from the block manager master .getPeers(),
> which works, but I don't know if that's the most reasonable way to do
> it.
>
> Relevant code:
>
>
> https://github.com/koeninger/spark-1/blob/aaef0fc6e7e3aae18e4e03271bc0707d09d243e4/external/kafka-beta/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L107
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message