spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sung Hwan Chung <coded...@cs.stanford.edu>
Subject Re: When does Spark switch from PROCESS_LOCAL to NODE_LOCAL or RACK_LOCAL?
Date Thu, 05 Jun 2014 22:53:07 GMT
Additionally, I've encountered some confusing situation where the locality
level for a task showed up as 'PROCESS_LOCAL' even though I didn't cache
the data. I wonder some implicit caching happens even without the user
specifying things.


On Thu, Jun 5, 2014 at 3:50 PM, Sung Hwan Chung <codedeft@cs.stanford.edu>
wrote:

> Thanks Andrew,
>
> Is there a chance that even with full-caching, that modes other than
> PROCESS_LOCAL will be used? E.g., let's say, an executor will try to
> perform tasks although the data are cached on a different executor.
>
> What I'd like to do is to prevent such a scenario entirely.
>
> I'd like to know if setting 'spark.locality.wait' to a very high value
> would guarantee that the mode will always be 'PROCESS_LOCAL'.
>
>
> On Thu, Jun 5, 2014 at 3:36 PM, Andrew Ash <andrew@andrewash.com> wrote:
>
>> The locality is how close the data is to the code that's processing it.
>>  PROCESS_LOCAL means data is in the same JVM as the code that's running, so
>> it's really fast.  NODE_LOCAL might mean that the data is in HDFS on the
>> same node, or in another executor on the same node, so is a little slower
>> because the data has to travel across an IPC connection.  RACK_LOCAL is
>> even slower -- data is on a different server so needs to be sent over the
>> network.
>>
>> Spark switches to lower locality levels when there's no unprocessed data
>> on a node that has idle CPUs.  In that situation you have two options: wait
>> until the busy CPUs free up so you can start another task that uses data on
>> that server, or start a new task on a farther away server that needs to
>> bring data from that remote place.  What Spark typically does is wait a bit
>> in the hopes that a busy CPU frees up.  Once that timeout expires, it
>> starts moving the data from far away to the free CPU.
>>
>> The main tunable option is how far long the scheduler waits before
>> starting to move data rather than code.  Those are the spark.locality.*
>> settings here: http://spark.apache.org/docs/latest/configuration.html
>>
>> If you want to prevent this from happening entirely, you can set the
>> values to ridiculously high numbers.  The documentation also mentions that
>> "0" has special meaning, so you can try that as well.
>>
>> Good luck!
>> Andrew
>>
>>
>> On Thu, Jun 5, 2014 at 3:13 PM, Sung Hwan Chung <codedeft@cs.stanford.edu
>> > wrote:
>>
>>> I noticed that sometimes tasks would switch from PROCESS_LOCAL (I'd
>>> assume that this means fully cached) to NODE_LOCAL or even RACK_LOCAL.
>>>
>>> When these happen things get extremely slow.
>>>
>>> Does this mean that the executor got terminated and restarted?
>>>
>>> Is there a way to prevent this from happening (barring the machine
>>> actually going down, I'd rather stick with the same process)?
>>>
>>
>>
>

Mime
View raw message