spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Ravi Kumar <reachb...@gmail.com>
Subject Re: Spark on Mesos / Executor Memory
Date Sat, 17 Oct 2015 05:47:38 GMT
Can someone respond if you're aware of the reason for such a memory
footprint? It seems unintuitive and hard to reason about.

Thanks,
Bharath

On Thu, Oct 15, 2015 at 12:29 PM, Bharath Ravi Kumar <reachbach@gmail.com>
wrote:

> Resending since user@mesos bounced earlier. My apologies.
>
> On Thu, Oct 15, 2015 at 12:19 PM, Bharath Ravi Kumar <reachbach@gmail.com>
> wrote:
>
>> (Reviving this thread since I ran into similar issues...)
>>
>> I'm running two spark jobs (in mesos fine grained mode), each belonging
>> to a different mesos role, say low and high. The low:high mesos weights are
>> 1:10. On expected lines, I see that the low priority job occupies cluster
>> resources to the maximum extent when running alone. However, when the high
>> priority job is submitted, it does not start and continues to await cluster
>> resources (as seen in the logs). Since the jobs run in fine grained mode
>> and the low priority tasks begin to finish, the high priority job should
>> ideally be able to start and gradually take over cluster resources as per
>> the weights. However, I noticed that while the "low" job gives up CPU cores
>> with each completing task (e.g. reduction from 72 -> 12 with default
>> parallelism set to 72), the memory resources are held on (~500G out of
>> 768G). The spark.executor.memory setting appears to directly impact the
>> amount of memory that the job holds on to. In this case, it was set to 200G
>> in the low priority task and 100G in the high priority task. The nature of
>> these jobs is such that setting the numbers to smaller values (say 32g)
>> resulted in job failures with outofmemoryerror.  It appears that the spark
>> framework is retaining memory (across tasks)  proportional to
>> spark.executor.memory for the duration of the job and not releasing memory
>> as tasks complete. This defeats the purpose of fine grained mode execution
>> as the memory occupancy is preventing the high priority job from accepting
>> the prioritized cpu offers and beginning execution. Can this be explained /
>> documented better please?
>>
>> Thanks,
>> Bharath
>>
>> On Sat, Apr 11, 2015 at 10:59 PM, Tim Chen <tim@mesosphere.io> wrote:
>>
>>> (Adding spark user list)
>>>
>>> Hi Tom,
>>>
>>> If I understand correctly you're saying that you're running into memory
>>> problems because the scheduler is allocating too much CPUs and not enough
>>> memory to acoomodate them right?
>>>
>>> In the case of fine grain mode I don't think that's a problem since we
>>> have a fixed amount of CPU and memory per task.
>>> However, in coarse grain you can run into that problem if you're with in
>>> the spark.cores.max limit, and memory is a fixed number.
>>>
>>> I have a patch out to configure how much max cpus should coarse grain
>>> executor use, and it also allows multiple executors in coarse grain mode.
>>> So you could say try to launch multiples of max 4 cores with
>>> spark.executor.memory (+ overhead and etc) in a slave. (
>>> https://github.com/apache/spark/pull/4027)
>>>
>>> It also might be interesting to include a cores to memory multiplier so
>>> that with a larger amount of cores we try to scale the memory with some
>>> factor, but I'm not entirely sure that's intuitive to use and what people
>>> know what to set it to, as that can likely change with different workload.
>>>
>>> Tim
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Apr 11, 2015 at 9:51 AM, Tom Arnfeld <tom@duedil.com> wrote:
>>>
>>>> We're running Spark 1.3.0 (with a couple of patches over the top for
>>>> docker related bits).
>>>>
>>>> I don't think SPARK-4158 is related to what we're seeing, things do run
>>>> fine on the cluster, given a ridiculously large executor memory
>>>> configuration. As for SPARK-3535 although that looks useful I think we'e
>>>> seeing something else.
>>>>
>>>> Put a different way, the amount of memory required at any given time by
>>>> the spark JVM process is directly proportional to the amount of CPU it has,
>>>> because more CPU means more tasks and more tasks means more memory. Even
if
>>>> we're using coarse mode, the amount of executor memory should be
>>>> proportionate to the amount of CPUs in the offer.
>>>>
>>>> On 11 April 2015 at 17:39, Brenden Matthews <brenden@diddyinc.com>
>>>> wrote:
>>>>
>>>>> I ran into some issues with it a while ago, and submitted a couple PRs
>>>>> to fix it:
>>>>>
>>>>> https://github.com/apache/spark/pull/2401
>>>>> https://github.com/apache/spark/pull/3024
>>>>>
>>>>> Do these look relevant? What version of Spark are you running?
>>>>>
>>>>> On Sat, Apr 11, 2015 at 9:33 AM, Tom Arnfeld <tom@duedil.com> wrote:
>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> Not sure whether it's best to ask this on the spark mailing list
or
>>>>>> the mesos one, so I'll try here first :-)
>>>>>>
>>>>>> I'm having a bit of trouble with out of memory errors in my spark
>>>>>> jobs... it seems fairly odd to me that memory resources can only
be set at
>>>>>> the executor level, and not also at the task level. For example,
as far as
>>>>>> I can tell there's only a *spark.executor.memory* config option.
>>>>>>
>>>>>> Surely the memory requirements of a single executor are quite
>>>>>> dramatically influenced by the number of concurrent tasks running?
Given a
>>>>>> shared cluster, I have no idea what % of an individual slave my executor
is
>>>>>> going to get, so I basically have to set the executor memory to a
value
>>>>>> that's correct when the whole machine is in use...
>>>>>>
>>>>>> Has anyone else running Spark on Mesos come across this, or maybe
>>>>>> someone could correct my understanding of the config options?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Tom.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message