spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nezih Yigitbasi <nyigitb...@netflix.com.INVALID>
Subject Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Date Thu, 14 Apr 2016 16:31:58 GMT
Thanks Imran. I will give it a shot when I have some time.

Nezih

On Thu, Apr 14, 2016 at 9:25 AM Imran Rashid <irashid@cloudera.com> wrote:

> Hi Nezih,
>
> I just reported a somewhat similar issue, and I have a potential fix --
> SPARK-14560, looks like you are already watching it :).  You can try out
> that patch, you have to explicitly enable the change in behavior with
> "spark.shuffle.spillAfterRead=true".  Honestly, I don't think these issues
> are the same, as I've always seen that case lead to acquiring 0 bytes,
> while in your case you are requesting GBs and getting something pretty
> close, so my hunch is that it is different ... but might be worth a shot to
> see if it is the issue.
>
> Turning on debug logging for TaskMemoryManager might help track the root
> cause -- you'll get information on which consumers are using memory and
> when there are spill attempts.  (Note that even if the patch I have for
> SPARK-14560 doesn't fix your issue, it might still make those debug logs a
> bit more clear, since it'll report memory used by Spillables.)
>
> Imran
>
> On Mon, Apr 4, 2016 at 10:52 PM, Nezih Yigitbasi <
> nyigitbasi@netflix.com.invalid> wrote:
>
>> Nope, I didn't have a chance to track the root cause, and IIRC we didn't
>> observe it when dyn. alloc. is off.
>>
>> On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <rxin@databricks.com> wrote:
>>
>>> BTW do you still see this when dynamic allocation is off?
>>>
>>> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rxin@databricks.com> wrote:
>>>
>>>> Nezih,
>>>>
>>>> Have you had a chance to figure out why this is happening?
>>>>
>>>>
>>>> On Tue, Mar 22, 2016 at 1:32 AM, james <yiazhou@gmail.com> wrote:
>>>>
>>>>> I guess different workload cause diff result ?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>

Mime
View raw message