livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: Session Memory Management
Date Fri, 01 Dec 2017 03:33:06 GMT
Hi Lauren,

Thanks for the feedbacks. How do you identify this REPL memory issue? If it
is a problem of Scala REPL, then I think we don't have pretty much
solutions for it. Do you have any thought on it?

BTW, which version of Spark are you using?

@Jeff, if it is a problem of Scala REPL, then I believe Zeppelin also
suffers from such issue, did you see any report from Zeppelin community?

Thanks
Jerry

2017-12-01 11:17 GMT+08:00 Lauren Spiegel <lauren.spiegel@scality.com>:

> The spark driver. I think it is related to this since a session is a spark
> shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
>
>
> On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
>>
>> Which component's memory keeps growing ? livy server or the spark driver ?
>>
>>
>> Lauren Spiegel <lauren.spiegel@scality.com>于2017年12月1日周五 上午8:55写道:
>>
>>> Hi Livyers,
>>>
>>> When I have an active session for a couple hours, the memory keeps
>>> growing and I have to kill it. What do people do to manage ever-growing
>>> memory? Are there any plans to limit the memory growth so that sessions can
>>> be long living?
>>>
>>> Thank you
>>>
>>>
>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>
>>
>
>
> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>

Mime
View raw message