mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Bessalah <samkiller....@gmail.com>
Subject Re: Spark (Streaming?) holding on to Mesos resources
Date Tue, 27 Jan 2015 18:39:17 GMT
Hi Geraard,
isn't this the same issueas this?
https://issues.apache.org/jira/browse/MESOS-1688

On Mon, Jan 26, 2015 at 9:17 PM, Gerard Maas <gerard.maas@gmail.com> wrote:

> Hi,
>
> We are observing with certain regularity that our Spark  jobs, as Mesos
> framework, are hoarding resources and not releasing them, resulting in
> resource starvation to all jobs running on the Mesos cluster.
>
> For example:
> This is a job that has spark.cores.max = 4 and spark.executor.memory="3g"
>
> IDFrameworkHostCPUsMem…5050-16506-1146497FooStreamingdnode-4.hdfs.private713.4
> GB…5050-16506-1146495FooStreaming
> dnode-0.hdfs.private16.4 GB…5050-16506-1146491FooStreaming
> dnode-5.hdfs.private711.9 GB…5050-16506-1146449FooStreaming
> dnode-3.hdfs.private74.9 GB…5050-16506-1146247FooStreaming
> dnode-1.hdfs.private0.55.9 GB…5050-16506-1146226FooStreaming
> dnode-2.hdfs.private37.9 GB…5050-16506-1144069FooStreaming
> dnode-3.hdfs.private18.7 GB…5050-16506-1133091FooStreaming
> dnode-5.hdfs.private11.7 GB…5050-16506-1133090FooStreaming
> dnode-2.hdfs.private55.2 GB…5050-16506-1133089FooStreaming
> dnode-1.hdfs.private6.56.3 GB…5050-16506-1133088FooStreaming
> dnode-4.hdfs.private1251 MB…5050-16506-1133087FooStreaming
> dnode-0.hdfs.private6.46.8 GB
> The only way to release the resources is by manually finding the process
> in the cluster and killing it. The jobs are often streaming but also batch
> jobs show this behavior. We have more streaming jobs than batch, so stats
> are biased.
> Any ideas of what's up here? Hopefully some very bad ugly bug that has
> been fixed already and that will urge us to upgrade our infra?
>
> Mesos 0.20 +  Marathon 0.7.4 + Spark 1.1.0
>
> -kr, Gerard.
>

Mime
View raw message