spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Gummelt <mgumm...@mesosphere.io>
Subject Re: Two questions about running spark on mesos
Date Mon, 14 Nov 2016 18:40:37 GMT
1. I had never even heard of conf/slaves until this email, and I only see
it referenced in the docs next to Spark Standalone, so I doubt that works.

2. Yes.  See the --kill option in spark-submit.

Also, we're considering dropping the Spark dispatcher in DC/OS in favor of
Metronome, which will be our consolidated method of running any one-off
jobs.  The dispatcher is really just a lesser maintained and more
feature-sparse metronome.  If I were you, I would look into running
Metronome rather than the dispatcher (or just run DC/OS).

On Mon, Nov 14, 2016 at 3:10 AM, Yu Wei <yu2003w@hotmail.com> wrote:

> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
>     According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
>     However, it doesn't take effect when deploying in cluster mode.
>
>     Is this expected behavior?
>
>    Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
>     I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
>     Then I want to kill the job.
>     How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
> Thanks,
>
> Jared, (韦煜)
> Software developer
> Interested in open source software, big data, Linux
>



-- 
Michael Gummelt
Software Engineer
Mesosphere

Mime
View raw message