Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3C8C819682 for ; Tue, 15 Mar 2016 16:22:30 +0000 (UTC) Received: (qmail 40402 invoked by uid 500); 15 Mar 2016 16:22:29 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 40339 invoked by uid 500); 15 Mar 2016 16:22:29 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 40329 invoked by uid 99); 15 Mar 2016 16:22:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2016 16:22:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 004311804C9 for ; Tue, 15 Mar 2016 16:22:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gruchalski-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Jy1KXA7pzzGK for ; Tue, 15 Mar 2016 16:22:26 +0000 (UTC) Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 892765FAE1 for ; Tue, 15 Mar 2016 16:22:26 +0000 (UTC) Received: by mail-wm0-f52.google.com with SMTP id p65so34381167wmp.0 for ; Tue, 15 Mar 2016 09:22:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gruchalski-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:message-id:in-reply-to:references:subject:mime-version; bh=IoV0Ainx7nXST8XwqTZ48oVBDfh2zl59x6Dl2kCthQc=; b=iWbm+kRJokkp4y6ELDZDRUHrw3IG6pqodC62ZrqD7qNYRxVNqvuvQ+A9+x831nvI0Q /nKQeZ+ohwtUtqMxrrK17IP80wdacAqciz0sW6MVT9gOaiwdbv1Lu4weywvF6Vx4LWBT 1sykT8XK+3fMDzmBhA8CP0kTxp3A65SYxJcCC+GSiRli8Gqfs/4vn3T9UnFHM8Wc97T3 fas2a2w8Wg80UjVsa+tOQey3xXSICU7sSVHGIXztwqcQrYGtMPj74clBYmoOM4IGO8hA wD6W+g6qET5AyZU1GWDETjSF0Kr3qeexmXZ1Uj/RUIKaDbT5e6wpCfZPUWHTYwwle/P6 8Q4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version; bh=IoV0Ainx7nXST8XwqTZ48oVBDfh2zl59x6Dl2kCthQc=; b=iW6mcMkWviou/5rshZ/N5dIXk3DMdGfKhlwMFs1vU2FaP3cG75+42B7lKNtShCCipJ ZBIMm/29E3FHM69El8KLOMIOcRgCWOMwHL3vgQG19Ip/+oGFqwSriXGugTt8cyJUE3Yn OW8dk2mqN8M+dSWAS3qUdUBJD6blsf9FmI2lZ56y9zibJ42I4DeUuJYC7YribzeA4DhS ilpi0fLHy8IG38sjdNooHQFlBue77UVv1QL8JFsSfBx6ufypK7dHs8XJvDjDjObVRwrl x9Y4SOlpS+rm5sMclaB1G1zwLebuU0GjP814tJVBxMOpr+/b7eik5NnJGxM7Ml8CdX1z apOA== X-Gm-Message-State: AD7BkJIBXEI8dDduU6/4y/jBsXLV2zPp1UASRWEn3FSczHne9Y6HwHy+s6oI4ns61g1cgw== X-Received: by 10.28.176.200 with SMTP id z191mr25124239wme.91.1458058946254; Tue, 15 Mar 2016 09:22:26 -0700 (PDT) Received: from [192.168.1.5] (d8D864992.access.telenet.be. [141.134.73.146]) by smtp.gmail.com with ESMTPSA id js8sm27758607wjc.37.2016.03.15.09.22.25 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 15 Mar 2016 09:22:25 -0700 (PDT) Date: Tue, 15 Mar 2016 17:22:25 +0100 From: Rad Gruchalski To: user@mesos.apache.org Message-ID: <84ADB257F2FB49C69EF0D4F212FA6204@gruchalski.com> In-Reply-To: <7E3F1713-CD24-495E-8131-C480794976C3@mesosphere.io> References: <7E3F1713-CD24-495E-8131-C480794976C3@mesosphere.io> Subject: Re: Apache Spark Over Mesos X-Mailer: sparrow 1.6.4 (build 1178) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="56e836c1_275ac794_322a" --56e836c1_275ac794_322a Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline That=E2=80=99s a great alternative=21 Kind regards,=E2=80=A8 Radek Gruchalski =E2=80=A8radek=40gruchalski.com (mailto:radek=40gruchalski.com)=E2=80=A8 = (mailto:radek=40gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalsk= i/) Confidentiality: This communication is intended for the above-named person and may be conf= idential and/or legally privileged. If it has come to you in error you must take no action based on it, nor m= ust you copy or show it to anyone; please delete/destroy and inform the s= ender immediately. On Tuesday, 15 March 2016 at 17:19, Timothy Chen wrote: > You can launch the driver and executor in docker containers as well by = setting spark.mesos.executor.docker.image to the image you want to use to= launch them. > =20 > Tim > =20 > On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski wrote: > =20 > > Pradeep, > > =20 > > You can mount a spark directory as a volume. This means you have to h= ave spark deployed on every agent. > > =20 > > Another thing you can do, place spark in hdfs, assuming that you have= hdfs available but that too will download a copy to the sandbox. > > =20 > > I'd prefer the former. > > =20 > > Sent from Outlook Mobile (https://aka.ms/qtex0l) > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F > > =46rom: Pradeep Chhetri > > Sent: Tuesday, March 15, 2016 4:41 pm > > Subject: Apache Spark Over Mesos > > To: > > =20 > > =20 > > Hello, =20 > > =20 > > I am able to run Apache Spark over Mesos. Its quite simple to run Spa= rk Dispatcher over marathon and ask it to run Spark Executor (I guess als= o can be called as Spark Driver) as docker container. =20 > > =20 > > I have a query regarding this: =20 > > =20 > > All spark tasks are spawned directly by first downloading the spark a= rtifacts. I was thinking if there is some way I can start them too as doc= ker containers. This will save the time for downloading the spark artifac= ts. I am running spark in fine-grained mode. =20 > > =20 > > I have attached a screenshot of a sample job =20 > > =20 > > =20 > > =E2=80=8B =20 > > Thanks, =20 > > =20 > > -- =20 > > Pradeep Chhetri =20 > > =20 --56e836c1_275ac794_322a Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
That=E2=80=99s a great alternative=21
=09 =09 =09

=09 =09 =09

=09 =09 =09

=09 =09 =09

= Kind regards,=E2=80=A8
Radek Gruchalski
=E2=80=A8radek=40gruchalski.com=E2=80=A8
de.linkedin.com/in/radgruch= alski/

Confidentiality:This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.

=20

On Tuesday, 15 March 2= 016 at 17:19, Timothy Chen wrote:

You can launch the drive= r and executor in docker containers as well by setting spark.mesos.execut= or.docker.image to the image you want to use to launch them.
Tim

On Mar 15, 2016, at 8:49 AM, Radoslaw = Gruchalski <radek=40gruc= halski.com> wrote:

Pradeep,

You can mount a spark = directory as a volume. This means you have to have spark deployed on ever= y agent.

Another thing you can do, place spark i= n hdfs, assuming that you have hdfs available but that too will download = a copy to the sandbox.

I'd prefer the former.

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F
=46rom: Pradeep Chhetri <pradeep.chhetri89=40gmail.com>
Sent: Tuesday= , March 15, 2016 4:41 pm
Subject: Apache Spark Over Mesos
To: <= user=40mesos.apache.org>

=
Hello,

= I am able to run Apache Spark over Mesos. Its quite simple to run Spark= Dispatcher over marathon and ask it to run Spark Executor (I guess also = can be called as Spark Driver) as docker container.
<= br>
I have a query regarding this:
=
All spark tasks are spawned directly by first = downloading the spark artifacts. I was thinking if there is some way I ca= n start them too as docker containers. This will save the time for downlo= ading the spark artifacts. I am running spark in fine-grained mode.

I have attached a screenshot of a= sample job

<= ;Screen Shot 2016-03-15 at 15.15.06.png>
=E2=80=8B
=
Thanks,

-- =
Pradeep Chhetri =


=20 =20 =20 =20
=20

--56e836c1_275ac794_322a--