Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8436218C61 for ; Thu, 17 Dec 2015 10:35:14 +0000 (UTC) Received: (qmail 93752 invoked by uid 500); 17 Dec 2015 10:35:12 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 93639 invoked by uid 500); 17 Dec 2015 10:35:12 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 93627 invoked by uid 99); 17 Dec 2015 10:35:11 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Dec 2015 10:35:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 4770D1A0AA6 for ; Thu, 17 Dec 2015 10:35:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=typesafe.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id IIwkTOR3iIEz for ; Thu, 17 Dec 2015 10:35:02 +0000 (UTC) Received: from mail-vk0-f49.google.com (mail-vk0-f49.google.com [209.85.213.49]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id BA1D32059D for ; Thu, 17 Dec 2015 10:35:01 +0000 (UTC) Received: by mail-vk0-f49.google.com with SMTP id a188so44323743vkc.0 for ; Thu, 17 Dec 2015 02:35:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=typesafe.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=J2BDLQUeGytOu/X4ABmYmZfK8XEXtMQhTMvrkyojCGc=; b=fjAof4yE5yzBMil1+nHhuI+b/Il1dxTtLigBpb7ejquA/H37/C4nhtAN2nlRtJf7u7 8GkcMvlMKHFZAFThHABErKUPIAB+3mvKRnwbNd+peuGi7q+JnqAD2xX+aeuBAoq8owo2 Q3Eyx98igto/zULdJysuZoXHEwFE8tgPz8vuk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=J2BDLQUeGytOu/X4ABmYmZfK8XEXtMQhTMvrkyojCGc=; b=GqLOqkp8LEHpsvCcvdzQUc8Kq0DMChhU2niqE9zPsYtmOHEP8JcZ5/yAm137sf6lkW H5ZCZsjIHzgSyqeykbuz17iDweC6weq/QhvnT2EoT16yWwmgohp5AwP6+3chISljIwep Eb+n0krzbZQyeNER+Pv+jzTnF66HZ6pMs0aDdQW9hEl+aX83pJg/nYhb8TDK63gU6xpo qxM21SErZ8bALXaU09oP6XygUtkqGNpY5ciDI6DZviRH74NK8JTvN7CqpEHsct/3Bymc n3/owCchFrvRgbo4Gp+TZ/tXCl3OpEduvnInnCVfojxWy+Og+26TphYQFiRrvaQgggD1 zgJQ== X-Gm-Message-State: ALoCoQl43sJiUlH5pLleIuh0q50w80IvfSElRZlyg0BQzgyJ8x9fFYX+B6jJ4OxDlk/ZPVkvajOE8gK1hIQEfvfk8/px3dkX6Q== X-Received: by 10.31.152.207 with SMTP id a198mr38037454vke.68.1450348500556; Thu, 17 Dec 2015 02:35:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.31.7.204 with HTTP; Thu, 17 Dec 2015 02:34:41 -0800 (PST) In-Reply-To: References: <11A70834-1E8F-46AA-9CC5-133D172147E7@gmail.com> From: =?UTF-8?Q?Iulian_Drago=C8=99?= Date: Thu, 17 Dec 2015 11:34:41 +0100 Message-ID: Subject: Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode To: Aaron Cc: Timothy Chen , "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=001a113d39e22d349a052715932d --001a113d39e22d349a052715932d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, Dec 16, 2015 at 5:42 PM, Aaron wrote: > Wrt to PR, sure, let me update the documentation, i'll send it out > shortly. My Fork is on Github..is the PR from there ok? > Absolutely. Have a look at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark if you haven't done so already, it should answer most questions about starting to contribute to Spark. thanks, iulian > > Cheers, > Aaron > > On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen wrote: > > Yes if want to manually override what IP to use to be contacted by the > > master you can set LIPROCESS_IP and LIBPROCESS_PORT. > > > > It is a Mesos specific settings. We can definitely update the docs. > > > > Note that in the future as we move to use the new Mesos Http API these > > configurations won't be needed (also libmesos!). > > > > Tim > > > > On Dec 16, 2015, at 8:09 AM, Iulian Drago=C8=99 > > wrote: > > > > LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a > > Mesos-specific setting. > > > > Have you tried setting SPARK_LOCAL_IP? > > > > On Wed, Dec 16, 2015 at 5:07 PM, Aaron wrote: > >> > >> Found this thread that talked about it to help understand it better: > >> > >> > >> > https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ6= 8qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E > >> > >> > > >> > When you run Spark on Mesos it needs to run > >> > > >> > spark driver > >> > mesos scheduler > >> > > >> > and both need to be visible to outside world on public iface IP > >> > > >> > you need to tell Spark and Mesos on which interface to bind - by > default > >> > they resolve node hostname to ip - this is loopback address in your > case > >> > > >> > Possible solutions - on slave node with public IP 192.168.56.50 > >> > > >> > 1. Set > >> > > >> > export LIBPROCESS_IP=3D192.168.56.50 > >> > export SPARK_LOCAL_IP=3D192.168.56.50 > >> > > >> > 2. Ensure your hostname resolves to public iface IP - (for testing) > edit > >> > /etc/hosts to resolve your domain name to 192.168.56.50 > >> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos > answer > >> > > >> > >> Cheers, > >> Aaron > >> > >> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Drago=C8=99 > >> wrote: > >> > Hi Aaron, > >> > > >> > I never had to use that variable. What is it for? > >> > > >> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron wrote: > >> >> > >> >> In going through running various Spark jobs, both Spark 1.5.2 and t= he > >> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we > >> >> noticed that is in order to run the Spark shells (both python and > >> >> scala), we needed to set the LIBPROCESS_IP environment variable > before > >> >> running. > >> >> > >> >> Was curious if the Spark on Mesos docs should be updated, under the > >> >> Client Mode section, to include setting this environment variable? > >> >> > >> >> Cheers > >> >> Aaron > >> >> > >> >> -------------------------------------------------------------------= -- > >> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org > >> >> For additional commands, e-mail: dev-help@spark.apache.org > >> >> > >> > > >> > > >> > > >> > -- > >> > > >> > -- > >> > Iulian Dragos > >> > > >> > ------ > >> > Reactive Apps on the JVM > >> > www.typesafe.com > >> > > > > > > > > > > > -- > > > > -- > > Iulian Dragos > > > > ------ > > Reactive Apps on the JVM > > www.typesafe.com > > > --=20 -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com --001a113d39e22d349a052715932d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Wed, Dec 16, 2015 at 5:42 PM, Aaron <aarongmldt@gmail.com>= ; wrote:
Wrt to PR, sure, let me update the docu= mentation, i'll send it out
shortly.=C2=A0 My Fork is on Github..is the PR from there ok?

Absolutely. Have a look at=C2=A0https://cwi= ki.apache.org/confluence/display/SPARK/Contributing+to+Spark if you hav= en't done so already, it should answer most questions about starting to= contribute to Spark.

thanks,
iulian
=C2=A0

Cheers,
Aaron

On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen <tnachen@gmail.com> wrote:
> Yes if want to manually override what IP to use to be contacted by the=
> master you can set LIPROCESS_IP and LIBPROCESS_PORT.
>
> It is a Mesos specific settings. We can definitely update the docs. >
> Note that in the future as we move to use the new Mesos Http API these=
> configurations won't be needed (also libmesos!).
>
> Tim
>
> On Dec 16, 2015, at 8:09 AM, Iulian Drago=C8=99 <iulian.dragos@typesafe.com>
> wrote:
>
> LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a=
> Mesos-specific setting.
>
> Have you tried setting SPARK_LOCAL_IP?
>
> On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aarongmldt@gmail.com> wrote:
>>
>> Found this thread that talked about it to help understand it bette= r:
>>
>>
>> https://mail-archives.apache.= org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFK= FgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>>
>> >
>> > When you run Spark on Mesos it needs to run
>> >
>> > spark driver
>> > mesos scheduler
>> >
>> > and both need to be visible to outside world on public iface = IP
>> >
>> > you need to tell Spark and Mesos on which interface to bind -= by default
>> > they resolve node hostname to ip - this is loopback address i= n your case
>> >
>> > Possible solutions - on slave node with public IP 192.168.56.= 50
>> >
>> > 1. Set
>> >
>> >=C2=A0 =C2=A0 export LIBPROCESS_IP=3D192.168.56.50
>> >=C2=A0 =C2=A0 export SPARK_LOCAL_IP=3D192.168.56.50
>> >
>> > 2. Ensure your hostname resolves to public iface IP - (for te= sting) edit
>> > /etc/hosts to resolve your domain name to 192.168.56.50
>> > 3. Set correct hostname/ip in mesos configuration - see Nikol= aos answer
>> >
>>
>> Cheers,
>> Aaron
>>
>> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Drago=C8=99
>> <iulian.dragos@ty= pesafe.com> wrote:
>> > Hi Aaron,
>> >
>> > I never had to use that variable. What is it for?
>> >
>> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aarongmldt@gmail.com> wrote:
>> >>
>> >> In going through running various Spark jobs, both Spark 1= .5.2 and the
>> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.= 25), we
>> >> noticed that is in order to run the Spark shells (both py= thon and
>> >> scala), we needed to set the LIBPROCESS_IP environment va= riable before
>> >> running.
>> >>
>> >> Was curious if the Spark on Mesos docs should be updated,= under the
>> >> Client Mode section, to include setting this environment = variable?
>> >>
>> >> Cheers
>> >> Aaron
>> >>
>> >> ---------------------------------------------------------= ------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: dev-help@spark.apache.org
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Iulian Dragos
>> >
>> > ------
>> > Reactive Apps on the JVM
>> > www.typesafe.com
>> >
>
>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>



--
=

--
Iulian Dragos
=

------
Reactive Apps on the JVM
www.typesafe.com

=
--001a113d39e22d349a052715932d--