spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iulian Dragoș <iulian.dra...@typesafe.com>
Subject Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode
Date Thu, 17 Dec 2015 10:34:41 GMT
On Wed, Dec 16, 2015 at 5:42 PM, Aaron <aarongmldt@gmail.com> wrote:

> Wrt to PR, sure, let me update the documentation, i'll send it out
> shortly.  My Fork is on Github..is the PR from there ok?
>

Absolutely. Have a look at
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark if
you haven't done so already, it should answer most questions about starting
to contribute to Spark.

thanks,
iulian


>
> Cheers,
> Aaron
>
> On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen <tnachen@gmail.com> wrote:
> > Yes if want to manually override what IP to use to be contacted by the
> > master you can set LIPROCESS_IP and LIBPROCESS_PORT.
> >
> > It is a Mesos specific settings. We can definitely update the docs.
> >
> > Note that in the future as we move to use the new Mesos Http API these
> > configurations won't be needed (also libmesos!).
> >
> > Tim
> >
> > On Dec 16, 2015, at 8:09 AM, Iulian Dragoș <iulian.dragos@typesafe.com>
> > wrote:
> >
> > LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a
> > Mesos-specific setting.
> >
> > Have you tried setting SPARK_LOCAL_IP?
> >
> > On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aarongmldt@gmail.com> wrote:
> >>
> >> Found this thread that talked about it to help understand it better:
> >>
> >>
> >>
> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
> >>
> >> >
> >> > When you run Spark on Mesos it needs to run
> >> >
> >> > spark driver
> >> > mesos scheduler
> >> >
> >> > and both need to be visible to outside world on public iface IP
> >> >
> >> > you need to tell Spark and Mesos on which interface to bind - by
> default
> >> > they resolve node hostname to ip - this is loopback address in your
> case
> >> >
> >> > Possible solutions - on slave node with public IP 192.168.56.50
> >> >
> >> > 1. Set
> >> >
> >> >    export LIBPROCESS_IP=192.168.56.50
> >> >    export SPARK_LOCAL_IP=192.168.56.50
> >> >
> >> > 2. Ensure your hostname resolves to public iface IP - (for testing)
> edit
> >> > /etc/hosts to resolve your domain name to 192.168.56.50
> >> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos
> answer
> >> >
> >>
> >> Cheers,
> >> Aaron
> >>
> >> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
> >> <iulian.dragos@typesafe.com> wrote:
> >> > Hi Aaron,
> >> >
> >> > I never had to use that variable. What is it for?
> >> >
> >> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aarongmldt@gmail.com> wrote:
> >> >>
> >> >> In going through running various Spark jobs, both Spark 1.5.2 and the
> >> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> >> >> noticed that is in order to run the Spark shells (both python and
> >> >> scala), we needed to set the LIBPROCESS_IP environment variable
> before
> >> >> running.
> >> >>
> >> >> Was curious if the Spark on Mesos docs should be updated, under the
> >> >> Client Mode section, to include setting this environment variable?
> >> >>
> >> >> Cheers
> >> >> Aaron
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >> For additional commands, e-mail: dev-help@spark.apache.org
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Iulian Dragos
> >> >
> >> > ------
> >> > Reactive Apps on the JVM
> >> > www.typesafe.com
> >> >
> >
> >
> >
> >
> > --
> >
> > --
> > Iulian Dragos
> >
> > ------
> > Reactive Apps on the JVM
> > www.typesafe.com
> >
>



-- 

--
Iulian Dragos

------
Reactive Apps on the JVM
www.typesafe.com

Mime
View raw message