mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Clasen <sc...@heroku.com>
Subject Re: callback port
Date Wed, 21 May 2014 18:42:06 GMT
Ok got it figured out. LIBPROCESS_PORT is being unset along the way
somewhere by marathon/mesos, the solution was to start the spark driver
process via a script that re exports LIBPROCESS_PORT by reading it from a
var named something other than LIBPROCESS_PORT (in my case
SPARK_LIBPROCESS_PORT)



On Mon, May 19, 2014 at 12:08 PM, Scott Clasen <scott@heroku.com> wrote:

> Oh for sure, setting quite a few that I am seeing in the env.
>
>
> On Mon, May 19, 2014 at 12:05 PM, Vinod Kone <vinodkone@gmail.com> wrote:
>
>> Can you set some other variable via the same mechanism and see if you can
>> see it in your app?
>>
>>
>> On Mon, May 19, 2014 at 10:54 AM, Scott Clasen <scott@heroku.com> wrote:
>>
>>> hmm, I am setting it via the Environment passed in the REST call to
>>> create the app, and if I look at the marathon UI I do see the var set.
>>>
>>>
>>> On Mon, May 19, 2014 at 10:19 AM, Vinod Kone <vinodkone@gmail.com>wrote:
>>>
>>>> Probably. How are you setting the LIBPROCESS_PORT in Marathon? It has
>>>> to be set via CommandInfo.Environment() of the task/executor for this to
>>>> take effect.
>>>>
>>>>
>>>> On Fri, May 16, 2014 at 9:41 AM, Scott Clasen <scott@heroku.com> wrote:
>>>>
>>>>> Aha, thanks!  I am still having an issue. I am executing the process
>>>>> via marathon, with LIBPROCESS_PORT set in the env.  However when I log
the
>>>>> value of LIBPROCESS_PORT in my program, I get 0.
>>>>>
>>>>> Found a thread in this ML called
>>>>> Re: Review Request: "Unset" LIBPROCESS_PORT before applying
>>>>> executor/command environment variables (in the event they modify it).
Is
>>>>> this what is happening to me?
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 15, 2014 at 7:16 PM, Benjamin Mahler <
>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>
>>>>>> You can set LIBPROCESS_PORT in the environment.
>>>>>>
>>>>>>
>>>>>> On Wed, May 14, 2014 at 1:58 PM, Scott Clasen <scott@heroku.com>wrote:
>>>>>>
>>>>>>> I raised this question on the Spark ML but it may be more a Mesos
>>>>>>> question.
>>>>>>>
>>>>>>> I would like to be able to configure the port used to communicate
>>>>>>> between the Mesos master and Spark tasks running across mesos
slaves. It
>>>>>>> appears that spark's usage of the mesos library must use the
default port =
>>>>>>> 0.
>>>>>>>
>>>>>>> At least that what I think happens in UPID.
>>>>>>> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/pid.cpp
>>>>>>>
>>>>>>> Here is an example...
>>>>>>>
>>>>>>> In this case if the port 56311 is not opened up via iptables
and
>>>>>>> security groups, the detecting new master step will hang indefinitely,
and
>>>>>>> failures will be logged on the master.
>>>>>>>
>>>>>>> group.cpp:310] Group process ((2)@1.2.3.4:56311) connected to
>>>>>>> ZooKeeper
>>>>>>> group.cpp:752] Syncing group operations: queue size (joins, cancels,
>>>>>>> datas) = (0, 0, 0)
>>>>>>> group.cpp:367] Trying to create path '/mesos' in ZooKeeper
>>>>>>> detector.cpp:134] Detected a new leader: (id='2')
>>>>>>> group.cpp:629] Trying to get '/mesos/info_0000000002' in ZooKeeper
>>>>>>> detector.cpp:351] A new leading master (UPID=
>>>>>>> master@10.118.41.27:5050) is detected
>>>>>>> sched.cpp:230] Detecting new master
>>>>>>>
>>>>>>>
>>>>>>> Is there a way to tell the mesos native lib to use a specific
port
>>>>>>> rather than a random port from the spark side of things?
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>> SC
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message