spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-13041) Add a driver ui link and a mesos sandbox link on the dispatcher's ui page for each driver
Date Wed, 27 Jan 2016 11:03:39 GMT

     [ https://issues.apache.org/jira/browse/SPARK-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stavros Kontopoulos updated SPARK-13041:
----------------------------------------
    Description: 
It would be convenient to have the uri of the driver's ui and the driver's mesos sandbox uri
on the dispatcher's ui. Things to consider:

a) sandbox link has the master's hostname name in it which may not be accessible if zk is
used.

b) In case we have multiple executors on the same machine we make sure the dispatcher allocates
a known port from the available offered, so the port value is known upfront to present it.

c) we need to make sure there are no conflicts with port isolation on slaves, eg multiple
drivers or multiple executors. Right now the Mesos containerizer doesn't provide much isolation:
http://mesos.apache.org/documentation/latest/containerizer/, for port isolation you need:
http://mesos.apache.org/documentation/latest/network-monitoring/

  was:
It would be convenient to have the uri of the driver's ui and the driver's mesos sandbox uri
on the dispatcher's ui. Two things to consider:

a) sandbox link has the master's hostname name in it which may not be accessible if zk is
used.

b) In case we have multiple executors on the same machine we make sure the dispatcher allocates
a known port from the available offered, so the port value is known upfront to present it.

c) we need to make sure there are no conflicts with port isolation on slaves, eg multiple
drivers or multiple executors. Right now the Mesos containerizer doesn't provide much isolation:
http://mesos.apache.org/documentation/latest/containerizer/, for port isolation you need:
http://mesos.apache.org/documentation/latest/network-monitoring/


> Add a driver ui link and a mesos sandbox link on the dispatcher's ui page for each driver
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-13041
>                 URL: https://issues.apache.org/jira/browse/SPARK-13041
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>            Reporter: Stavros Kontopoulos
>            Priority: Minor
>
> It would be convenient to have the uri of the driver's ui and the driver's mesos sandbox
uri on the dispatcher's ui. Things to consider:
> a) sandbox link has the master's hostname name in it which may not be accessible if zk
is used.
> b) In case we have multiple executors on the same machine we make sure the dispatcher
allocates a known port from the available offered, so the port value is known upfront to present
it.
> c) we need to make sure there are no conflicts with port isolation on slaves, eg multiple
drivers or multiple executors. Right now the Mesos containerizer doesn't provide much isolation:
http://mesos.apache.org/documentation/latest/containerizer/, for port isolation you need:
http://mesos.apache.org/documentation/latest/network-monitoring/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message