flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9703) Mesos does not expose TM Prometheus port
Date Thu, 12 Jul 2018 20:42:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542188#comment-16542188

ASF GitHub Bot commented on FLINK-9703:

Github user tillrohrmann commented on a diff in the pull request:

    --- Diff: flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/LaunchableMesosWorker.java
    @@ -332,6 +334,22 @@ public String toString() {
     		return taskInfo.build();
    +	/**
    +	 * Get port keys representing the TM's configured endpoints. This includes mandatory
TM endpoints such as
    +	 * data and rpc as well as optionally configured endpoints for services such as prometheus
    +	 *
    +	 * @return A deterministicly ordered Set of port keys to expose from the TM container
    +	 */
    +	private Set<String> getPortKeys() {
    +		LinkedHashSet<String> tmPortKeys = new LinkedHashSet<>(Arrays.asList(TM_PORT_KEYS));
    +		containerSpec.getDynamicConfiguration().keySet().stream()
    +			.filter(key -> key.endsWith(".port") || key.endsWith(".ports"))  // This matches
property naming convention
    --- End diff --
    I agree that simply taking all configuration values which end with `port` and `ports`
is problematic. What about configuration values whose value is needed, e.g. remote ports,
and should not be overwritten. For example the `jobmanager.rpc.port` is one of these configuration
    I would rather prefer a mesos specific configuration value which we can use to define
whose ports need to be dynamically assigned. For example `mesos.dynamic-port-assignment: "metrics.prom.port,
taskmanager.rpc.port, taskmanager.data.port"`. What do you think?

> Mesos does not expose TM Prometheus port
> ----------------------------------------
>                 Key: FLINK-9703
>                 URL: https://issues.apache.org/jira/browse/FLINK-9703
>             Project: Flink
>          Issue Type: Bug
>          Components: Mesos
>            Reporter: Rune Skou Larsen
>            Priority: Major
>              Labels: pull-request-available
> LaunchableMesosWorker makes Mesos expose these ports for a Task Manager:
> {{private static final String[] TM_PORT_KEYS = {}}
> {{ "taskmanager.rpc.port",}}
> {{ "taskmanager.data.port"};}}
> But when running Prometheus Exporter on a TM, another port needs to be exposed to make Flink's Prometheos
endpoint externally scrapable by the Prometheus server. By default this is port 9249, but it
is configurable according to:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter]
> My plan is to make a PR, that just adds another config option for mesos, to enable custom
ports to be exposed in the provisioned TMs.
> I considered carrying parts of the Metrics config into the Mesos code to automatically map metrics
ports in mesos. But making such a "shortcut" between Flink's metrics and mesos modules would probably
need some sort of integration testing, so I prefer the simple solution of just adding another
Mesos config option. But comments are welcome.

This message was sent by Atlassian JIRA

View raw message