flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Task and Operator Monitoring via JMX / naming
Date Thu, 20 Oct 2016 13:55:28 GMT
Well the issue is the following:

the metric system assumes the following naming scheme for tasks based on 
the DataSet API and simple streaming jobs: [CHAIN] operatorName1 [=> 
operatorName2 [ ...]]
To retrieve the operator name the above is split by "=>", giving us a 
String[] of all operator names in a task, from which we then select the 
correct one based on the position in the chain.

However, the Stremaing API has some fancy chaining stuff going on, where 
multiple operations can be chained to a single one which results in a 
name like this: operatorName1 => (operatorName2, operatorName3)

For both op2 and op3 the chain index is identical (since for a tree 
structure the index is the depth), resulting in both picking 
(operatorName2, operatorName3) as their name, which is obviously wrong.

The solution (which i already implemented, sorry for that) is to simply 
stop inferring the operator names from the task (it was hacky to being 
with) and just encode them in the configuration for the operator.
This can be seen here: 
https://github.com/zentol/flink/commit/7f439525a26504e98b72f2d39b987ac878464419

Regards,
Chesnay

On 20.10.2016 14:21, Philipp Bussche wrote:
> Thanks Chesnay,
> I am happy to share more around my environment and do additional testing for
> this.
> Also I would be happy to help fixing if we see there might be an issue in
> the code somewhere.
> In fact I am still trying to get a Hacktoberfest T-Shirt and I am still pull
> requests short  ;)
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Task-and-Operator-Monitoring-via-JMX-naming-tp9560p9650.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>


Mime
View raw message