flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "huangwei (G)" <huangwei...@huawei.com>
Subject Re: Some questions about storm metrics
Date Mon, 21 Sep 2015 13:31:36 GMT
@Matthias
Thanks for the advice.
I opened a JIRA and a PR for the first step about the storm metrics.
If you think this is OK, I'll do a integrated test and add all the storm metrics in future.

Greetings,
Wei Huang


=================================================

Hi,

there is no need to apologize :)

If I understand you correctly, you need access to RuntimeContext within FlinkTopologyContext.
FlinkTopologyContext in created within StormWrapperSetupHelper.convertToTopologyContext which
has already access to StreamingRuntimeContext (which is just a special RuntimeContext). So
just extend the constructor of FlinkTopologyContext by a RuntimeContext parameter an add a
member to FlinkTopologyContext that hold a refrence to the RuntimeContext.

-Matthias


On 09/19/2015 10:28 AM, huangwei (G) wrote:
> Hi Matthias,
> 
> Very sorry for the email in weekend.
> I try to add a wrapper(FlinkCountMetric) for the Storm-CountMetric and it's also easy
to do it.
> But I get a trouble in `FlinkTopologyContext.registerMetric` method that I don't know
how to use the `getRuntimeContext` in this class since I want to register the CountMetric
into Flink.
> Do you have any idea?
> 
> Greetings,
> Wei Huang.
> 
> ======================================================================
> =====
> 
> I personally never used Metrics in Storm. It would be best, if you design an own example.
> 
> I guess you are familiar with
> https://storm.apache.org/documentation/Metrics.html
> 
> As a first step, I would try to include the predefined Storm metric and work on custom
metrics later. As the Storm documentation points out, the metrics are used to get the value
for Storm UI. Maybe, you can start to add Tuple-count metrics for whole topologies to be collected
and displayed in Flink WebUI.
> 
> -Matthias
> 
> 
> On 09/07/2015 02:51 PM, huangwei (G) wrote:
>> Hi Kostas,
>>
>> Thanks for your support!
>> I created the JIRA and pushed a PR for it.
>>
>> @Matthias
>> Could you give me a storm example in using the cust metrics?
>> Since it is needed for follow-up development.
>> Many thanks!
>>
>> Greetings,
>> Huang Wei
>>
>> =====================================
>>
>> Sounds great. It is easy to create an AverageAccumulator by implementing the org.apache.flink.api.common.accumulators.Accumulator
interface, accumulating the sum and the count internally, and dividing in the end.
>>
>> This would be something useful also outside of Storm compatibility.
>>
>> Then, MultiReduceMetrics can be implemented on top of that.
>>
>> Looking forward to the JIRA.
>>
>> On Wed, Sep 2, 2015 at 8:36 AM, huangwei (G) <huangwei111@huawei.com> wrote:
>>
>>> Dear all,
>>>
>>> I focused on the Flink-accumulator and found that there is not any 
>>> feature to count the average of values.
>>> And there is a MultiReducedMetric class in storm that calculates the 
>>> average.
>>> This class can be used internally to get the latency in storm.
>>> So as the first step, I`d like to add a class like MultiReducedMetric.
>>> And the detail about design I`ll describe in a new JIRA if you agree 
>>> this above.
>>> Please let me know if I`m wrong.
>>> Thank you!
>>>
>>> ========================================
>>>
>>> Hi Kostas,
>>> Thank you for the information.
>>> I`ll take a look.
>>> And I`m comparing the storm metrics with Flink accumulators.
>>> I`ll let you know if I have some ideas.
>>>
>>> ========================================
>>>
>>> Flink accumulators are documented here:
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-master/apis/programm
>>> i
>>> n
>>> g_guide.html#accumulators--counters
>>>
>>> I think Aljoscha's idea is to implement Storm metrics on top of those.
>>>
>>> Huangwei, perhaps you can look whether the interfaces match 
>>> (basically if Flink accumulators have all the features needed to 
>>> support the kinds of metrics that Storm exposes, or if there is a 
>>> significant impedance mismatch).
>>>
>>> On Mon, Aug 31, 2015 at 2:06 PM, Matthias J. Sax < 
>>> mjsax@informatik.hu-berlin.de> wrote:
>>>
>>>> I did not look into it and I am not super familiar with Flink's 
>>>> accumulators... Can not provide much help right now. Sorry :(
>>>>
>>>> -Matthias
>>>>
>>>> On 08/31/2015 01:01 PM, huangwei (G) wrote:
>>>>> Hi,
>>>>> I think I can take a simple design for it.
>>>>> Do you have any idea?
>>>>>
>>>>> ========================================
>>>>>
>>>>> Maybe. I did not put any thoughts into this...
>>>>>
>>>>> On 08/31/2015 12:25 PM, Aljoscha Krettek wrote:
>>>>>> Hi,
>>>>>> could the metrics not be implemented on top of Flink accumulators?
>>>>>> They would even be live-updated in the new web front-end.
>>>>>>
>>>>>> Cheers,
>>>>>> Aljoscha
>>>>>>
>>>>>> On Mon, 31 Aug 2015 at 11:32 Matthias J. Sax 
>>>>>> <mjsax@informatik.hu-berlin.de>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> custom metrics are currently not supported by the compatibility
>>> layer.
>>>>>>> See README here:
>>>>>>>
>>>>>>> https://github.com/apache/flink/tree/master/flink-contrib/flink-
>>>>>>> st orm -compatibility/flink-storm-compatibility-core
>>>>>>>
>>>>>>> It is an open question, if/how Flink can support this feature.
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 08/31/2015 09:53 AM, huangwei (G) wrote:
>>>>>>>> Hi Matthias and all,
>>>>>>>>
>>>>>>>> I`d like to do the custom metrics in flink-storm-compatibility.
>>>>>>>> And there are some questions following:
>>>>>>>> Is there anything like storm metrics in Flink?
>>>>>>>> If not, does Flink need to have metrics mechanism in internal
first?
>>>>>>>> Or just let the metrics use in flink-storm-compatibility?
>>>>>>>>
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>> Huang Wei
>>>>>>>> 华为技术有限公司 Huawei Technologies Co., Ltd.
>>>>>>>>
>>>>>>>>
>>>>>>>> Tel:+86 18106512602
>>>>>>>> Email:huangwei111@huawei.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
> 

Mime
View raw message