flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Monitoring single-run job statistics
Date Wed, 06 Jan 2016 19:08:42 GMT
Good luck with the Thesis!

On Wed, Jan 6, 2016 at 6:18 PM, Filip Łęczycki <filipleczycki@gmail.com>
wrote:

> Hi Stephan,
>
> Thank you for you answer. I would love to contribute but currently I have
> no capacity as I am buried with my thesis.
>
> I will reach out after graduating :)
>
> Bestr regards.
> Filip
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2016-01-05 10:35 GMT+01:00 Stephan Ewen <sewen@apache.org>:
>
>> Hi Filip!
>>
>> There are thoughts and efforts to extend Flink to push the result
>> statistics of Flink jobs to the YARN timeline server. That way, you can
>> explore jobs that are completed.
>>
>> Since the whole web dashboard in Flink has a pure REST design, this is a
>> quite straightforward fix.
>>
>> From the capacities I see in the community, I can not promise that to be
>> fixed immediately. Let me know, though, if you are interested in
>> contributing an addition there, and I can walk you through the steps that
>> would be needed.
>>
>> Greetings,
>> Stephan
>>
>>
>> On Mon, Jan 4, 2016 at 9:17 PM, Filip Łęczycki <filipleczycki@gmail.com>
>> wrote:
>>
>>> Hi Till,
>>>
>>> Thank you for you answer however I am sorry to hear that. I was
>>> reluctant to execute jobs with long running Flink cluster due to the fact
>>> that multiple jobs would cloud yarn statistics regarding cpu and memory
>>> time as well as Flink's garbage collector statistics in log, as they would
>>> be stored for the whole Flink cluster, instead of a single job.
>>>
>>> Do you know whether is there a way to extract mentioned stats (cpu time,
>>> mem time, gc time) for a single job ran on long running Flink cluster?
>>>
>>> I will be very grateful for an answer:)
>>>
>>> Best regards,
>>> Filip
>>>
>>> Pozdrawiam,
>>> Filip Łęczycki
>>>
>>> 2016-01-04 10:05 GMT+01:00 Till Rohrmann <till.rohrmann@gmail.com>:
>>>
>>>> Hi Filip,
>>>>
>>>> at the moment it is not possible to retrieve the job statistics after
>>>> the job has finished with flink run -m yarn-cluster. The reason is
>>>> that the YARN cluster is only alive as long as the job is executed. Thus,
I
>>>> would recommend you to execute your jobs with a long running Flink cluster
>>>> on YARN.
>>>>
>>>> Cheers,
>>>> Till
>>>> ​
>>>>
>>>> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <
>>>> filipleczycki@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am running filnk aps on YARN cluster and I am trying to get some
>>>>> benchmarks. When I start a long-running flink cluster on my YARN cluster
I
>>>>> have an access to web UI and rest API that provide me statistics of the
>>>>> deployed jobs (as desribed here:
>>>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
>>>>> I was wondering is this possible to get such information about a single
run
>>>>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
>>>>> finished there is no flink client running so I cannot use rest api to
get
>>>>> stats.
>>>>>
>>>>> Thanks for any help:)
>>>>>
>>>>>
>>>>> Best regards/Pozdrawiam,
>>>>> Filip Łęczycki
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message