flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Lee <philjj...@gmail.com>
Subject Re: Hello, a question about Dashborad in Flink
Date Fri, 29 Jan 2016 08:55:23 GMT
Great,

you menat the difference between narrow shuffle and global shuffle?

I use Flink version 0.9,
but it did not not work to access REST interface when I use "ssh tunnel" to
remote server.

it is from version of probelm?

Best,
Phil



On Fri, Jan 29, 2016 at 9:46 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> The REST interface does also provide metrics about the number of records
> and the size of the input and output of all tasks.
> See:
> - /jobs/<jobid>/vertices/<vertexid>
> -
> /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
> in
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
>
> However, not all of this data is going over the network because some tasks
> can be locally connected.
>
> Best, Fabian
>
> 2016-01-29 8:50 GMT+01:00 Philip Lee <philjjoon@gmail.com>:
>
>> Thanks,
>>
>> Is there any way to measure shuffle data (read and write) on Flink or
>> Dashboard?
>>
>> I did not find the network usage metric in it.
>>
>> Best,
>> Phil
>>
>> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>>
>>> You can start a job and then periodically request and store information
>>> about the running job and vertices from using corresponding REST calls [1].
>>> The data will be in JSON format.
>>> After the job finished, you can stop requesting data.
>>>
>>> Next you parse the JSON, extract the information you need and give it to
>>> some plotting library.
>>> As I said, it is not possible to pass this data back into Flink's
>>> dashboard, but you have to process and plot it yourself.
>>>
>>> Best, Fabian
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>>
>>>
>>>
>>> 2016-01-25 16:15 GMT+01:00 Philip Lee <philjjoon@gmail.com>:
>>>
>>>> Hello,
>>>>
>>>> According to
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>>>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>>>
>>>> But as Fabian mentioned here,
>>>> "However, you can query all stats that are displayed by the dashboard
>>>> via a REST API [1] while the JM is running and save them yourself. This way
>>>> you can analyze the data also after the JM was stopped" could you explain
>>>> about this sentence in detail.
>>>>
>>>> I want to evaluate timeline view of each function after a job is done.
>>>>
>>>> Thanks,
>>>> Phil
>>>>
>>>
>>>
>>
>

Mime
View raw message