flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bajaj, Abhinav" <abhinav.ba...@here.com>
Subject Re: Documentation for translation of Job graph to Execution graph
Date Tue, 21 Jun 2016 23:15:39 GMT
Thanks Robert for helpful reply.

I have follow up on the Q2 - "In general, we recommend running one JobManager per job”
I understand how this can be achieved while running in Yarn, I.e. by submitting single Flink
Jobs.

Is their some other way of setting Flink to configure single Jobmanager per job ?



From: Robert Metzger <rmetzger@apache.org<mailto:rmetzger@apache.org>>
Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>
Date: Tuesday, June 21, 2016 at 8:23 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>,
Till Rohrmann <trohrmann@apache.org<mailto:trohrmann@apache.org>>
Cc: Aljoscha Krettek <aljoscha@apache.org<mailto:aljoscha@apache.org>>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,
the link has been added newly, yes.

Regarding Q1, since there is no documentation right now, I have to refer you to our code.
In the JobManager.scala class, there is a method "private def submitJob(jobGraph, ...") where
the ExecutionGraph is created. I think that's a good starting point for looking through the
code. (I also added Till to the message if he wants to chime in)

Q2: Currently, Flink doesn't add the job name to the logs, so its indeed not very easy to
separate the log entries generated by different jobs. In general, we recommend running one
JobManager per job (multiple jobs is of course also supported).




On Sat, Jun 18, 2016 at 1:41 AM, Bajaj, Abhinav <abhinav.bajaj@here.com<mailto:abhinav.bajaj@here.com>>
wrote:
Hi,

Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0
docs. I will go through it.

In general, there are two things I am trying to understand and get comfortable with -

  1.  How a Job graph is translated to Execution graph. The logs and monitoring APIs are for
the Execution graph. So, I need to map them to the Job graph. I am trying to bridge this gap.
  2.  The job manager & task manager logs are tricky to decipher. Especially when there
are multiple jobs running. Is there a way to filter the logs for a single job ?

~ Abhi


From: Aljoscha Krettek <aljoscha@apache.org<mailto:aljoscha@apache.org>>
Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>
Date: Friday, June 17, 2016 at 2:31 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>
Subject: Re: Documentation for translation of Job graph to Execution graph

Hi,
I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html.

With what parts are you having trouble? Maybe I can help.

Cheers,
Aljoscha

On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav <abhinav.bajaj@here.com<mailto:abhinav.bajaj@here.com>>
wrote:
Hi,

When troubleshooting a flink job, it is tricky to map the Job graph (application code) to
the logs & monitoring REST APIs.

So, I am trying to find documentation on how a Job graph is translated to Execution graph.
I found this - https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/job_scheduling.html

Any detailed documentation on the design and code components will be helpful.

Thanks,
Abhi

Mime
View raw message