flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: How to run a job with job cluster mode on top of mesos?
Date Tue, 02 Apr 2019 14:49:57 GMT
By the way, did the Mesos job mode work for you in the end?

On Tue, Apr 2, 2019 at 7:47 AM Till Rohrmann <trohrmann@apache.org> wrote:

> Sure I will help with the review. Thanks for opening the PR Jacky.
>
> Cheers,
> Till
>
> On Tue, Apr 2, 2019 at 2:17 AM Jacky Yin 殷传旺 <jacky.yin@eoitek.com> wrote:
>
>> Hello Till,
>>
>>
>>
>> I submitted a PR(#8084) for this issue. Could you help review it?
>>
>>
>>
>> Many thanks!
>>
>>
>>
>> *Jacky Yin*
>>
>> *发件人**: *Till Rohrmann <trohrmann@apache.org>
>> *日期**: *2019年3月29日 星期五 下午11:06
>> *收件人**: *Jacky Yin 殷传旺 <jacky.yin@eoitek.com>
>> *抄送**: *"user@flink.apache.org" <user@flink.apache.org>
>> *主题**: *Re: How to run a job with job cluster mode on top of mesos?
>>
>>
>>
>> Thanks a lot Jacky.
>>
>>
>>
>> Cheers,
>>
>> Till
>>
>>
>>
>> On Fri, Mar 29, 2019 at 1:50 PM Jacky Yin 殷传旺 <jacky.yin@eoitek.com>
>> wrote:
>>
>> Hello Till,
>>
>>
>>
>> Thanks very much for the clear steps. And I noticed that you have already
>> created a jira ticket for this issue and I would like to contribute it. (I
>> have already assigned it to me). 😊
>>
>>
>>
>>
>>
>> *Jacky Yin *
>>
>> *发件人**: *Till Rohrmann <trohrmann@apache.org>
>> *日期**: *2019年3月26日 星期二 下午6:31
>> *收件人**: *Jacky Yin 殷传旺 <jacky.yin@eoitek.com>
>> *抄送**: *"user@flink.apache.org" <user@flink.apache.org>
>> *主题**: *Re: How to run a job with job cluster mode on top of mesos?
>>
>>
>>
>> Hi Jacky,
>>
>>
>>
>> you're right that we are currently lacking documentation for the
>> `mesos-appmaster-job.sh` script. I've added a JIRA issue to cover this [1].
>>
>>
>>
>> In order to use this script you first need to store a serialized version
>> of the `JobGraph` you want to run somewhere where the Mesos appmaster can
>> read it. Moreover, you need to make sure that the user code jars are on the
>> classpath (e.g. putting them in the lib directory). See how the
>> AbstractYarnClusterDescriptor serializes the `JobGraph` [2] for some
>> details. The last thing before you can start the MesosJobClusterEntrypoint
>> is to specify the path to the serialized job graph via the
>> `internal.jobgraph-path` configuration option which you can specify in your
>> flink-conf.yaml or pass in via dynamic property.
>>
>>
>>
>> I know that this is not super streamlined and needs to be improved. E.g.
>> one could do it similarly to the `StandaloneJobClusterEntrypoint` that one
>> includes the user code jar and specifies the class name of the user code to
>> load. That way one would not need to generate the JobGraph yourself and
>> then serialize it.
>>
>>
>>
>> I hope that I could help you a little bit.
>>
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-12020
>>
>> [2]
>> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L846
>>
>>
>>
>> Cheers,
>>
>> Till
>>
>>
>>
>> On Fri, Mar 22, 2019 at 7:24 AM Jacky Yin 殷传旺 <jacky.yin@eoitek.com>
>> wrote:
>>
>> Anybody can help? I found the help documentation about how to do with the
>> job cluster mode for yarn, docker and k8s. However only the help of session
>> cluster mode is provided in flink web site for mesos.
>>
>> It looks like the shell “mesos-appmaster-job.sh” should be the right one
>> to run a job with the mode of job cluster on top of mesos.  However I
>> cannot find any help or example for this shell.
>>
>> Any help will be greatly appreciated.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> *Jacky Yin *
>>
>> *发件人**: *Jacky Yin 殷传旺 <jacky.yin@eoitek.com>
>> *日期**: *2019年3月21日 星期四 下午2:31
>> *收件人**: *"user@flink.apache.org" <user@flink.apache.org>
>> *主题**: *Documentation of mesos-appmaster-job.sh
>>
>>
>>
>> Hello All,
>>
>>
>>
>> I cannot find any documentation or help about how to use
>> $flin_home/bin/mesos-appmaster-job.sh.  Anybody help?
>>
>>
>>
>> Thanks!
>>
>>
>>
>> *Jacky Yin*
>>
>>

Mime
View raw message