From user-return-29957-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Sep 23 12:36:59 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 610F7180651 for ; Mon, 23 Sep 2019 14:36:59 +0200 (CEST) Received: (qmail 10682 invoked by uid 500); 23 Sep 2019 12:36:58 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 10673 invoked by uid 99); 23 Sep 2019 12:36:58 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Sep 2019 12:36:58 +0000 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id AAA116224 for ; Mon, 23 Sep 2019 12:36:57 +0000 (UTC) Received: by mail-wr1-f54.google.com with SMTP id l3so13726963wru.7 for ; Mon, 23 Sep 2019 05:36:57 -0700 (PDT) X-Gm-Message-State: APjAAAUEJ/qgpldk3LsWN9z0vu99lmWoofgfNNWxRuqRv0ZXR/m4KpTC srjCk9Tv1W9DBszcv09GDp72k7x6wtNPqqFqzTY= X-Google-Smtp-Source: APXvYqzFucdZXqkmKBPROBuYFoh7lhjXwBluDW3D/rOeAb44JYiCnAk7fsxdViO9ig7JN2+rOemJduH2Rsz2DtD5JTQ= X-Received: by 2002:a5d:4247:: with SMTP id s7mr20676621wrr.110.1569242216774; Mon, 23 Sep 2019 05:36:56 -0700 (PDT) MIME-Version: 1.0 References: <27E99033-F935-40DF-AC42-DC2FFA3D3C0E@gmail.com> In-Reply-To: From: Till Rohrmann Date: Mon, 23 Sep 2019 14:36:46 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: How to prevent from launching 2 jobs at the same time To: David Morin Cc: user Content-Type: multipart/alternative; boundary="000000000000e5c597059337ac7a" --000000000000e5c597059337ac7a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi David, you could use Flink's RestClusterClient and call #listJobs to obtain the list of jobs being executed on the cluster (note that it will also report finished jobs). By providing a properly configured Configuration (e.g. loading flink-conf.yaml via GlobalConfiguration#loadConfiguration) it will automatically detect where the JobManager is running (e.g. via ZooKeeper if HA is enabled or it picks up the configured JobManager address from the configuration). Of course, you could also provide the JobManager address as a parameter. Cheers, Till On Mon, Sep 23, 2019 at 9:08 AM David Morin wrote: > Hi, > > Thanks for your replies. > Yes, it could be useful to have a way to define jobid. Thus, I would have > been able to define the jbid based on the name for example. At the moment > we do not use the REST API but the cli to submit our jobs on Yarn. > Nevertheless, I can implement a little trick: at startup query the Rest > API and throw an Exception if a job with the same same is running. > Question: is there a way to retrieve the Job manager uri from my code or > should I provide it as parameter ? > thx. > David > > On 2019/09/23 03:09:42, Zili Chen wrote: > > The situation is as Dian said. Flink identifies jobs by job id instead = of > > job name. > > > > However, I think it is still a valid question if it is an alternative > Flink > > identifies jobs by job name and > > leaves the work to distinguish jobs by name to users. The advantages in > > this way includes a readable > > display and interaction, as well as reduce some hardcode works on job i= d, > > such as we always set > > job id to new JobID(0, 0) in standalone per-job mode for getting the sa= me > > ZK path. > > > > Best, > > tison. > > > > > > Dian Fu =E4=BA=8E2019=E5=B9=B49=E6=9C=8823=E6= =97=A5=E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D=8810:55=E5=86=99=E9=81=93=EF=BC=9A > > > > > Hi David, > > > > > > The jobs are identified by job id, not by job name internally in Flin= k > and > > > so It will only check if there are two jobs with the same job id. > > > > > > If you submit the job via CLI[1], I'm afraid there are still no > built-in > > > ways provided as currently the job id is generated randomly when > submitting > > > a job via CLI and the generated job id has nothing to do with the job > name. > > > However, if you submit the job via REST API [2], it did provide an > option > > > to specify the job id when submitting a job. You can generate the job > id by > > > yourself. > > > > > > Regards, > > > Dian > > > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/ops/cli.html > > > [2] > > > > https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_ap= i.html#jars-jarid-run > > > > > > =E5=9C=A8 2019=E5=B9=B49=E6=9C=8823=E6=97=A5=EF=BC=8C=E4=B8=8A=E5=8D= =884:57=EF=BC=8CDavid Morin =E5=86=99=E9=81=93= =EF=BC=9A > > > > > > Hi, > > > > > > What is the best way to prevent from launching 2 jobs with the same > name > > > concurrently ? > > > Instead of doing a check in the script that starts the Flink job, I > would > > > prefer to stop a job if another one with the same name is in progress > > > (Exception or something like that). > > > > > > David > > > > > > > > > > > > --000000000000e5c597059337ac7a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi David,

you could use Flink's Res= tClusterClient and call #listJobs to obtain the list of jobs being executed= on the cluster (note that it will also report finished jobs). By providing= a properly configured Configuration (e.g. loading flink-conf.yaml via Glob= alConfiguration#loadConfiguration) it will automatically detect where the J= obManager is running (e.g. via ZooKeeper if HA is enabled or it picks up th= e configured JobManager address from the configuration).

Of course, you could also provide the JobManager address as a parame= ter.

Cheers,
Till

On Mon, Sep 23, 2= 019 at 9:08 AM David Morin <morin.david.bzh@gmail.com> wrote:
Hi,

Thanks for your replies.
Yes, it could be useful to have a way to define jobid. Thus, I would have b= een able to define the jbid based on the name for example. At the moment we= do not use the REST API but the cli to submit our jobs on Yarn.
Nevertheless, I can implement a little trick: at startup query the Rest API= and throw an Exception if a job with the same same is running.
Question: is there a way to retrieve the Job manager uri from my code or sh= ould I provide it as parameter ?
thx.
David

On 2019/09/23 03:09:42, Zili Chen <wander4096@gmail.com> wrote:
> The situation is as Dian said. Flink identifies jobs by job id instead= of
> job name.
>
> However, I think it is still a valid question if it is an alternative = Flink
> identifies jobs by job name and
> leaves the work to distinguish jobs by name to users. The advantages i= n
> this way includes a readable
> display and interaction, as well as reduce some hardcode works on job = id,
> such as we always set
> job id to new JobID(0, 0) in standalone per-job mode for getting the s= ame
> ZK path.
>
> Best,
> tison.
>
>
> Dian Fu <dian0511.fu@gmail.com> =E4=BA=8E2019=E5=B9=B49=E6=9C=8823=E6=97=A5= =E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D=8810:55=E5=86=99=E9=81=93=EF=BC=9A
>
> > Hi David,
> >
> > The jobs are identified by job id, not by job name internally in = Flink and
> > so It will only check if there are two jobs with the same job id.=
> >
> > If you submit the job via CLI[1], I'm afraid there are still = no built-in
> > ways provided as currently the job id is generated randomly when = submitting
> > a job via CLI and the generated job id has nothing to do with the= job name.
> > However, if you submit the job via REST API [2], it did provide a= n option
> > to specify the job id when submitting a job. You can generate the= job id by
> > yourself.
> >
> > Regards,
> > Dian
> >
> > [1] https://ci.apache.o= rg/projects/flink/flink-docs-master/ops/cli.html
> > [2]
> > https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_= api.html#jars-jarid-run
> >
> > =E5=9C=A8 2019=E5=B9=B49=E6=9C=8823=E6=97=A5=EF=BC=8C=E4=B8=8A=E5= =8D=884:57=EF=BC=8CDavid Morin <morin.david.bzh@gmail.com> =E5=86=99=E9=81=93= =EF=BC=9A
> >
> > Hi,
> >
> > What is the best way to prevent from launching 2 jobs with the sa= me name
> > concurrently ?
> > Instead of doing a check in the script that starts the Flink job,= I would
> > prefer to stop a job if another one with the same name is in prog= ress
> > (Exception or something like that).
> >
> > David
> >
> >
> >
>
--000000000000e5c597059337ac7a--