Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 882C6200C68 for ; Wed, 3 May 2017 12:37:16 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 851F2160BB5; Wed, 3 May 2017 10:37:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A3AA2160BAA for ; Wed, 3 May 2017 12:37:15 +0200 (CEST) Received: (qmail 93131 invoked by uid 500); 3 May 2017 10:37:14 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 93121 invoked by uid 99); 3 May 2017 10:37:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 May 2017 10:37:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 40E6C18F165 for ; Wed, 3 May 2017 10:37:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.88 X-Spam-Level: * X-Spam-Status: No, score=1.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 5k03r84vMTwZ for ; Wed, 3 May 2017 10:37:12 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 954455FB29 for ; Wed, 3 May 2017 10:37:11 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id l18so82742195ywh.3 for ; Wed, 03 May 2017 03:37:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=5Wky83cmwNl7hOebkkDPZU38FNWcoBuVp4ry1v+syZM=; b=IPGrdc6d/byQKsaS2GYssshWkDEeQyZtNLFiWCgPOBqf6wMdKpytb20WxUJ8yaEOct Vw3xWkhJKS/gnkd2WUmpe6cyi9x7oog2175EB3zNIkFHyyMmS6TOO18+p2xlo/GXgeEg ZTVGTBJCy+dO3KzQLvq9yAvkFKsox7FrJ7Csdq2aZ7yZZrkmQUt0hwPEtw1ScCOdgihw B1a9tW694YaBO67CrCzT56t7iDQcgJ2nn8EOAOZqV6cS9XmMNfKF0Qr8Rv6MEjoRLs25 AwK7saxhb8kiS65YQBK5FxwFE3b4R02iuHWa/ydoZy7nEbXNTZ8cnzXrEORf+1AtxO4k gmAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=5Wky83cmwNl7hOebkkDPZU38FNWcoBuVp4ry1v+syZM=; b=bhNuw5fvnJrKWxXJpfl5qgi75W7B1GUfT9lSg3Knekzf2J0+rSm4e8yo1eeig7O+OY 2bBcxYrutnJ2tBnJJY91gfiYgfve+8gT9K1KSEeZV70/QUZh7wN/o8+vY+R/1p/Dg00B fvU2ZXWZI3rnvZxDyBbkOLm5Jq2FmVbjm4dbjLiGGKzAqZXW/QUVGV5+ixFFrp+DArj2 txLlXoNvtzU/4y/RAWGI4Yuanolv0TuzTB6KRpUsD2D37PHLeXS0ZW0+XGomGqJ+VOIs vSJmGQVduOAR8YoL/OQM/pO1uNoe6VHu3LWKe4SLrkTn2l9c1dSMMKYhWNm7PyyvMVtC jPrg== X-Gm-Message-State: AN3rC/4++R7OHDPNv+E26by0ktWBugOXOysWS2VBr4ijXIdmngRy9CtO dnIkWA/uEV3k7uENyTx3K7uLy2tmww== X-Received: by 10.129.67.15 with SMTP id q15mr21428167ywa.186.1493807830516; Wed, 03 May 2017 03:37:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.97.85 with HTTP; Wed, 3 May 2017 03:36:50 -0700 (PDT) In-Reply-To: References: <0E534A40-B11D-42A1-9860-2A54F3E65E8B@apache.org> From: Moiz S Jinia Date: Wed, 3 May 2017 16:06:50 +0530 Message-ID: Subject: Re: Programmatic management of Flink jobs To: Aljoscha Krettek Cc: user@flink.apache.org Content-Type: multipart/alternative; boundary=f403045ec20c19e2a2054e9c3dfb archived-at: Wed, 03 May 2017 10:37:16 -0000 --f403045ec20c19e2a2054e9c3dfb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params. Does that sound like it'd work or am I missing something? Moiz On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia wrote: > Not sure I understand Operators. What I need is to have a Pattern that > starts consuming from a Kafka stream. And I need the Patterns to come and > go. > > Another option that comes to mind is this - > The Patterns I'll need are well known in advance. Only certain parameters > such as the time duration of the within clause, and maybe certain filter > confitions of the where clause need tweaking. So I could pre-deploy the > Patterns (or jobs) and start or stop them (with parameters). > > Does that sound feasible? > > On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek > wrote: > >> What would the pattern be added to. An existing custom operator? >> >> The REST interface only allows for managing the lifecycle of a job, not >> modifying their graph structure. >> >> On 3. May 2017, at 11:43, Moiz S Jinia wrote: >> >> Thanks for the references. Looking at the REST API, would adding new >> Patterns not work via this? >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.3/ >> monitoring/rest_api.html#submitting-programs >> >> Moiz >> >> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek >> wrote: >> >>> Hi, >>> For managing a Job you can either use the bin/flink command-line tool o= r >>> the Rest API [1]. As for dynamically adding patterns, that=E2=80=99s ou= tside of the >>> scope of Flink right now. There are, however, some users that implement= ed >>> this on top of Flink, see for example RBEA [2]. The basic idea is to us= e a >>> ConnectedStream where one input is the main input and the other input i= s a >>> control stream that updates the existing patterns. >>> >>> [1] https://ci.apache.org/projects/flink/flink-docs-release- >>> 1.3/monitoring/rest_api.html >>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ >>> >>> On 3. May 2017, at 10:02, Moiz S Jinia wrote: >>> >>> Is there an API that allows remotely adding, modifying, and cancelling >>> Flink jobs? Example - changing the time window of a deployed Pattern, >>> adding new Patterns, etc. >>> >>> Whats the best way to go about this? To the end user the Pattern would >>> manifest as rules that can be updated anytime. >>> >>> >>> Moiz >>> >>> >>> >> >> > --f403045ec20c19e2a2054e9c3dfb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The kind of program I intend to submit would be one that s= ets up a=C2=A0StreamExecutionEnvironment, connects to a stream from a Kafka= topic, and uses a=C2=A0PatternStream over the kafka events. I could have t= he jar for this program readily available in the "jobmanager.web.uploa= d.dir", and use the REST API for submitting a program with some config= uration params.

Does that sound like it'd work or am= I missing something?

Moiz

On Wed, May 3, 2017 at 3:23= PM, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
Not sure I understand Operators. = What I need is to have a Pattern that starts consuming from a Kafka stream.= And I need the Patterns to come and go.

Another option that comes t= o mind is this -
The Patterns I'll need are well known in advance. O= nly certain parameters such as the time duration of the within clause, and = maybe certain filter confitions of the where clause need tweaking. So I cou= ld pre-deploy the Patterns (or jobs) and start or stop them (with parameter= s).

Does that sound feasible?

On Wed, M= ay 3, 2017 at 3:15 PM, Aljoscha Krettek <aljoscha@apache.org> wrote:
What would the pattern be added to. An existing custom operator?
=
The REST interface only allows for managing the lifecycle of= a job, not modifying their graph structure.=C2=A0

On 3. May 2017, = at 11:43, Moiz S Jinia <moiz.jinia@gmail.com> wrote:

Thanks for the references. Looking at the REST API, would adding n= ew Patterns not work via this?

<= div class=3D"gmail_quote">On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek = <aljoscha@apache.org> wrote:
Hi,
For managing a Job you can= either use the bin/flink command-line tool or the Rest API [1]. As for dyn= amically adding patterns, that=E2=80=99s outside of the scope of Flink righ= t now. There are, however, some users that implemented this on top of Flink= , see for example RBEA [2]. The basic idea is to use a ConnectedStream wher= e one input is the main input and the other input is a control stream that = updates the existing patterns.

[2]=C2=A0https://techblog.king.com/rbea-scalable-real-time-a= nalytics-king/

On 3. May = 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com> wrote:

Is there an API that allows remotely addi= ng, modifying, and cancelling Flink jobs? Example - changing the time windo= w of a deployed Pattern, adding new Patterns, etc.

Whats= the best way to go about this? To the end user the Pattern would manifest = as rules that can be updated anytime.


Moiz



--f403045ec20c19e2a2054e9c3dfb--