From user-return-32667-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Feb 17 18:06:50 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D24F4180636 for ; Mon, 17 Feb 2020 19:06:49 +0100 (CET) Received: (qmail 67871 invoked by uid 500); 17 Feb 2020 18:06:48 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 67860 invoked by uid 99); 17 Feb 2020 18:06:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2020 18:06:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CF69DC019E for ; Mon, 17 Feb 2020 18:06:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Fd6KTqpoPQ2S for ; Mon, 17 Feb 2020 18:06:46 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::334; helo=mail-ot1-x334.google.com; envelope-from=lehuede.s@gmail.com; receiver= Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id ED9707E133 for ; Mon, 17 Feb 2020 18:06:45 +0000 (UTC) Received: by mail-ot1-x334.google.com with SMTP id i6so16940203otr.7 for ; Mon, 17 Feb 2020 10:06:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=uuyqA1XgS8iHbqWFklrk3FHPk7nosjZehNTUAiGJcgw=; b=Zk4SYzMtK/zeU4fBDZTlH+Fc0frzradbje9xTyPfqiAzMbsMdWHUObUaWiGeTkeWDB 3JuHFBQQquJFK5seTI7FIfdGPOeeHRz/UbCGVH9BwLFNkPv4T9bKHOlEfCvEfnt2xH4k x0K0z3Mo55lvzL520k0f9OXsThWz8YL+vir4kODQjQZOzk+qRc0Se/4v/tbT4DyjYEbq fm2Q9mnLNlzf0SMdzGs+1TOeG5QfbSfGXfhk8JU7aRJyW2o2H2a9aI05ZxFL9ZIm4Eff h9BTrr1GN4jIc98yLi0V7OLAH3h8ONnIV+kvGPp0g1SLIutyQ8uWouUE/P/ynKFcztIT pYVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uuyqA1XgS8iHbqWFklrk3FHPk7nosjZehNTUAiGJcgw=; b=SdbeXSyfhxVRfCx9jKcQFKzunhuYiA14NXvcMoUU5BhRQniMs6RWAOwKhZutrTctv3 rFtLfirjoasKCB/LtjnUHUP4z6Ejj/HFiyR6uoEwiT1ZfyDFkhpc92GgaphORxXZsnBN 92SFHypQdVD7sNBdu8P0JARVs4zFVikvVip6OCebFaYyUnPD0JjiMDI14f2+eGUQ4DMf 03etjcEecrwzaK4FNDu8V2oqKvNsd9kx5MOhldDvKfqPfYekMHpeSkFgXroQ8mKkVwEP nwqMC+blDlqoy2gm9S2p4WCXcUTOqOC3dvEhT1fIb2BebZVl2OcveUaLd4dkxRRnjxsm /OFg== X-Gm-Message-State: APjAAAWoe0IJC3/qR27nW5tHorquggiKBHvaDoXUr+vUj0FzeQN3i4Ow uJEf+j9FpyrsamFV9IzucVOCSBDkJJ1+EOA9v95BIrE3 X-Google-Smtp-Source: APXvYqzgAWvfzkDyuwD1SYvl8KPi9Yft5aewAdWu0xbCEw83dHmkMvQ48iw8iJnseM+kSkGSuMTRx+qkLvhNSUWSlUM= X-Received: by 2002:a9d:6183:: with SMTP id g3mr13084634otk.304.1581962804384; Mon, 17 Feb 2020 10:06:44 -0800 (PST) MIME-Version: 1.0 From: Lehuede sebastien Date: Mon, 17 Feb 2020 10:06:33 -0800 Message-ID: Subject: Process stream multiple time with different KeyBy To: user@flink.apache.org Content-Type: multipart/alternative; boundary="00000000000000d7e3059ec96b63" --00000000000000d7e3059ec96b63 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi all, I'm currently working on a Flink Application where I match events against a set of rules. At the beginning I wanted to dynamically create streams following the category of events (Event are JSON formatted and I've a field like "category":"foo" in each event) but I'm stuck by the impossibility to create streams at runtime. So, one of the solution for me is to create a single Kafka topic and then use the "KeyBy" operator to match events with "category":"foo" against rules also containing "category":"foo" in rule specification. Now I have some cases where events and rules have one category and one subcategory. At this point I'm not sure about the "KeyBy" operator behavior= . Example : - Events have : "category":"foo" AND "subcategory":"bar" - Rule1 specification has : "category":"foo" AND "subcategory":"bar" - Rule2 specification has : "category':"foo" - Rule3 specification has : "category":"bar" In this case, my events need to be match against Rule1, Rule2 and Rule3. If I'm right, if I apply a multiple key "KeyBy()" with "category" and "subcategory" fields and then apply two single key "KeyBy()" with "category" field, my events will be consumed by the first "KeyBy()" operator and no events will be streamed in the operators after ? Is there any way to process the same stream one time for multi key KeyBy() and another time for single key KeyBy() ? Thanks ! S=C3=A9bastien. --00000000000000d7e3059ec96b63 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,

I'm currently working = on a Flink Application where I match events against a set of rules. At the = beginning I wanted to dynamically create streams following the category of = events (Event are JSON formatted and I've a field like "category&q= uot;:"foo" in each event) but I'm stuck by the impossibility = to create streams at runtime.

So, one of the solut= ion for me is to create a single Kafka topic and then use the "KeyBy&q= uot; operator to match events with "category":"foo" aga= inst rules also containing "category":"foo" in rule spe= cification.=C2=A0

Now I have some cases where even= ts and rules have one category and one subcategory. At this point I'm n= ot sure about the "KeyBy" operator behavior.

=
Example :
  • Events have : "category":"f= oo" AND "subcategory":"bar"
  • Rule1 specific= ation has : "category":"foo" AND "subcategory"= ;:"bar"=C2=A0 =C2=A0
  • Rule2 specification has : "cate= gory':"foo"
  • Rule3 specification has : "category&= quot;:"bar"
In this case, my events need to be matc= h against Rule1, Rule2 and Rule3.=C2=A0

If I'm right, if I apply a multiple key "KeyBy()" with &quo= t;category" and "subcategory" fields and then apply two sing= le key "KeyBy()" with "category" field, my events will = be consumed by the first "KeyBy()" operator and no events will be= streamed in the operators after ?=C2=A0

Is there = any way to process the same stream one time for multi key KeyBy() and anoth= er time for single key KeyBy() ?=C2=A0

Thanks !
S=C3=A9bastien.=C2=A0=C2=A0
--00000000000000d7e3059ec96b63--