From user-return-18145-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Feb 13 21:39:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4ECAF180656 for ; Tue, 13 Feb 2018 21:39:01 +0100 (CET) Received: (qmail 60841 invoked by uid 500); 13 Feb 2018 20:38:59 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 60831 invoked by uid 99); 13 Feb 2018 20:38:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Feb 2018 20:38:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6C2B5C0110 for ; Tue, 13 Feb 2018 20:38:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.88 X-Spam-Level: * X-Spam-Status: No, score=1.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, HTML_OBFUSCATE_05_10=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 4Ae0SqERY5hy for ; Tue, 13 Feb 2018 20:38:58 +0000 (UTC) Received: from mail-yw0-f179.google.com (mail-yw0-f179.google.com [209.85.161.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 37A955F17F for ; Tue, 13 Feb 2018 20:38:58 +0000 (UTC) Received: by mail-yw0-f179.google.com with SMTP id v196so13238278ywc.6 for ; Tue, 13 Feb 2018 12:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=3UpCv0qfvfHvlgQ8SO6UpUukHSkEGiah9WPZyzVvSog=; b=CUt+eKww9CW5F2lSj01RQNEWzI9v7eSrgVTdwdp37bdeihhCVpCgh3sZnR8qHR5/zT 319dyRR5Bauy3sVcoZZKIgDFj/pIMyvsUEL0gVrQ17iC9oyOZ6/TJ4pD02TRs4gWqfpR kwHpvNruuQWdGpt9ZE2dAOy3I10jUzUrpagvEC377PlVaQwPTYlIK3AFi8spNGLt+i5Z VUsmAVcGovBBEP2fFluEuYPfm8fjqoDSholnKR9HKYVzgVfpQIu0vHGwu+kjeYcqQfEu G+U3DkkVCVz6A6y8hToq5RnXmQnWxmkoxNPRUlpF5HKNHnEB6aQ9n6yN1kgdtJ7DOBKP YD+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=3UpCv0qfvfHvlgQ8SO6UpUukHSkEGiah9WPZyzVvSog=; b=fix9LPBzUPj/KKaL2t5ZbWtTHA3aRH15F1zuRQ8cx8gs7N7i08fwpbuWO3VNMspEK8 EZpVi6OHxoH7p6501012YmY3ml58A8abel/phLoA5D7MhB7Rt0/z45jfpd0nNAT42vhG xBn9RkMnYrjdu9AbrqZkJGWbmbqgT6eT4f15r9yhYcbSP/itP8rK0qnHd++zbS2/CIsL 6ZAQYiBVHSJAYxpXS8gDf/WLlQuCzgrC3LpUTSMJVPB0yn3uxBtXXczqOJ52VQ0zbvco f2thwwNg10hE54NtXrYbkF+Oz9N7itZ40OLgbHNIGM+OKz5cxeYecAQdlZWB4lda5Qd9 6AxQ== X-Gm-Message-State: APf1xPBla20d5dDzAZJYUK8ubmzHE5uaVlcXnmvyT00JLgs3hMVtg0T3 uvomtX1Uqj6uR8mwZn7uZE/qooRmKqHwNbzv/R+Pib8N+w== X-Google-Smtp-Source: AH8x227nh/3Pbe72CBJ0IIXZ0hHwKYzbpHD5cBPf9qJb+SIfTPCU3Ve3F7MmJfneDY7SRcIpJlodFY4kx2+u40XzfwA= X-Received: by 10.129.56.194 with SMTP id f185mr1628470ywa.8.1518554337681; Tue, 13 Feb 2018 12:38:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.13.244.132 with HTTP; Tue, 13 Feb 2018 12:38:57 -0800 (PST) From: =?UTF-8?Q?Krzysztof_Bia=C5=82ek?= Date: Tue, 13 Feb 2018 21:38:57 +0100 Message-ID: Subject: Flink + Consul as HA backend. What do you think? To: user@flink.apache.org Content-Type: multipart/alternative; boundary="001a114c7b3adea5d505651dfb4d" --001a114c7b3adea5d505651dfb4d Content-Type: text/plain; charset="UTF-8" I'd like to get your opinion about this idea. I found related JIRA issue FLINK-2366, but it seems to be dead. To attract your attention I copy my comment here. As an experiment I've implemented Flink HA on top of Consul. The implementation is working fine in the "lab" but is not battle tested yet. The source code is available at https://github.com/kbialek/ flink/tree/feature/consul (flink-runtime package org.apache.flink.runtime. consul) Why?. Generally I'd like to keep as less moving parts as possible. We do not have Zookeeper running, but Consul is already in place. And in the end freedom of choice is a good thing. It would be great to see built-in Consul support in Flink someday, but if it is not expected then I suggest a little refactoring to open possibility to configure HighAvailabilityServicesFactory. As far as I can see this should be enough to inject any HA implementation. Regards, Krzysztof --001a114c7b3adea5d505651dfb4d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I'd like to get your opinion about this idea. I found rel= ated JIRA issue=C2=A0FLINK-2366, but= it seems to be dead. To attract your attention I copy my comment here.

As an experiment I've implemented Flink HA on top of = Consul. The implementation is working fine in the "lab" but is no= t battle tested yet. The source code is available at=C2=A0https://github.com/kbialek/flink/tr= ee/feature/consul=C2=A0(flink-runtime package org.apache.f= link.runtime.consul)

Why?. Generally I'd = like to keep as less moving parts as possible. We do not have Zookeeper run= ning, but Consul is already in place. And in the end freedom of choice is a= good thing.

It would be great to see built-in Con= sul support in Flink someday, but if it is not expected then I suggest a li= ttle refactoring to open possibility to configure HighAvailabilityServicesF= actory. As far as I can see this should be enough to inject any HA imp= lementation.

Regards,
Krzysztof

--001a114c7b3adea5d505651dfb4d--