Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DC506200C4B for ; Mon, 20 Mar 2017 15:44:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DAC47160B81; Mon, 20 Mar 2017 14:44:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2CE0D160B76 for ; Mon, 20 Mar 2017 15:44:07 +0100 (CET) Received: (qmail 17520 invoked by uid 500); 20 Mar 2017 14:44:05 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 17510 invoked by uid 99); 20 Mar 2017 14:44:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Mar 2017 14:44:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D2C6E1AFBFA for ; Mon, 20 Mar 2017 14:44:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id b7nM7cICgurc for ; Mon, 20 Mar 2017 14:44:03 +0000 (UTC) Received: from mail-it0-f45.google.com (mail-it0-f45.google.com [209.85.214.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D07795F341 for ; Mon, 20 Mar 2017 14:44:02 +0000 (UTC) Received: by mail-it0-f45.google.com with SMTP id g138so98084026itb.0 for ; Mon, 20 Mar 2017 07:44:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=r8VgerykC24SbjaNjudt/P7G6I7FH4fbDzM0hvw666U=; b=pwMsxZrqS1jbLi2NiMNHW6kmLGeDKBR8VFg9jqSIN2XmqYNLcElCNzcf4Xhp2lfSno eiG+EY5N7y2MKYyxlxB4L0FsxSIoFSsT1bV8GAVyI2gImYTw6MQZdcYUs+FYKSOquKvZ XB+F2ItW55zQ9DVH1nnXMgJ3fwRrYoR/9YesXBI5bAC8V8FSHiV9LOZpQ7ap9rgmYNuU KldD8aei4rqh9AtL1qD9fjLcvUq8xrNLL57v28U6msUR3MGfKGjGLivxxR0zjZZbpRDi QZLgYRDRWwSIohUJctnaB0iw4NzMls8N9vUHtGayRZSPmOITF8PQBJ6Cn42Zoq/QMFpB 7B2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=r8VgerykC24SbjaNjudt/P7G6I7FH4fbDzM0hvw666U=; b=ZYhUSk7xaaw7h3U4+p4FIqw/xjmfbHks1eMtus98SkU6jqcNDJF/2wZj1Ln22ZQk8I kkfrBzS2Jl+55u/5teHb1Hf+IAvA55oozIQrOs/w9ky3Q/7/M8UBl8EiICQdkRaDDOQg rkaXGSwms8xDQ6+mJroLFNG7RXsmFrTE+p88Mo53ojIrJfamn4/oU41/tNq0fY+3yqBz +H4RyxjXBogvZ9HiRQQ1U2FXpGh9z4jQWrWWQWgPh7iEonSDaep0AdnjfS+MCcSaHZ9g 4F4MqoyZ4nEe3VmuKLEaGmUGygTnt/5frkjLBthDFfZHWrzNWwNxNvUfE9wmnyjO0soS Xqzg== X-Gm-Message-State: AFeK/H06mXqWuvlmeJA8KA+Rs5HJO/cIFTu+Geyfl4NwePzWN2dVkobkuHl3oedJ7V8KuMhjkx76qCkcH7JgVg== X-Received: by 10.36.12.197 with SMTP id 188mr9460423itn.51.1490021031736; Mon, 20 Mar 2017 07:43:51 -0700 (PDT) MIME-Version: 1.0 References: <0104ACD3-1407-4A59-8D65-67CA7326F1C6@micardo.com> In-Reply-To: From: Bruno Aranda Date: Mon, 20 Mar 2017 14:43:40 +0000 Message-ID: Subject: Re: Telling if a job has caught up with Kafka To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a1144b87c4e30c6054b2a8eff archived-at: Mon, 20 Mar 2017 14:44:08 -0000 --001a1144b87c4e30c6054b2a8eff Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, Thanks! The proposal sounds very good to us too. Bruno On Sun, 19 Mar 2017 at 10:57 Florian K=C3=B6nig wrote: > Thanks Gordon for the detailed explanation! That makes sense and explains > the expected behaviour. > > The JIRA for the new metric also sounds very good. Can=E2=80=99t wait to = have this > in the Flink GUI (KafkaOffsetMonitor has some problems and stops working > after 1-2 days, don=E2=80=99t know the reason yet). > > All the best, > Florian > > > > Am 18.03.2017 um 08:38 schrieb Tzu-Li (Gordon) Tai >: > > > > @Florian > > the 0.9 / 0.10 version and 0.8 version behave a bit differently right > now for the offset committing. > > > > In 0.9 / 0.10, if checkpointing is enabled, the =E2=80=9Cauto.commit.en= able=E2=80=9D > etc. settings will be completely ignored and overwritten before used to > instantiate the interval Kafka clients, hence committing will only happen > on Flink checkpoints. > > > > In 0.8, this isn=E2=80=99t the case. Both automatic periodic committing= and > committing on checkpoints can take place. That=E2=80=99s perhaps why you= =E2=80=99re > observing the 0.8 consumer to be committing more frequently. > > > > FYI: This behaviour will be unified in Flink 1.3.0. If you=E2=80=99re > interested, you can take a look at > https://github.com/apache/flink/pull/3527. > > > > - Gordon > > > > > > On March 17, 2017 at 6:07:38 PM, Florian K=C3=B6nig ( > florian.koenig@micardo.com) wrote: > > > >> Why is that so? The checkpoint contains the Kafka offset and would be > able to start reading wherever it left off, regardless of any offset stor= ed > in Kafka or Zookeeper. Why is the offset not committed regularly, > independently from the checkpointing? Or did I misconfigure anything? > > > --001a1144b87c4e30c6054b2a8eff Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

Thanks! The proposal sounds very go= od to us too.

Bruno

On Sun, 19 Mar 2017 at 10:57 Florian K=C3=B6ni= g <florian.koenig@micardo.= com> wrote:
Thanks Gordon fo= r the detailed explanation! That makes sense and explains the expected beha= viour.

The JIRA for the new metric also sounds very good. Can=E2=80=99t wait to ha= ve this in the Flink GUI (KafkaOffsetMonitor has some problems and stops wo= rking after 1-2 days, don=E2=80=99t know the reason yet).

All the best,
Florian


> Am 18.03.2017 um 08:38 schrieb Tzu-Li (Gordon) Tai <tzulitai@apac= he.org>:
>
> @Florian
> the 0.9 / 0.10 version and 0.8 version behave a bit differently right = now for the offset committing.
>
> In 0.9 / 0.10, if checkpointing is enabled, the =E2=80=9Cauto.commit.e= nable=E2=80=9D etc. settings will be completely ignored and overwritten bef= ore used to instantiate the interval Kafka clients, hence committing will o= nly happen on Flink checkpoints.
>
> In 0.8, this isn=E2=80=99t the case. Both automatic periodic committin= g and committing on checkpoints can take place. That=E2=80=99s perhaps why = you=E2=80=99re observing the 0.8 consumer to be committing more frequently.=
>
> FYI: This behaviour will be unified in Flink 1.3.0. If you=E2=80=99re = interested, you can take a look at http= s://github.com/apache/flink/pull/3527.
>
> - Gordon
>
>
> On March 17, 2017 at 6:07:38 PM, Florian K=C3=B6nig (florian.= koenig@micardo.com) wrote:
>
>> Why is that so? The checkpoint contains the Kafka offset and would= be able to start reading wherever it left off, regardless of any offset st= ored in Kafka or Zookeeper. Why is the offset not committed regularly, inde= pendently from the checkpointing? Or did I misconfigure anything?


--001a1144b87c4e30c6054b2a8eff--