From user-return-21961-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Aug 8 08:18:47 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C906C180600 for ; Wed, 8 Aug 2018 08:18:46 +0200 (CEST) Received: (qmail 54523 invoked by uid 500); 8 Aug 2018 06:18:45 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 54513 invoked by uid 99); 8 Aug 2018 06:18:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Aug 2018 06:18:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0BC56C7ADF for ; Wed, 8 Aug 2018 06:18:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.995 X-Spam-Level: * X-Spam-Status: No, score=1.995 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FROM_EXCESS_BASE64=0.105, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=naver.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id gygMMXFvMZUq for ; Wed, 8 Aug 2018 06:18:43 +0000 (UTC) Received: from cnsmtp02.nm.naver.com (cnsmtp02.nm.naver.com [125.209.212.110]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EA96E5F41A for ; Wed, 8 Aug 2018 06:18:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=naver.com; s=s20171208; t=1533709112; bh=BzN3EuZpho2U56/KztcEFiLm4oDcAjbvJwc5+CnI0BY=; h=From:Date:Subject:Message-Id:To; b=qwC6+cLSHBqXp2klDA0rOjfITiFnJBRQprJo8f/EX2g67SYP1A+SjtSjBgVfm+Z3N NwF+t+hppeaqVajQv5+DWRu6Y9C3t7F80iwkuQ1uFJrL9xzOjLPiHar06e3AtupvVy NZxxZww+kb0auhjVMbzp11S2G/Xc97Mn3shYzF4Mh4w8paRmmW+njFHrMtkWZpQeYw WxglTEgsg6w+kmR6chj3o3XENloK9Fsbkd7bwafatW6jrvyESiKuBZyVNvrkHuzUEo AOB4aXQmPDV4bKo5QSBDYe4OGZmZxgG5570z/xUMB+Up2bNihQyFPn5fnq0VNbzccd 6YhoJ/L0L7qZw== X-Naver-CIP: 73.158.253.211 Received: from [10.0.0.246] (c-73-158-253-211.hsd1.ca.comcast.net [73.158.253.211]) by cnsmtp02.nm.naver.com with ESMTP id np4QITlRRb2RNsnoET6PHA for ; Wed, 08 Aug 2018 06:18:32 -0000 From: =?utf-8?B?64Sk7J2067KE?= Content-Type: multipart/alternative; boundary=Apple-Mail-EAD812DD-5173-41CA-BD59-9A964D2C5197 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Date: Tue, 7 Aug 2018 23:18:30 -0700 Subject: unsubscribtion Message-Id: <9D4DF815-DE8A-43D3-A08A-7C7483A2E368@naver.com> To: user X-Mailer: iPhone Mail (15G77) --Apple-Mail-EAD812DD-5173-41CA-BD59-9A964D2C5197 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > On 7 Aug 2018, at 19:42, Yan Zhou [FDS Science] wrote:= >=20 > Thank you Vino. It is very helpful. > From: vino yang > Sent: Tuesday, August 7, 2018 7:22:50 PM > To: Yan Zhou [FDS Science] > Cc: user > Subject: Re: checkpoint recovery behavior when kafka source is set to star= t from timestamp > =20 > Hi Yan Zhou: >=20 > I think the java doc of the setStartFromTimestamp method has been explaine= d very clearly, posted here: >=20 > /** > * Specify the consumer to start reading partitions from a specified timest= amp. > * The specified timestamp must be before the current timestamp. > * This lets the consumer ignore any committed group offsets in Zookeeper /= Kafka brokers. > * > *

The consumer will look up the earliest offset whose timestamp is grea= ter than or equal > * to the specific timestamp from Kafka. If there's no such offset, the con= sumer will use the > * latest offset to read data from kafka. > * > *

This method does not affect where partitions are read from when the c= onsumer is restored > * from a checkpoint or savepoint. When the consumer is restored from a che= ckpoint or > * savepoint, only the offsets in the restored state will be used. > * > * @param startupOffsetsTimestamp timestamp for the startup offsets, as mil= liseconds from epoch. > * > * @return The consumer object, to allow function chaining. > */ >=20 > Thanks, vino. >=20 > Yan Zhou [FDS Science] =E4=BA=8E2018=E5=B9=B48=E6=9C=88= 8=E6=97=A5=E5=91=A8=E4=B8=89 =E4=B8=8A=E5=8D=889:06=E5=86=99=E9=81=93=EF=BC=9A= > Hi Experts, >=20 > In my application, the kafka source is set to start from a specified times= tamp, by calling method FlinkKafkaConsumer010#setStartFromTimestamp(long sta= rtupOffsetsTimestamp).=20 >=20 > If the application have run a while and then recover from a checkpoint bec= ause of failure, what's the offset will the kafka source to read from? I sup= pose it will read from the offset that has been committed before the failure= . Is it right?=20 >=20 > I am going to verify it, however some clarification is good in case my tes= t result doesn't meet my assumption. >=20 > Best > Yan >=20 --Apple-Mail-EAD812DD-5173-41CA-BD59-9A964D2C5197 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable



On 7 Aug= 2018, at 19:42, Yan Zhou [FDS Science] <yzhou@coupang.com> wrote:

Thank you Vino. It is very helpful= .

From: vino yang <yanghua1127@gmail.com>
Sent: Tuesday, August 7, 2018 7:22:50 PM
To: Yan Zhou [FDS Science]
Cc: user
Subject: Re: checkpoint recovery behavior when kafka source is set to= start from timestamp
 
Hi Yan Zhou:

I think the java doc of the setStartFromTimestamp method has been expla= ined very clearly, posted here:

/**
* Specify the consumer to start reading partitions from a specifi= ed timestamp.
* The specified timestamp must be before the current timestamp.
* This lets the consumer ignore any committed group offsets in Zo= okeeper / Kafka brokers.
*
* <p>The consumer will look up the earliest offset whose ti= mestamp is greater than or equal
* to the specific timestamp from Kafka. If there's no such offset= , the consumer will use the
* latest offset to read data from kafka.
*
* <p>This method does not affect where partitions are read f= rom when the consumer is restored
* from a checkpoint or savepoint. When the consumer is restored f= rom a checkpoint or
* savepoint, only the offsets in the restored state will be used.=
*
* @param startupOffsetsTimestamp timestamp for the startup offset= s, as milliseconds from epoch.
*
* @return The consumer object, to allow function chaining.
*/

Thanks, vino.

Yan Zhou [FDS Science] <yzhou@coupang.com> =E4=BA=8E2018=E5=B9=B48=E6=9C=888=E6=97=A5=E5= =91=A8=E4=B8=89 =E4=B8=8A=E5=8D=889:06=E5=86=99=E9=81=93=EF=BC=9A

Hi Experts,


In my application, the k= afka source is set to start from a specified timestamp, by callin= g method FlinkKafkaConsumer010#setStartFromTimestamp(long s= tartupOffsetsTimestamp). 


If the applicat= ion have run a while and then recover from a checkpoint because of failure, w= hat's the offset will the kafka source to read from? I suppose it will read f= rom the offset that has been committed before the failure. Is it right? 


I am going to v= erify it, however some clarification is good in case my test result doesn't meet my assumptio= n.


Best

Yan


= --Apple-Mail-EAD812DD-5173-41CA-BD59-9A964D2C5197--