From user-return-2707-archive-asf-public=cust-asf.ponee.io@beam.apache.org Wed Jan 10 19:34:54 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 3BE6518072F for ; Wed, 10 Jan 2018 19:34:54 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 2BE61160C2E; Wed, 10 Jan 2018 18:34:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7B9C6160C1E for ; Wed, 10 Jan 2018 19:34:53 +0100 (CET) Received: (qmail 56267 invoked by uid 500); 10 Jan 2018 18:34:52 -0000 Mailing-List: contact user-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@beam.apache.org Delivered-To: mailing list user@beam.apache.org Received: (qmail 56257 invoked by uid 99); 10 Jan 2018 18:34:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jan 2018 18:34:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 02D6AC2A3E for ; Wed, 10 Jan 2018 18:34:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.87 X-Spam-Level: * X-Spam-Status: No, score=1.87 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, HTML_OBFUSCATE_05_10=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id OFZZXOO-EyLS for ; Wed, 10 Jan 2018 18:34:50 +0000 (UTC) Received: from mail-it0-f43.google.com (mail-it0-f43.google.com [209.85.214.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B898B5F2C4 for ; Wed, 10 Jan 2018 18:34:49 +0000 (UTC) Received: by mail-it0-f43.google.com with SMTP id d137so540678itc.2 for ; Wed, 10 Jan 2018 10:34:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=M9trfbTOcdLptQBUrsDBl1TsCcLcr9OrLHsHNZ1shv0=; b=hfNw5BN9W9pnPlsgy5DkifDwGABprkfdGpLYlnt/+63ZXjxbn4yi8D2DTk5w8RYMqk KDz3b0Jl4mwqJWSkc/V3PcFz8OOMJJFoPISVsBLcQlBQ5jg4plv2VGChZpUOi76KJQuF G20hn4cOCi7iyfuY1/XHk24/nVYTjV4MkYtCA3FX6uL9lUsxP05qlUJE6mjHRmp4Rqzy 4qfKPWkBamGLNQGmWY+p3ahDgJVDkJ574PNwH2z0IVfbA3Irm1xMottTT1NlJt8bcHDl tuiD2bjGEcbNCf/BxV3CqAUJ7IEJ8QSB4vZRGgmft3IcPuSjFqEU5BZ/RN8rUtW/u/bU iEvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=M9trfbTOcdLptQBUrsDBl1TsCcLcr9OrLHsHNZ1shv0=; b=ivMeX8HjoTQGFhfkJXzgr5FdkSQqvq1oLNIwxrCDEdiOQklhuR5E2ZKNttYcA03Di3 +qU9ORK57enbvFCMUh3N+/dh5l6dosw3x17RbLuostq3SE9WfPE17y+zDZcz9y8nwq/n k1rR7d+3FFc3cTjm6f9GXBuZ0pxfr4aizm1f9eTOXxShOJRavKH0Og+YIs4Ysn8p4MTi gYgVw34FocIXNOuJCYQ0kmf+/t7GIWJ1Y4CxxzW3VcJKnAa4cq+u9I7Og6ZiBMG+Jdjg RiOsFOvn0ZIkI53oXxtXXbifhbq22LxMOq3gcyZs+1iuqgyqa2x1peTWWusfw4tFEcSw nRjw== X-Gm-Message-State: AKGB3mJiwxBfuFHpkFBkjX03t6X/H415YAsNIZ2bvePVmDzZX3pKXcbM q8kJdShMJIIRsUnELdbDYj9Apw/IxKl6R3XUckg876td X-Google-Smtp-Source: ACJfBouYTTXvt7OZFuF+iDnbc5JCiwf5BmaqyefiQE4E22Ie3dk7VJLOD4xopIvI2FvY+IrC6LaSUuAD0SD7IKf+RX8= X-Received: by 10.36.164.6 with SMTP id z6mr18970382ite.2.1515609288632; Wed, 10 Jan 2018 10:34:48 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.132.217 with HTTP; Wed, 10 Jan 2018 10:34:48 -0800 (PST) In-Reply-To: References: <795A4AC2-79D3-4DA1-B74A-43C919F73C85@apache.org> From: Raghu Angadi Date: Wed, 10 Jan 2018 10:34:48 -0800 Message-ID: Subject: Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner To: user Cc: dev@beam.apache.org Content-Type: multipart/alternative; boundary="f403045fad4845421c0562704976" --f403045fad4845421c0562704976 Content-Type: text/plain; charset="UTF-8" How often does your pipeline checkpoint/snapshot? If the failure happens before the first checkpoint, the pipeline could restart without any state, in which case KafkaIO would read from latest offset. There is probably some way to verify if pipeline is restarting from a checkpoint. On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks wrote: > HI Aljoscha, > The issue is let's say I consumed 100 elements in 5 > mins Fixed Window with *GroupByKey* and later I applied *ParDO* for all > those elements. If there is an issue while processing element 70 in > *ParDo *and the pipeline restarts with *UserCodeException *it's skipping > the rest 30 elements. Wanted to know if this is expected? In case if you > still having doubt let me know will share a code snippet. > > Regards, > Sushil Ks > --f403045fad4845421c0562704976 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
How often does your pipeline checkpoint/snapshot? If the f= ailure happens before the first checkpoint, the pipeline could restart with= out any state, in which case KafkaIO would read from latest offset. There i= s probably some way to verify if pipeline is restarting from a checkpoint.<= /div>

On Sun, Jan = 7, 2018 at 10:57 PM, Sushil Ks <sushil416@gmail.com> wrote= :
HI Aljoscha,
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The issue=C2= =A0is let's say I consumed 100 elements in 5 mins Fixed Window with = GroupByKey=C2=A0and later I applied ParDO=C2=A0for all those ele= ments. If there is an issue while processing element 70 in ParDo=C2=A0and the pipeline restarts with UserCodeException it's skipping= the rest 30 elements. Wanted to know if this is expected? In case if you s= till having doubt let me know will share a code snippet.

Regards,Sushil Ks

--f403045fad4845421c0562704976--