Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A8A5B200B8D for ; Fri, 9 Sep 2016 00:31:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A73E7160AD0; Thu, 8 Sep 2016 22:31:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A8CE8160AAD for ; Fri, 9 Sep 2016 00:31:11 +0200 (CEST) Received: (qmail 2398 invoked by uid 500); 8 Sep 2016 22:31:10 -0000 Mailing-List: contact user-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@beam.incubator.apache.org Delivered-To: mailing list user@beam.incubator.apache.org Received: (qmail 2388 invoked by uid 99); 8 Sep 2016 22:31:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Sep 2016 22:31:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 59CD9C05BE for ; Thu, 8 Sep 2016 22:31:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.228 X-Spam-Level: X-Spam-Status: No, score=-0.228 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, RP_MATCHES_RCVD=-1.426, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id TnZJyLMn90Sk for ; Thu, 8 Sep 2016 22:31:08 +0000 (UTC) Received: from nm4-vm4.bullet.mail.gq1.yahoo.com (nm4-vm4.bullet.mail.gq1.yahoo.com [98.136.218.163]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 3FCCE5FC21 for ; Thu, 8 Sep 2016 22:31:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1473373861; bh=DYDp83AEIcSH3G+d/19yRrUaRzbeLGnKcv+X9qQK36s=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=Rtd8UGHBz5a7FxBQYyH0EJ4IQV4fzK3J4PaO+7T8JIWGm6Rgeg6e8z06TXt97LUa/fcO60D43rHuo7vDztpaf0LcJ61V6YsJ2nycrk7dvl5PB1KAZoBJE8hnxj8KEQ5nkQvnjOEOiR/5i0BqVTOix/wYjk7y7uP6ylOWbwCPYLY6vM+WVf++uLSrq4jAakRHp16wXWiI1bsYYCFBAmA04BwUJ056iL1ny2j3W4RaYsIcCAfiPoJ39bsFKRgZSjdT31ppiZcib7QJph/wiF6AaRv6pQhYQVl9nH4UM99xqUgLvlWdFRvIlSAbXiaWbC2iCnaOl8QNrDHgYxn+hj7Ymw== Received: from [216.39.60.184] by nm4.bullet.mail.gq1.yahoo.com with NNFMP; 08 Sep 2016 22:31:01 -0000 Received: from [98.137.12.219] by tm20.bullet.mail.gq1.yahoo.com with NNFMP; 08 Sep 2016 22:31:01 -0000 Received: from [127.0.0.1] by omp1027.mail.gq1.yahoo.com with NNFMP; 08 Sep 2016 22:31:01 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 104824.42271.bm@omp1027.mail.gq1.yahoo.com X-YMail-OSG: GjcIoFQVM1nj7XlHEWh6Vv7_L.78artPatvGn_jUKqOY2p5SLqT0mEvZDTwbMMT UjxpfmiUVN98.9agJL.AzyJX9V.H5CPJzOcVhkKMBu9uAFuXbm0gvrLKsCzHgvjvG9RKKZhrvG.2 zSpux8N_XMzb7QYk.DoDd8TDSYT3HiJ8OL8F.AZygfee76UMX62QgLXAQrZ2dCiXj9w9Xjub0e6E jAsAhiiRuWinNP7fmHx2pCKZv4mDYIsAI71lgbO2bbqgOUFkUyBz07etEv9SmcLiFEORiy9Pr9Va 2WDcqGTpPPZ8L7flbM98JWmc0OyZEFxdBU3pK_AbZD04HeOYE630qLWIgckQ8usa9yko2a0bktna yvstHHJU3wbzgQaZGpbX4dqkG8wmkFTp.rmrwWBHKx4dGnkWp4BcbHfIUS0lnCQy8GhId88ag.r4 jz1ZgpjBIWK23HsB3aJjPaZjpiSPh_SR1bwi2XQ7G.VTartC.oinqrrH4szFrkdEm4rPqCzTbAXq a.o.nHh0J_lHZnrVmgPyoxEI- Received: from jws10762.mail.gq1.yahoo.com by sendmailws104.mail.gq1.yahoo.com; Thu, 08 Sep 2016 22:31:00 +0000; 1473373860.738 Date: Thu, 8 Sep 2016 22:31:00 +0000 (UTC) From: amir bahmanyari Reply-To: amir bahmanyari To: "user@beam.incubator.apache.org" Message-ID: <1084368144.1470303.1473373860189@mail.yahoo.com> In-Reply-To: <1840887603.1425560.1473371785626@mail.yahoo.com> References: <1840887603.1425560.1473371785626.ref@mail.yahoo.com> <1840887603.1425560.1473371785626@mail.yahoo.com> Subject: Re: KafkaIO() "no records have been read yet." Warning MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1470302_1881804636.1473373860183" archived-at: Thu, 08 Sep 2016 22:31:12 -0000 ------=_Part_1470302_1881804636.1473373860183 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This is where its happening in KafkaIO() class in beam.sdk source code . | @Override | | | public Instant getWatermark() { | | | if (curRecord =3D=3D null) { | | | LOG.debug("{}: getWatermark() : no records have been read yet.", name= ); | | | return initialWatermark; | | | } | | | | | | return source.watermarkFn.isPresent() | | | ? source.watermarkFn.get().apply(curRecord) : curTimestamp; | | | } | | | | Its checking curRecord =3D=3D null...Does this mean Kafka is shooting blank= ? A neglected hiccup checking in KafkaIO & Kafka sender?Can someone respond= pls?=C2=A0I am afraid I am losing tuples & thats critical to my whole work= I am doing on bench-marking Beam in a Flink Cluster.. Thanks+regardsAmir- From: amir bahmanyari To: "user@beam.incubator.apache.org" =20 Sent: Thursday, September 8, 2016 2:56 PM Subject: KafkaIO() "no records have been read yet." Warning =20 Hi Colleagues,I am running a Beam app on a 4-nodes Flink Cluster while rece= iving data from a single Kafka server.In all nodes flink-abahman-taskmanage= r-0-beam2.log file, I see continuesly see this Warning message as data is p= rocessed:2016-09-08 21:52:26,523 WARN =C2=A0org.apache.beam.sdk.io.kafka.Ka= fkaIO =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0- Reader-19: getWatermark() : no records have been rea= d yet. What is this? Does it mean maybe losing tuples?=C2=A0How can I enhance the = KafkaIO() call to not seeing this Warning?Thanks+regardsAmir- =20 ------=_Part_1470302_1881804636.1473373860183 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This is where its happ= ening in KafkaIO() class in beam.sdk source code .
@Override
publi= c Instant = getWatermark() {<= /td>
if (curRecord =3D=3D null) {
LOG.debug("{}: getWatermark() : no = records have been read yet.", name);
retur= n initialWatermark;
}
return source.watermarkFn.isPresent()
= ? source.watermarkFn.get().apply(curRecord) : curTimestamp;
}
Its checking curRecord =3D=3D null...Does this mean Kaf= ka is shooting blank? A neglected hiccup checking in KafkaIO & Kafka se= nder?
Can someone respond pls? 
I am a= fraid I am losing tuples & thats critical to my whole work I am doing o= n bench-marking Beam in a Flink Cluster..
Thanks+regards
Amir-
From: amir bahmanyari <amirtousa@yahoo.com>
= To: "user@beam.incubator.= apache.org" <user@beam.incubator.apache.org>
Sent: Thursday, September 8, 2016 2:56 PM
= Subject: KafkaIO() "no re= cords have been read yet." Warning

Hi Colleagues,
I am running a Beam app on a 4-nodes Flink Clust= er while receiving data from a single Kafka server.
In all nodes flink= -abahman-taskmanager-0-beam2.log file, I see continuesly see this Warning m= essage as data is processed:
2016-09-08 21:52:26,523 WARN  org.ap= ache.beam.sdk.io.kafka.KafkaIO             &n= bsp;            - Reader-19: getWatermark() := no records have been read yet.

What is this? Does it mean maybe losing tuple= s? 
How can I enhance the KafkaIO() call to not seeing this Warni= ng?
Thanks+regards
Amir-


------=_Part_1470302_1881804636.1473373860183--