From user-return-31377-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Dec 11 08:56:13 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 56D73180638 for ; Wed, 11 Dec 2019 09:56:13 +0100 (CET) Received: (qmail 27512 invoked by uid 500); 11 Dec 2019 08:56:11 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 27502 invoked by uid 99); 11 Dec 2019 08:56:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2019 08:56:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EA872C0584 for ; Wed, 11 Dec 2019 08:56:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.251 X-Spam-Level: X-Spam-Status: No, score=0.251 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id SmboMedH2tZq for ; Wed, 11 Dec 2019 08:56:08 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::141; helo=mail-il1-x141.google.com; envelope-from=qcx978132955@gmail.com; receiver= Received: from mail-il1-x141.google.com (mail-il1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 02F437DDB9 for ; Wed, 11 Dec 2019 08:56:07 +0000 (UTC) Received: by mail-il1-x141.google.com with SMTP id n1so4218296ilm.6 for ; Wed, 11 Dec 2019 00:56:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hWK3vgiLocbIU2pV8L6w3rKCpQnmRPoRGSsGxpFTCjg=; b=ixUgea2GA9uAQhPtk/1xsxRkTguqibp1/X5oqA12vMqflMCZX+z0YtrMP3oDTYdwJZ DiCuw8lgSi7lviBCLVB65wWuEPsLJp2f4LXDhHOvfWTpi78mZNtbnKQSSXUJiK83bhWm mzH4DcQeURN6iCAEnGJOE9xDL4bjVsrT14KoqIBb9rG2F+k0B+kIkGJo9s+yR8jjFR8s r2p+oHRlPztpRpADh72xCgkjqa5hL2C1tQaoarhuLa17u7khzLT1iPVtSh/0V2elgmUj ImzA+6tUcQaaS+7uLRtuRoNfMZ7L5MDR+H67rOJDMfrPnVBqiGafreTrpWlVuL8ZwnnB 9hEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hWK3vgiLocbIU2pV8L6w3rKCpQnmRPoRGSsGxpFTCjg=; b=g/FeZuQG/lDPOufHf1wsSzhl0RuYvL5K+05Lw4CRgafWOjj7N/k37Y+oYSP6PdWoZn 1yzT9vM/VdMzykO+prD78E56YYSPHm9y1Zv2KLSt2P5yLjj7STGbNbgklIOkr7sGqprY I+o72DoCce8Q08hvReT1n+loU2pbx8XdpatVlaDv0ovZfBXlRhechUUlerUajjnRsKsP ovA8KYKQkW/8ypGk+6t4fg+iG+ZFf2kTNBDO6wFIAkPofuZPNWO1rUE7ADufKkVeWEMk m20W0eIlhAbJf+5wWsCqWGxJNhQZbbsaV/K38jKeyb4gfnwBWb3EEy4abJTY1QSSMxGb Oy9w== X-Gm-Message-State: APjAAAUq1UgL8cGBr6XYbZg0FfMyDpRXAYk2ee0UOtGAwIgKm4hDLlal xVWULSuo4c76DOf7GJmwKJe3pCHVQTgUh7Qizwc= X-Google-Smtp-Source: APXvYqysHUq6n23wMP60hAt9ioKhbMIbFQyl50vjqc8uujJQlM51ZeFScOhEv0OH6i608lVsdsRumdvcgboUUSb6/lg= X-Received: by 2002:a92:8c96:: with SMTP id s22mr1832182ill.159.1576054560696; Wed, 11 Dec 2019 00:56:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Congxian Qiu Date: Wed, 11 Dec 2019 16:55:49 +0800 Message-ID: Subject: Re: Interval Join Late Record Metrics To: Chris Gillespie Cc: user Content-Type: multipart/alternative; boundary="0000000000003cb030059969cc3b" --0000000000003cb030059969cc3b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Chris From the code[1], currently, IntervalJoin will ignore the late data silently, maybe you can create an issue to track this. [1] https://github.com/apache/flink/blob/5c89d12849ea2aa332126b32808e363f12d436= a2/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operat= ors/co/IntervalJoinOperator.java#L222 Best, Congxian Chris Gillespie =E4=BA=8E2019=E5=B9=B412=E6=9C=881= 1=E6=97=A5=E5=91=A8=E4=B8=89 =E4=B8=8A=E5=8D=888:09=E5=86=99=E9=81=93=EF=BC= =9A > Hello Flink users, first time poster here. > > I'm using an interval join in my Flink project, however I haven't found > where late records get logged in metrics. Window Joins have > "numLateRecordsDropped" implemented > , > but is there an equivalent within an interval join? > > My main use case is to track how often a record falls outside of the lowe= r > and upper bounds when trying to join two streams. Interval Join looks lik= e > it simply short circuits > when > there is a late record? Maybe I am not understanding what defines a late > record in this situation. > > Is there a good way to monitor when an interval join fails to join two > streams? Currently I'm looking at the delta between two operator metrics, > but it hasn't looked that reliable so far. > > Thanks, > Chris Gillespie > --0000000000003cb030059969cc3b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Chris

From the code[1], currently, I= ntervalJoin will ignore the late data silently, maybe you can create an iss= ue to track this.


Chri= s Gillespie <cgillespie@zendes= k.com> =E4=BA=8E2019=E5=B9=B412=E6=9C=8811=E6=97=A5=E5=91=A8=E4=B8= =89 =E4=B8=8A=E5=8D=888:09=E5=86=99=E9=81=93=EF=BC=9A
Hello Flink use= rs, first time poster here.

I'm using an inter= val join in my Flink project, however I haven't found where late record= s get logged in metrics. Window Joins have "numLateRecordsDropped"= ; implemented, but is there an e= quivalent within an interval join?=C2=A0

My main u= se case is to track how often a record falls outside of the lower and upper= bounds when trying to join two streams. Interval Join looks like it simply short circuits when there is a late = record? Maybe I am not understanding what defines a late record in this sit= uation.

Is there a good way to monitor when a= n interval join fails to join two streams? Currently I'm looking at the= delta between two operator metrics, but it hasn't looked that reliable= so far.

Thanks,
Chris Gillespie
--0000000000003cb030059969cc3b--