Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <CAHvM9d0oPD8EiG3dJ18dNS2iNZixdBsXs9nsY8vnQwHyPR+SNA@mail.gmail.com>
References: <CAHvM9d0oPD8EiG3dJ18dNS2iNZixdBsXs9nsY8vnQwHyPR+SNA@mail.gmail.com>
From: AJAY GUPTA <ajaygit158@gmail.com>
Date: Fri, 10 Mar 2017 17:51:20 +0530
Message-ID: <CA+mX7fqDyCdCm1qeX2TEEL5DeCTas4FMKstZ63fh4Wj_58MsQw@mail.gmail.com>
Subject: Re: Watermark tuples in Apex
To: dev@apex.apache.org
Content-Type: multipart/alternative; boundary=94eb2c091dc233c514054a5f662a
archived-at: Fri, 10 Mar 2017 12:21:27 -0000

--94eb2c091dc233c514054a5f662a
Content-Type: text/plain; charset=UTF-8

Hi Bhupesh,

For point 1, cant we make use of implicitWatermarkGenerator?


Ajay

On Wed, Mar 8, 2017 at 12:16 PM, Bhupesh Chawda <bhupesh@apache.org> wrote:

> Hi All,
>
> Watermark tuples in Apex are very tightly coupled to event time processing.
> For this reason, usually they are modeled as having a timestamp.
>
> public interface WatermarkTuple
> {
>   long getTimestamp();
> }
>
> Even though, watermarks are meant for such time related processing, I think
> we should expand the concept of watermarks for the following types:
>
> 1. Labelled watermarks
> This could be useful in scenarios where instead of a timestamp (which is an
> ordered field), we have categorical values. For example, consider tuples
> which are labeled by city names. For each city, we want to have separate
> windows and isolate the processing. If the watermark returns a different
> city name, we end the previous window and start a new one. Or, in this case
> we could make use of both high and low watermarks indicating the start and
> end of a city's data. This could mean having multiple windows' data
> incoming at the same time.
>
> 2. Ordered watermarks
> Instead of having the ordered field as time, why not consider something
> like an Ordered Watermark. TimeBased Watermarks could extend from that.
> An ordered watermark could be used in case we have a sequence of data
> tuples and we need to demarcate every n tuples. Even though we can say that
> every n tuples the window is definitely closed, but the decision is made
> only when the upstream sends the watermark tuple. The windowed operator
> does not have any clue about it. It blindly opens and closes windows based
> on watermarks received from upstream. This could mean different windows may
> have different values of n.
>
> Please let me know your thoughts on this.
>
> ~ Bhupesh
>

--94eb2c091dc233c514054a5f662a--