apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@apache.org>
Subject Re: APEXMALHAR-2354 Heuristic Watermark in windowed operator
Date Mon, 28 Nov 2016 10:28:47 GMT
Thanks Guys.

If there are no further comments, I can go about writing code and creating
a PR.

-Chinmay.

On Thu, Nov 24, 2016 at 5:10 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> Good idea, +1
>
> ~ Bhupesh
>
> On Thu, Nov 24, 2016 at 5:08 PM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
> > +1, but can you change name of HeuristicWatermark to
> > WatermarkGenerator as the purpose of
> > this interface is to generate watermarks.
> >
> > - Tushar.
> >
> >
> > On Thu, Nov 24, 2016 at 4:22 PM, Chinmay Kolhatkar <chinmay@apache.org>
> > wrote:
> > > Dear Community,
> > >
> > > I'm working on adding support for heuristic watermark in Windowed
> > Operator.
> > > Heuristic watermark give users of WindowedOperator a way to logically
> > > determine whether watermark condition is met or not by inspecting the
> > > tuples received.
> > > This can act as a replacement for or way to work along with Control
> Tuple
> > > received on control port.
> > >
> > > Here is the approach I'm considering:
> > >
> > > 1. A new interface lets say "HeuristicWatermark" will be added which
> > > extends Component<Context.OperatorContext>
> > > The reason why its extended with Component is then it can follow a
> > > lifecycle.
> > >
> > > 2. This method contains a single method something like this:
> > >
> > > ControlTuple.Watermark processTupleForWatermark(
> > Tuple.WindowedTuple<InputT>
> > > input);
> > >
> > > 3. Object of this type can optionally be set to
> AbstractWindowedOperator
> > as
> > > a plugin which identified whether watermark condition has reached.
> > >
> > > 4. If heuristicWatermark is set, processTupleForWatermark will be
> called
> > > for every received tuple and the method can return the Watermark object
> > if
> > > watermark condition is met OR return null if not so.
> > >
> > > 5. If return value of this method is non-null, then processWatermark
> > method
> > > will be called which sets the nextWatermark value. And then rest of the
> > > watermark processing can continue to happen in endWindow.
> > >
> > >
> > > Please share your opinion on above approach.
> > >
> > > Thanks,
> > > Chinmay.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message