flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [DISCUSS] Enhance Window Evictor in Flink
Date Wed, 06 Jul 2016 18:36:53 GMT
@Maxim: That's perfect I didn't think about using Iterator.remove() for
that. I'll update the doc. What do you think Vishnu? This should also cover
your before/after case nicely.

@Vishnu: The steps would be these:
 - Converge on a design in this discussion
 - Add a Jira issue here: https://issues.apache.org/jira/browse/FLINK
 - Work on the code an create a pull request on github

The steps are also outlined here
http://flink.apache.org/how-to-contribute.html and here
http://flink.apache.org/contribute-code.html.

-
Aljoscha

On Wed, 6 Jul 2016 at 19:45 Maxim <mfateev@gmail.com> wrote:

> The new API forces iteration through every element of the buffer even if a
> single value to be evicted. What about implementing Iterator.remove()
> method for elements? The API would look like:
>
> public interface Evictor<T, W extends Window> extends Serializable {
>
>    /**
>     *  Optionally evicts elements. Called before windowing function.
>     *
>     * @param elements The elements currently in the pane. Use
> Iterator.remove to evict.
>     * @param size The current number of elements in the pane.
>     * @param window The {@link Window}
>     */
>    void evictBefore(Iterable<T> elements, int size, EvictorContext ctx);
>
>    /**
>     *  Optionally evicts elements. Called after windowing function.
>     *
>     * @param elements The elements currently in the pane. Use
> Iterator.remove to evict.
>     * @param size The current number of elements in the pane.
>     * @param window The {@link Window}
>     */
>    void evictAfter(Iterable<T> elements, int size, EvictorContext ctx);
> }
>
> Such API allows to abort iteration at any point and evict elements in any
> order.
>
> Thanks,
>
> Maxim.
>
> On Wed, Jul 6, 2016 at 9:04 AM, Vishnu Viswanath <
> vishnu.viswanath25@gmail.com> wrote:
> >
> > Hi Aljoscha,
> >
> > Thanks. Yes the new interface seems to address points 1 and 2. of
> >
> > *1) I am having a use case where I have to create a custom Evictor that
> > will evict elements from the window based on the value (e.g., if I have
> > elements are of case class Item(id: Int, type:String) then evict elements
> > that has type="a"). I believe this is not currently possible.*
> > *2) this is somewhat related to 1) where there should be an option to
> evict
> > elements from anywhere in the window. not only from the beginning of the
> > window. (e.g., apply the delta function to all elements and remove all
> > those don't pass. I checked the code and evict method just returns the
> > number of elements to be removed and processTriggerResult just skips
> those
> > many elements from the beginning.  *
> > *3) Add an option to enables the user to decide if the eviction should
> > happen before the apply function or after the apply function. Currently
> it
> > is before the apply function, but I have a use case where I need to first
> > apply the function and evict afterward.*
> >
> > I would be interested in contributing to the code base. Please let me
> know
> > the steps.
> >
> > Thanks and Regards,
> > Vishnu Viswanath
> >
> > On Wed, Jul 6, 2016 at 11:49 AM, Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> >
> > > Hi,
> > > as mentioned in the thread on improving the Windowing API I also have a
> > > design doc just for improving WindowEvictors. I had this in my head for
> a
> > > while but was hesitant to publish but since people are asking about
> this
> > > now might be a good time to post it. Here's the doc:
> > >
>
> https://docs.google.com/document/d/1Rr7xzlItYqvFXLyyy-Yv0vvw8f29QYAjm5i9E4A_JlU/edit?usp=sharing
> > >
> > > Feedback/Suggestions are very welcome! Please let me know what you
> think.
> > >
> > > @Vishnu: Are you interested in contributing a solution for this to the
> > > Flink code base? I'd be very happy to work with you on this.
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > P.S. I think it would be best to keep discussions to the ML because
> > > comments on the doc will not be visible here for everyone.
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message