flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishnu Viswanath <vishnu.viswanat...@gmail.com>
Subject Re: [DISCUSS] Allowed Lateness in Flink
Date Wed, 06 Jul 2016 15:17:11 GMT

I was going through the suggested improvements in window, and I have
few questions/suggestion on improvement regarding the Evictor.

1) I am having a use case where I have to create a custom Evictor that will
evict elements from the window based on the value (e.g., if I have elements
are of case class Item(id: Int, type:String) then evict elements that has
type="a"). I believe this is not currently possible.
2) this is somewhat related to 1) where there should be an option to evict
elements from anywhere in the window. not only from the beginning of the
window. (e.g., apply the delta function to all elements and remove all
those don't pass. I checked the code and evict method just returns the
number of elements to be removed and processTriggerResult just skips those
many elements from the beginning.
3) Add an option to enables the user to decide if the eviction should
happen before the apply function or after the apply function. Currently it
is before the apply function, but I have a use case where I need to first
apply the function and evict afterward.

I am doing these for a POC so I think I can modify the flink code base to
make these changes and build, but I would appreciate any suggestion on
whether these are viable changes or will there any performance issue if
these are done. Also any pointer on where to start(e.g, do I create a new
class similar to EvictingWindowOperator that extends WindowOperator?)

Thanks and Regards,
Vishnu Viswanath,

On Wed, Jul 6, 2016 at 9:39 AM, Aljoscha Krettek <aljoscha@apache.org>

> I did:
> https://mail-archives.apache.org/mod_mbox/flink-dev/201606.mbox/%3cCANMXwW0AbTTjjg9EWdxRUGxkjM7jscBeNmVRZOHPt2qO3pQMwA@mail.gmail.com%3e
>  ;-)
> On Wed, 6 Jul 2016 at 15:31 Ufuk Celebi <uce@apache.org> wrote:
> > On Wed, Jul 6, 2016 at 3:19 PM, Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> > > In the future, it might be good to to discussions directly on the ML
> and
> > > then change the document accordingly. This way everyone can follow the
> > > discussion on the ML. I also feel that Google Doc comments often don't
> > give
> > > enough space for expressing more complex opinions.
> >
> > I agree! Would you mind raising this point as a separate discussion on
> dev@
> > ?
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message