flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: ListState to List
Date Thu, 18 May 2017 13:44:28 GMT
I think the ListState interface is pretty well suited for this job.
It allows to add elements with low effort and can serve all elements of a
list through an iterator. Depending on the implementation the elements
could be deserialized as needed.
If the user code needs a List with all elements, it won't make a big
performance difference whether the state backend or the user code copies
the elements into a List object.

2017-05-18 15:24 GMT+02:00 Radu Tudoran <radu.tudoran@huawei.com>:

> Actually that was one option that I was considering. I am still a bit
> fuzzy about the advantages and disadvantages of using one type of state
> over another. I know that using ValueState would mean that when getting the
> object value (i.e. a List in this case) the whole would be deserialized at
> once. This is ok if I need to go anyway through all elements. However, I
> understand that when I need to update the state the same will hold - the
> whole list would be serialized and re-write instead of a single element.
> Therefore if we want to get always the best performances it seemed to me
> that it would be worth considering have many specialized types of states -
> hence my proposal.
>
>
>
> -----Original Message-----
> From: Kostas Kloudas [mailto:k.kloudas@data-artisans.com]
> Sent: Thursday, May 18, 2017 11:49 AM
> To: dev@flink.apache.org
> Subject: Re: ListState to List
>
> Hi Radu,
>
> Why not using a ValueState that inside stored the whole list.
> Whenever you state#get() you get the whole list and you can sort it.
>
> Kostas
>
> > On May 18, 2017, at 3:31 AM, Radu Tudoran <radu.tudoran@huawei.com>
> wrote:
> >
> > Hi Aljoscha,
> >
> > Thanks for the clarification. I understand that there might be
> advantages in some cases not to have the List-like interface, while in
> other scenarios (like the one I described there aren't). Considering this,
> why not having 2 type of states: ListState and StreamInListState - users
> would use the one it is more appropriate. What do you think?
> >
> > -----Original Message-----
> > From: Aljoscha Krettek [mailto:aljoscha@apache.org]
> > Sent: Thursday, May 18, 2017 12:15 AM
> > To: dev@flink.apache.org
> > Subject: Re: ListState to List
> >
> > Hi,
> > The interface is restrictive on purpose because depending on the state
> backend it might not be possible to provide a List-like interface. There
> might be state backends that stream in the list from somewhere else or
> other restrictions. If we now allowed a more general interface here we
> would possibly prevent optimisations in the future or make certain
> implementations very hard to to efficiently.
> >
> > Best,
> > Aljoscha
> >
> >> On 16. May 2017, at 21:56, Radu Tudoran <radu.tudoran@huawei.com>
> wrote:
> >>
> >> Hi,
> >>
> >> I would like to work with ListState, more specifically I would need to
> access the contents and sort them. For this I would need a collection type
> (e.g., the List, Array...).
> >> However, I see that if I have a variable of type <<ListState state=..>>
> the only interfaces I have are:
> >> state.get -> which returns an Iterable Or state.get.getIterator which
> >> returns an Iterator
> >>
> >> Basically if I use any of these I need now to copy the contents in an
> actual List of Array.  Is there any way to avoid this? ..perhaps there is
> an implicit type that I can convert to...
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message