beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Knowles <k...@apache.org>
Subject Re: StateNamespaces.global() != StateNamespaces.window(GlobalWindowCoder, GlobalWindow)
Date Tue, 20 Aug 2019 04:40:52 GMT
Good point. "Non-window" or "cross-window" might be better. Or "per key" or
something.

Kenn

On Sat, Aug 17, 2019 at 2:25 AM Jan Lukavský <je.ik@seznam.cz> wrote:

> Hi Kenn,
>
> cool, thanks for explanation. The name is maybe a little misleading then,
> because the model uses word "global" for global-windowed scope, but that is
> absolutely minor issue.
>
> Jan
> On 8/16/19 10:57 PM, Kenneth Knowles wrote:
>
> Actually, this is not a bug. The "global" namespace is for storage that is
> per-key but not related to any window. So it is explicitly a separate
> domain from the global window. It is a concept just for runner
> implementors, not really part of the model and definitely not user facing.
>
> Kenn
>
> On Fri, Aug 16, 2019 at 4:22 AM Jan Lukavský <je.ik@seznam.cz> wrote:
>
>> Hi Max,
>>
>> I can do that, but the correct solution seems not to be that easy. I
>> think that changing either variant is a breaking change - the new
>> version will not correctly load from previous version of state
>> checkpoint. That is a serious problem, because pipelines would not be
>> easily upgradable. It looks a little "better" to change
>> StateNamespaces.global(), as it is used less often, but it is not a good
>> solution either. A correct solution would be somewhat complex I'm afraid
>> - i.e. in namespace '/' try to fetch '/' first, and if it is not
>> present, fetch '//'. Use '/' for storing.
>>
>> Any other ideas?
>>
>>   Jan
>>
>> On 8/16/19 12:14 PM, Maximilian Michels wrote:
>> > Hi Jan,
>> >
>> > I don't think this is intentional. It looks like an inconsistency which
>> > could result in unexpected behavior in UDFs, e.g. when storing state and
>> > timers.
>> >
>> > Want to open a PR?
>> >
>> > -Max
>> >
>> > On 12.08.19 14:22, Jan Lukavský wrote:
>> >> Hi,
>> >>
>> >> I noticed, that StateNamespaces.global() generates a different
>> stringKey
>> >> than StateNamespaces.window(GlobalWindowCoder, GlobalWindow). In the
>> >> first case, the stringKey will be simply '/',  in the other it will be
>> >> '//'. That has some other implications, like that
>> >> StateNamespaces.fromString(StateNamespaces.window(GlobalWindowCoder,
>> >> GlobalWindow).stringKey()) != StateNamespaces.global().
>> >>
>> >> It looks like a bug, but maybe I'm wrong - is this intentional?
>> >>
>> >> Jan
>> >>
>>
>

Mime
View raw message