flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From LakeShen <shenleifight...@gmail.com>
Subject Re: join state TTL
Date Wed, 29 Apr 2020 02:40:47 GMT
Thank you for the clarification. Jark

Jark Wu <imjark@gmail.com> 于2020年4月29日周三 上午10:39写道:

> If 'uu' in stream A is not updated for more than 24 hours, then it will be
> cleared.  (blink planner)
> The state expiration strategy is "not updated for more than x time".
>
> Best,
> Jark
>
> On Wed, 29 Apr 2020 at 10:19, LakeShen <shenleifighting@gmail.com> wrote:
>
>> Hi Jark,
>>
>> I am a little  confused about how double stream joining state cleared(not
>> window join).
>>
>> For example, there are two stream , A , B . The sql like this :
>>
>> select a ,b from A  join B on  A.a = B.b
>>
>>
>> If I config the idle state retention time, such min idle state retention
>> time is 24 hour, max is 25 hour.
>>
>> There exist a key 'uu'  which not joined in B for 29 hour , the key 'uu'
>> state in A stream , is it cleared by flink ?
>>
>> Thanks to your reply.
>>
>> Best,
>> LakeShen
>>
>> Jark Wu <imjark@gmail.com> 于2020年4月28日周二 下午7:47写道:
>>
>>> Hi Lec,
>>>
>>> StateTtlConfig in DataStream API is a configuration on specific state,
>>> not a job level configuration.
>>> TableConfig#setIdleStateRetentionTime in TableAPI&SQL is a job level
>>> configuration which will enable state ttl for all non-time-based operator
>>> states.
>>> In blink planner, the underlying of
>>> TableConfig#setIdleStateRetentionTime uses the StateTtlConfig.
>>> Time-based operators are window aggregation, time-windowed join, and so
>>> on.
>>>
>>> StateTtlConfig is a state TTL mechanism only works on processing time,
>>> not event-time.
>>> In TableAPI&SQL and DataStream, the window aggregation and time-windowed
>>> join will clear expired state using Timers which is triggered by watermark.
>>> So time-based operators don't use StateTtlConfig to clear expired state.
>>>
>>> Best,
>>> Jark
>>>
>>>
>>> On Tue, 28 Apr 2020 at 14:48, lec ssmi <shicheng31604@gmail.com> wrote:
>>>
>>>> Hi:
>>>>   When the stream is joined with another stream , the cached stream
>>>> data will be saved as a state and deleted as the watermark advances.
>>>>   I found that there is also a parameter that can set the state
>>>> expiration time, such as StateTtlConfig in DataStream API, TableConfig
in
>>>> TableAPI &SQL .This setting is effective for the state of group by
>>>> operator. And now the state TTL seems to be based on processing time.If the
>>>> configured TTL  has been reached and the watermark has not moved to the
>>>> edge. The state of join will be cleared ? What is the relationship between
>>>> StateTtlConfig  and TableConfig? If I use  StateTtlConfig  and program with
>>>> TableAPI, can the configuration take effect?
>>>>
>>>>   Best regards
>>>>   Lec Ssmi
>>>>
>>>>
>>>

Mime
View raw message