flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 徐涛 <happydexu...@gmail.com>
Subject Re: Will idle state retention trigger retract in dynamic table?
Date Tue, 21 Aug 2018 10:03:10 GMT
Hi Fabian,
	Is the behavior a bit weird? Because it leads to data inconsistency.

Best,
Henry

> 在 2018年8月21日,下午5:14,Fabian Hueske <fhueske@gmail.com> 写道:
> 
> Hi,
> 
> In the given example, article_id 123 will always remain in the external storage. The
state is removed and hence it cannot be retracted anymore.
> Once the state was removed and the count reaches 10, a second record for article_id 123
will be emitted to the data store.
> 
> As soon as you enable state retention and state is needed that was removed, the query
result can become inconsistent.
> 
> Best, Fabian
> 
> 2018-08-21 10:52 GMT+02:00 徐涛 <happydexutao@gmail.com <mailto:happydexutao@gmail.com>>:
> Hi Fabian,
> 	SELECT article_id FROM praise GROUP BY article_id having count(1)>=10
> 	If article_id 123 has 100 praises and remains its state in the dynamic table ,and when
the time passed, its state is removed, but later the article_id 123 has never reached to 10
praises.
> 	How can other program know that the state is been removed? Because the sink currently
has the praises count stored as 100, it is not consistent as the dynamic table.
> 
> Best, 
> Henry
> 
> 
>> 在 2018年8月21日,下午4:16,Fabian Hueske <fhueske@gmail.com <mailto:fhueske@gmail.com>>
写道:
>> 
>> Hi,
>> 
>> No, it won't. I will simply remove state that has not been accessed for the configured
time but not change the result.
>> For example, if you have a GROUP BY aggregation and the state for a grouping key
is removed, the operator will start a new aggregation if a record with the removed grouping
key arrives.
>> 
>> Idle state retention is not meant to affect the semantics of a query. 
>> The semantics of updating the result should be defined in the query, e.g., with a
WHERE clause that removes all records that are older than 1 day (note, this is not supported
yet).
>> 
>> Best, Fabian
>> 
>> 2018-08-21 10:04 GMT+02:00 徐涛 <happydexutao@gmail.com <mailto:happydexutao@gmail.com>>:
>> Hi All,
>>         Will idle state retention trigger retract in dynamic table?
>> 
>> Best,
>> Henry
>> 
> 
> 


Mime
View raw message