flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Partitioning operator state
Date Thu, 08 Dec 2016 09:04:26 GMT
Hi Dominik,

as Gordon’s response only covers keyed-state, I will briefly explain what happens for non-keyed
operator state. In contrast to Flink 1.1, Flink 1.2 checkpointing does not write a single
blackbox object (e.g. ONE object that is a set of all kafka offsets is emitted), but a list
of blackbox objects instead (e.g. think of all kafka offsets being emitted individually, as
MULTIPLE objects). While Flink 1.2 still has no knowledge about the emitted objects in the
list (thus they remain blackboxes), what the contract allows is that those objects can be
freely redistributed in case of scale-out or scale-in. Scaling is merely splitting or merging
of the checkpointed lists.

Best,
Stefan 

> Am 08.12.2016 um 08:00 schrieb Tzu-Li (Gordon) Tai <tzulitai@apache.org>:
> 
> Hi Dominik,
> 
> Do you mean how Flink redistributes an operator’s state when the parallelism of the
operator is changed?
> If so, you can take a look at [1] and [2].
> 
> Cheers,
> Gordon
> 
> [1] https://issues.apache.org/jira/browse/FLINK-3755 <https://issues.apache.org/jira/browse/FLINK-3755>
> [2] https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit#
<https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit#>
> 
> On December 8, 2016 at 4:40:18 AM, Dominik Safaric (dominiksafaric@gmail.com <mailto:dominiksafaric@gmail.com>)
wrote:
> 
>> Hi everyone, 
>> 
>> In the case of scaling out a Flink cluster, how does Flink handle operator state
partitioning of a staged topology?  
>> 
>> Regards, 
>> Dominik 


Mime
View raw message