flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Blocking or pipelined mode for batch job
Date Tue, 10 May 2016 09:21:53 GMT
On Tue, May 10, 2016 at 10:56 AM, wangzhijiang999
<wangzhijiang999@aliyun.com> wrote:
>        As I reviewed the flink source code, if the ExecutionMode is set
> "Pipelined", the DataExechangeMode will be set "Batch" if breaking pipelined
> property is true for two input or iteration situation in order to avoid
> deadlock. When the DataExechangeMode is set "Batch", the ResultPartitionType
> will be set "Pipelined" if dynamic path for iteration . So the final mode
> for dynamic node in iteration is "Pipelined"? My understanding is right ? If
> it is , why must use pipelined mode for dynamic node in iteration?

Yes, this is a current shortcoming, because the blocking results can
not be used within iterations (they can not be reset after each
superstep).

>     Another question is that when running a batch job, if one task failed,
> do i must restart the whole topology or are there any machnism to just
> restart failed task?  Thank you for any advise !

Currently, yes, the whole topology is restarted. There was an attempt
to add support for more fine-grained fault tolerance, but it was never
completed/merged.

– Ufuk

Mime
View raw message