flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: Option to disable chaining?
Date Mon, 08 Feb 2016 20:39:21 GMT
Disabling operator chaining is also the only way to see metrics from each
step right now.

On Monday, February 8, 2016, Stephan Ewen <sewen@apache.org> wrote:

> There is no fundamental reason to not implement this for batch as well.
>
> In Streaming, users seem to want more control about threads and resources
> (given that these are often continuous pipelines), while in batch that was
> not requested so far.
>
> But I see that a non-chained function is more safe with respect to
> accidental object reuse by a user in their functions...
>
> On Mon, Feb 8, 2016 at 8:19 PM, Greg Hogan <code@greghogan.com
> <javascript:;>> wrote:
>
> > Is there a reason to not also implement this for batch processing? This
> > would allow object reuse to be truly disabled.
> >
> > On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <sewen@apache.org
> <javascript:;>> wrote:
> >
> > > It may be useful for example when you have two MapFunctions and each
> does
> > > something CPU intensive, or communicates with an external service.
> > >
> > > Without chaining, you will have two threads and an elastic channel
> > between
> > > the functions to buffer some records, which may help in such a case.
> > >
> > >
> > > On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <code@greghogan.com
> <javascript:;>> wrote:
> > >
> > > > When is this useful in streaming?
> > > >
> > > > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <ndimiduk@apache.org
> <javascript:;>>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > > > >
> > > > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <code@greghogan.com
> <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Is it possible to force operator chaining to be disabled? Similar
> > to
> > > > how
> > > > > > object reuse can be enabled or disabled?
> > > > > >
> > > > > > Greg
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message