kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization
Date Tue, 08 May 2018 22:08:19 GMT
Bill:That makes sense.
Using boolean should suffice.
-------- Original message --------From: Bill Bejeck <bbejeck@gmail.com> Date: 5/8/18
 2:48 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-295: Add Streams
Configuration Allowing for Optional Topology Optimization 
Thanks for the comments Guozhang and Ted.

Guozhang:

  1) I'll update the KIP in the "Compatibility, Deprecation and Migration
Plan" with the expected impact of turning on optimization. But at this
point, I have not identified a migration plan that doesn't involve having
to stop all instances and restart.

  2) Setting the type to String was just so we could have the default of
null, indicating run no optimizations. As for partially enabling
optimizations, I'm not sure I had that in mind, at least at this point.
 To me having the topology optimized should be an "all or nothing"
proposition.  For now, I'll change the type to boolean (with a default
value of false) to better reflect the intent of the configuration.

Ted, thanks again for the comments.

The intent of the new configuration, as I mentioned above, is whether to
turn optimization on or off in aggregate.  The main reason for having the
configuration is for backward compatibility.  As optimization may result in
a slightly different topology from the original one, we need to allow users
to turn it off until they are ready for migrating to the new topology.

I don't think having to select each optimization is a feasible solution.  I
say this for few reasons:

   1. Maintenance will be an issue.  While the initial target is only a
   small number of optimizations, but as the number grows, having to maintain
   the number of options in the code will difficult at best.
   2. Users probably don't want to reason about which combination of
   optimizations to have.  Thinking about which optimizations to turn on or
   off raises the complexity of deploying an application.
   3. When using a relational database or other another framework that may
   optimize queries, as far as I know, it's not a choice of which
   optimizations to have, they are performed automatically.

Does this make sense?


On Mon, May 7, 2018 at 7:49 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> There are 4 subtasks for KAFKA-6034
>
> If each optimization can be switched on/off, there should be 4 enums for
> the switch.
>
> On Mon, May 7, 2018 at 4:39 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > Hello Bill,
> >
> > Thanks for the KIP. My comments are the following:
> >
> > 1) We should state clearly what are the expected impact in
> "Compatibility,
> > Deprecation, and Migration Plan" when optimization is turned on.
> >
> > 2) I'm wondering why we define this config as a String typed, rather than
> > boolean typed? Or are you expecting to extend it to allow partially
> > enabling part of the optimizations in the future?
> >
> >
> > Hi Ted,
> >
> > The cover JIRA for topology optimization can be found here:
> > https://issues.apache.org/jira/browse/KAFKA-6034
> >
> >
> > Guozhang
> >
> >
> > On Mon, May 7, 2018 at 11:04 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Which JIRA is for the Topology Optimization itself ?
> > >
> > > Thanks
> > >
> > > On Mon, May 7, 2018 at 10:26 AM, Bill Bejeck <bbejeck@gmail.com>
> wrote:
> > >
> > > > All,
> > > > I'd like to start a discussion about adding a configuration parameter
> > > > allowing for the forthcoming topology optimization to be optional via
> > > > configuration.
> > > >
> > > >
> > > > The KIP can be found here:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 295%3A+Add+Streams+
> > > > Configuration+Allowing+for+Optional+Topology+Optimization
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message