kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API
Date Sat, 15 Sep 2018 06:14:34 GMT
On Wed, Aug 15, 2018, at 05:04, Viktor Somogyi wrote:
> Hi,
> 
> To weigh-in, I agree with Colin on the API naming, overloads shouldn't
> change behavior. I think all of the Java APIs I've used so far followed
> this principle and I think we shouldn't diverge.
> 
> Also I think I have an entry about this incremental thing in KIP-248. It
> died off a bit at voting (I guess 2.0 came quick) but I was about to revive
> and restructure it a bit. If you remember it would have done something
> similar. Back then we discussed an "incremental_update" flag would have
> been sufficient to keep backward compatibility with the protocol. Since
> here you designed a new protocol I think I'll remove this bit from my KIP
> and also align the other parts/namings to yours so we'll have a more
> unified interface on this front.
> 
> At last, one minor comment: is throttling a part of your protocol similarly
> to alterConfigs?

It should cover the same ground as alterConfigs, basically.

Does it make sense to start a VOTE thread on this?

best,
Colin

> 
> Viktor
> 
> 
> On Fri, Jul 20, 2018 at 8:05 PM Colin McCabe <cmccabe@apache.org> wrote:
> 
> > I updated the KIP.
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+IncrementalAlterConfigs+API
> >
> > Updates:
> > * Use "incrementalAlterConfigs" rather than "modifyConfigs," for
> > consistency with the other "alter" APIs.
> > * Implement Magnus' idea of supporting "append" and "subtract" on
> > configuration keys that contain lists.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Jul 16, 2018, at 14:12, Colin McCabe wrote:
> > > Hi Magnus,
> > >
> > > Thanks for taking a look.
> > >
> > > On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote:
> > > > Thanks for driving this KIP, Colin.
> > > >
> > > > I agree with Dong that a new similar modifyConfigs API (and protocol
> > API)
> > > > is confusing and that
> > > > we should try to extend the current alterConfigs interface to support
> > the
> > > > incremental mode instead,
> > > > deprecating the non-incremental mode in the process.
> > >
> > > In the longer term, I think that the non-incremental mode should
> > > definitely go away, and not be an option at all.  That's why I don't
> > > think of this KIP as "adding more options  to AlterConfigs" but as
> > > getting rid of a broken API.  I've described a lot of reasons why non-
> > > incremental mode is broken.  I've also described why the brokenness is
> > > subtle and an easy trap for newbies to fall into.  Hopefully everyone
> > > agrees that getting rid of non-incremental mode completely should be the
> > > eventual goal.
> > >
> > > I do not think that having a different name for modifyConfigs is
> > > confusing.  "Deleting all the configs and then setting some designated
> > > ones" is a very different operation from "modifying some
> > > configurations".  Giving the two operations different names expresses
> > > the fact  that they really are very different.  Would it be less
> > > confusing if the new function were called alterConfigsIncremental rather
> > > than modifyConfigs?
> > >
> > > I think it's important to have a new name for the new function.  If the
> > > names are the same, how can we even explain to users which API they
> > > should or should not use?  "Use the three argument overload, or the two
> > > argument overload where the Options class is not the final parameter"
> > > That is not user-friendly.
> > >
> > > You could say that some of the overloads would be deprecated.  However,
> > > my experience as a Hadoop developer is that most users simply don't care
> > > about deprecation warnings.  They will use autocomplete in their IDEs
> > > and use whatever function seems to have the parameters they need.
> > > Hadoop and Kafka themselves use plenty of deprecated APIs.  But somehow
> > > we expect that our users have much more respect for @deprecated than we
> > > ourselves do.
> > >
> > > I would further argue that function overloads in Java are intended to
> > > provide additional parameters, not to fundamentally change the semantics
> > > of a function.  If you have two functions int addTwoNumbers(int a, int
> > > b) and int addTwoNumbers(int a, int b, boolean verbose), they should
> > > both add together two numbers.  And a user should be able to expect that
> > > the plain old addTwoNumbers is equivalent to either
> > > addTwoNumbers(verbose=true) or addTwoNumbers(verbose=false), not a
> > > totally different operation.
> > >
> > > Every time programmers violate this contract, it inevitably leads to
> > > misunderstanding.  One example is how in HDFS there are multiple
> > > function overloads for renaming a file.  Depending on which one you
> > > call, you will get either RENAME or RENAME2, which have different
> > > semantics.  I think RENAME2 has a different set of return codes
> > > surrounding "destination exists" conditions, among other things.  Of
> > > course users have no idea of whether they're getting RENAME or RENAME2
> > > unless they're developers.  It's not obvious from the function call,
> > > which is named "rename" in both cases, just with different function
> > > parameters.  So the whole thing is just a huge source of confusion and
> > > user pain.
> > >
> > > Another thing to consider is that since  function overloads are also not
> > > an option in C or Go, we need a different solution for those languages
> > > anyway.  A separate function name works well for this.
> > >
> > > >
> > > > Another thing that I believe is missing is a per-config-entry operation
> > > > type, namely SET (ovewrite), APPEND or DELETE.
> > > > The current proposal only has SET (changes) and DELETE (removals)
> > > > semantics, but I understand there are configuration properties (such
> > as SASL auth) where
> > > > it should be possible to APPEND to a configuration property, otherwise
> > we'll have the same
> > > > non-atomic read-modify-write cycle problems as with the current API.
> > > > Instead of providing two sets of config: changes and removals, I think
> > > > it might be better to just have one set where each Config entry has
> > > > the operation type set, this also voids any confusion on what happens
> > > > if a property is included in both changes,removals sets.
> > >
> > > That's a very good point.  I guess the idea is that APPEND would add a
> > > new entry in the comma-separated (or other delimiter-separated) list of
> > > the config key?  That would require per-key support, since not all
> > > configuration keys have the same delimiter.  That's probably not too
> > > difficult, though.
> > >
> > > There are probably also lots of keys where APPEND makes no sense and
> > > should be rejected.  For example, APPENDing to a configuration
> > > controlling the number of active threads for a subsystem does not make
> > > sense.  Also, if we have APPEND, we probably also want SUBTRACT, right?
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > > 2018-07-16 20:23 GMT+02:00 Dong Lin <lindong28@gmail.com>:
> > > >
> > > > > Hey Colin,
> > > > >
> > > > > Thanks much for the explanation. Yeah it makes sense to deprecate
the
> > > > > existing non-incremental mode completely. LGTM. I just have two minor
> > > > > comments.
> > > > >
> > > > > I am not too strong with the following argument but it may be useful
> > to
> > > > > just put it here for discussion. I am still wondering whether we
can
> > just
> > > > > overload alterConfigs(...) instead of using modifyConfigs(...). In
> > the
> > > > > "Rejected Alternatives" section it is said that this approach does
> > not
> > > > > allow us to deprecate the non-incremental mode. Not sure why it is
> > the
> > > > > case. Can you explain a bit more?
> > > > >
> > > > > Regarding whether this approach is surprising and baffling to users,
> > > > > personally I feel that with the existing approach in the KIP, a new
> > name
> > > > > such as modifyConfigs(...) does make it very explicit that this API
> > is
> > > > > different from the existing alterConfigs(...). But it does not
> > really tell
> > > > > user how they are different and users will have to read the Java
doc
> > to
> > > > > figure this out. On the other hand, if we just overload the
> > > > > alterConfigs(...) and deprecate the existing non-incremental
> > > > > alterConfigs(...), it would also make it reasonable clear to user
> > that the
> > > > > two methods are different. Commonly-used IDE such as IntellIj would
> > show
> > > > > that one API is deprecated and the other is not. And user would then
> > read
> > > > > the Java doc to understand the difference. So the difference between
> > these
> > > > > two approaches in the short term is probably not much. And in the
> > long term
> > > > > it might be preferred to use "alter" instead of "modify".
> > > > >
> > > > > Another minor comment: should we include specify in the
> > "Compatibility,
> > > > > Deprecation, and Migration Plan" section that we intend to deprecate
> > the
> > > > > existing API? And do we plan to deprecate the
> > > > > AlterConfigsRequest/AlterConfigsResponse
> > > > > as well? The latter may be important to non-Kafka projects that have
> > > > > implemented AdminClient interface.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Dong
> > > > >
> > > > >
> > > > > On Mon, Jul 16, 2018 at 9:43 AM, Colin McCabe <cmccabe@apache.org>
> > wrote:
> > > > >
> > > > > > On Fri, Jul 13, 2018, at 23:20, Dong Lin wrote:
> > > > > > > Hey Colin,
> > > > > > >
> > > > > > > Thanks for the KIP!
> > > > > > >
> > > > > > > It seems that the AlterConfigsResult is pretty much the
same as
> > > > > > > ModifyConfigsResult. Instead of adding ModifyConfigs API
and
> > > > > deprecating
> > > > > > > AlterConfigs API, would it be simpler to just add API
> > alterConfigs(
> > > > > > > Map<ConfigResource, Config> changes, Set<ConfigResource>
> > removals,
> > > > > final
> > > > > > > ModifyConfigsOptions options)?
> > > > > > >
> > > > > > > Currently we use the word "alter" in method names such
as
> > > > > > > "alterReplicaLogDirs" and "alterCheckpointDir". So it is
> > probably more
> > > > > > > preferred to keep using the word "alter" instead of "modify"
if
> > > > > > posssible.
> > > > > > > And if we can overload the alterConfigs(...) API to allow
> > incremental
> > > > > > > change, it might make sense to keep the existing
> > > > > > > alterConfigs(Map<ConfigResource,
> > > > > > > Config> configs) for users who simply want to overwrite
the
> > entire
> > > > > > configs.
> > > > > >
> > > > > > If we have two functions with these type signatures:
> > > > > >
> > > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource,
Config>
> > changes);
> > > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource,
Config>
> > changes,
> > > > > > Set<ConfigResource> removals);
> > > > > >
> > > > > > It will be extremely surprising, even baffling, to users  that
the
> > second
> > > > > > function overload makes incremental changes, whereas the first
> > function
> > > > > > overload clears the entire configuration before applying changes.
> > Just
> > > > > > looking at the type signatures (which is all most developers
will
> > look
> > > > > at,
> > > > > > especially if they're using IDE autocomplete), you would not
> > expect such
> > > > > a
> > > > > > radical difference between them.  You would expect the second
one
> > to work
> > > > > > just like the first, except maybe it would also perform some
> > removals.
> > > > > >
> > > > > > Calling the two functions different names is good because it
> > reflects the
> > > > > > fact that they are very different.
> > > > > >
> > > > > > > And those user would not have to make code change due to
API
> > > > > deprecation.
> > > > > > > What do you think?
> > > > > >
> > > > > > alterConfigs needs to be deprecated, though.  Code using
> > alterConfigs is
> > > > > > almost certainly buggy because of the possibility of simultaneous
> > > > > > read-modify-write cycles, and the fact that some configs can't
be
> > read.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dong
> > > > > > >
> > > > > > > On Wed, Jul 11, 2018 at 10:54 AM, Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Previously, we discussed some issues with alterConfigs
here on
> > the
> > > > > > mailing
> > > > > > > > list, and eventually came to the conclusion that the
RPC as
> > > > > implemented
> > > > > > > > can't be used for a shell command modifying configurations.
> > > > > > > >
> > > > > > > > I wrote up a small KIP to fix the issues with the
RP  Please
> > take a
> > > > > > look
> > > > > > > > at https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > 339%3A+Create+a+new+ModifyConfigs+API
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > >
> > > > >
> >

Mime
View raw message