kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From BigData dev <bigdatadev...@gmail.com>
Subject Re: [DISCUSS] KIP-154 Add Kafka Connect configuration properties for creating internal topics
Date Tue, 09 May 2017 03:42:21 GMT
Thank You got it.


On Mon, May 8, 2017 at 8:34 PM, Randall Hauch <rhauch@gmail.com> wrote:

> Yes, that's the approach I'm suggesting and that is mentioned in the KIP. I
> also propose that the distributed configuration provided in the examples
> set the replication factor to one but include a relevant comment.
>
> On Mon, May 8, 2017 at 11:14 PM, BigData dev <bigdatadevguy@gmail.com>
> wrote:
>
> > So, when Kafka broker is less than 3, and the user has not set the
> > replication configuration it will throw an error to the user, to correct
> > the configuration according to his setup? Is this the approach you are
> > suggesting here?
> >
> >
> >
> > On Mon, May 8, 2017 at 7:13 PM, Randall Hauch <rhauch@gmail.com> wrote:
> >
> > > One of the "Rejected Alternatives" was to do something "smarter" by
> > > automatically reducing the replication factor when the cluster size is
> > > smaller than the replication factor. However, this is extremely
> > > unintuitive, and in rare cases (e.g., during a partial outage) might
> even
> > > result in internal topics being created with too small of a replication
> > > factor. And defaulting to 1 is certainly bad for production use cases,
> so
> > > that's not an option, either.
> > >
> > > While defaulting to 3 and failing if the cluster doesn't have 3 nodes
> is
> > a
> > > bit harsher than I'd like, it does appear to be the safer option: an
> > error
> > > message (with instructions on how to correct) is better than
> > inadvertently
> > > setting the replication factor too small and not knowing about it until
> > it
> > > is too late.
> > >
> > > On Mon, May 8, 2017 at 6:12 PM, BigData dev <bigdatadevguy@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I liked the KIP, as it will avoid so many errors which user can make
> > > during
> > > > setup.
> > > > I have 1 questions here.
> > > > 1. As default replication factor is set to 3, but if Kafka cluster is
> > > setup
> > > > for one node, then the user needs to override the default
> configuraion,
> > > > till then topics will not be created.
> > > > So, this is the behavior we want to give?
> > > >
> > > > On Mon, May 8, 2017 at 2:25 PM, Konstantine Karantasis <
> > > > konstantine@confluent.io> wrote:
> > > >
> > > > > Thanks a lot for the KIP Randall. This improvement should simplify
> > both
> > > > > regular deployments and testing!
> > > > >
> > > > > A minor comment. Maybe it would be nice to add a note about why
> > there's
> > > > no
> > > > > need for the property: config.storage.partitions
> > > > > I'm mentioning this for the sake of completeness, in case someone
> > > notices
> > > > > this slight asymmetry with respect to the newly introduced config
> > > > > properties.
> > > > >
> > > > > This is by no means a blocking comment.
> > > > >
> > > > > Thanks,
> > > > > Konstantine
> > > > >
> > > > > On Fri, May 5, 2017 at 7:18 PM, Randall Hauch <rhauch@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks, Gwen.
> > > > > >
> > > > > > Switching to low-priority is a great idea.
> > > > > >
> > > > > > The default value for the replication factor configuration is
3,
> > > since
> > > > > > that makes sense and is safe for production. Using the default
> > values
> > > > in
> > > > > > the example would mean it could only be run against a Kafka
> cluster
> > > > with
> > > > > a
> > > > > > minimum of 3 nodes. I propose overriding the example's
> replication
> > > > factor
> > > > > > configurations to be 1 so that the examples could be run on
any
> > sized
> > > > > > cluster.
> > > > > >
> > > > > > The rejected alternatives mentions why the implementation doesn't
> > try
> > > > to
> > > > > > be too smart by calculating the replication factor.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > > On May 5, 2017, at 8:02 PM, Gwen Shapira <gwen@confluent.io>
> > > wrote:
> > > > > > >
> > > > > > > Looks great to me :)
> > > > > > >
> > > > > > > Just one note - configurations have levels (which reflect
in
> the
> > > > docs)
> > > > > -
> > > > > > I
> > > > > > > suggest putting the whole thing as LOW. Most users will
never
> > need
> > > to
> > > > > > worry
> > > > > > > about these. For same reason I recommend leaving them out
of
> the
> > > > > example
> > > > > > > config files - we already have issues with users playing
with
> > > configs
> > > > > > > without understanding what they are doing and not liking
the
> > > results.
> > > > > > >
> > > > > > >> On Fri, May 5, 2017 at 3:42 PM, Randall Hauch <
> rhauch@gmail.com
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >> Hi, all.
> > > > > > >>
> > > > > > >> I've been working on KAFKA-4667 to change the distributed
> worker
> > > of
> > > > > > Kafka
> > > > > > >> Connect to look for the topics used to store connector
and
> task
> > > > > > >> configurations, offsets, and status, and if those tasks
do not
> > > exist
> > > > > to
> > > > > > >> create them using the new AdminClient. To make this
as useful
> as
> > > > > > possible
> > > > > > >> and to minimize the need to still manually create the
topics,
> I
> > > > > propose
> > > > > > >> adding several new distributed worker configurations
to
> specify
> > > the
> > > > > > >> partitions and replication factor for these topics,
and have
> > > > outlined
> > > > > > them
> > > > > > >> in "KIP-154 Add Kafka Connect configuration properties
for
> > > creating
> > > > > > >> internal topics".
> > > > > > >>
> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> 154+Add+Kafka+Connect+configuration+properties+for+
> > > > > > >> creating+internal+topics
> > > > > > >>
> > > > > > >> Please take a look and provide feedback. Thanks!
> > > > > > >>
> > > > > > >> Best regards,
> > > > > > >>
> > > > > > >> Randall
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Gwen Shapira*
> > > > > > > Product Manager | Confluent
> > > > > > > 650.450.2760 | @gwenshap
> > > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc>
| blog
> > > > > > > <http://www.confluent.io/blog>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message