kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: [DISCUSS] URIs on Producer and Consumer
Date Thu, 05 Oct 2017 23:22:36 GMT
You can update your properties file in the same way with the current
clients, right? I don't understand how mapping properties into a single
string makes things easier.

Ismael

On Thu, Oct 5, 2017 at 11:22 PM, Clebert Suconic <clebert.suconic@gmail.com>
wrote:

> Another useful feature for this...
>
> Say that I am writing an application... if I write this URI on any of
> my internal properties.. I can tweak my Consumer or Producer without
> changing any code.
>
> Say, you store the URI on your project's configuration xml..read it
> and start your consumer... later on you could just update the XML, and
> your user wouldn't need to rewrite his application.
>
>
> right now if you (as an user) want to do such thing.. you would have
> to write the parser yourself.
>
> On Thu, Oct 5, 2017 at 4:33 PM, Michael Pearce <Michael.Pearce@ig.com>
> wrote:
> > To me, this is a lot more in line with many other systems connections,
> to have the ability to have a single connection string / uri, is this
> really that left field suggesting or wanting this?
> >
> > If anything this bring kafka more standardised approach imo, to have a
> unified resource identifier, protocol name and a set schema for that.
> >
> > e.g.
> > Database connection strings like
> >
> > oracle:
> > jdbc:oracle:thin:@(description=(address_list=
> >    (address=(protocol=tcp)(port=1521)(host=prodHost)))
> > (connect_data=(INSTANCE_NAME=ORCL)))
> >
> > or
> >
> > postgres:
> > "jdbc:postgresql://localhost/test?user=fred&password=secret&ssl=true"
> >
> > And then like wise on messaging front, systems like
> >
> > rabbitmq
> > “amqp://myhost?heartbeat=10&connection_timeout=10000”
> >
> >
> > I personally like the suggestion +1.
> >
> >
> > Cheers
> > Mike
> >
> >
> > On 05/10/2017, 20:10, "Clebert Suconic" <clebert.suconic@gmail.com>
> wrote:
> >
> >     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cmccabe@apache.org>
> wrote:
> >     > We used URIs as file paths in Hadoop.  I think it was a mistake,
> for a
> >     > few different reasons.
> >     >
> >     > URIs are actually very complex.  You probably know about scheme,
> host,
> >     > and port, but did you know about authority, user-info, query,
> fragment,
> >     > scheme-specific-part?  Do you know what they do in Hadoop?  The
> mapping
> >     > isn't obvious (and it wouldn't be obvious in Kafka either).
> >
> >     URIs are just a hashmap of key=string.. just like Properties...
> >
> >     The Consumer and Producer is just having such hashMap.. and these
> >     values are easy to translate to boolean, integer.. etc. We would just
> >     need to add such mapping as part of this task when done. I don't see
> >     anything difficult there.
> >
> >
> >     >
> >     > When you flip back and forth between URIs and strings (and you
> >     > inevitably will do this, when serializing or sending things over
> the
> >     > wire), you run into tons of really hard problems.  Should you
> preserve
> >     > the "fragment" (the thing after the hash mark) for your URI, or
> not?  It
> >     > may not do anything now, but maybe it will do something later.
> URIs
> >     > also have complex string escaping rules.  Parsing URIs is very
> messy,
> >     > especially when you start talking about non-Java programming
> languages.
> >
> >
> >     Why flip back and forth? URIs would generate the same HashMap that's
> >     being generated today.. I don't see any mess here.
> >     Besides... This would be an addition, not replacement...
> >
> >     And I'm talking only about the Java API now.
> >
> >     Again, All the properties on ProducerConfig and ConsumerConfig seems
> >     easy to be mapped as primitive types (String, numbers.. booleans).
> >
> >     Serialization shouldn't be a problem there. it would generate the
> same
> >     properties it's generated now.
> >
> >     >
> >     > URIs are designed for a world where you talk to a single host over
> a
> >     > single port.  That isn't the world distributed systems live in.
> You
> >     > don't want your clients to fail to bootstrap because the single
> server
> >     > you specified is having a bad day, even when the other 8 servers
> are up.
> >
> >     I have seen a few projects using this style of URI: I would make it
> >     doing the same here:
> >
> >     If you have multiple hosts:
> >
> >     KafkaConsumer consumer = new
> >     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)
> ?property1=value");
> >
> >     if you have a single host:
> >     KafkaConsumer consumer = new
> >     KafkaConsumer("kafka://host2:port?property1=value&
> property2=value2");
> >
> >
> >     One example of an apache project using a similar approach is
> qpid-jms:
> >     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/
> index.html#failover-configuration-options
> >
> >
> >     > The bottom line is that URIs are the wrong abstraction for the job.
> >     > They just don't express what we really want, and they introduce a
> lot of
> >     > complexity and ambiguity.
> >
> >     I have seen the opposite to be honest. this has been simpler for me
> >     and users I know than using a HashMap.. .  users in my experience
> tend
> >     to write this faster.
> >
> >     users can certainly put up with the HashMap.. but this is easier to
> >     remember. I'm just proposing what I think it's a simpler API.
> >
> >
> >
> >
> >     Perhaps we should move into the KIP discussion itself here.. I first
> >     intended to start this thread to see if it would make sense or not...
> >     But I don't have authorization to create the KIP page.. so again..
> >     based on the contributing page.. can someone add me authorizations to
> >     the WIKI space?
> >
> >
> > The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>
>
> --
> Clebert Suconic
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message