kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: [DISCUSS] URIs on Producer and Consumer
Date Thu, 05 Oct 2017 22:22:49 GMT
Another useful feature for this...

Say that I am writing an application... if I write this URI on any of
my internal properties.. I can tweak my Consumer or Producer without
changing any code.

Say, you store the URI on your project's configuration xml..read it
and start your consumer... later on you could just update the XML, and
your user wouldn't need to rewrite his application.


right now if you (as an user) want to do such thing.. you would have
to write the parser yourself.

On Thu, Oct 5, 2017 at 4:33 PM, Michael Pearce <Michael.Pearce@ig.com> wrote:
> To me, this is a lot more in line with many other systems connections, to have the ability
to have a single connection string / uri, is this really that left field suggesting or wanting
this?
>
> If anything this bring kafka more standardised approach imo, to have a unified resource
identifier, protocol name and a set schema for that.
>
> e.g.
> Database connection strings like
>
> oracle:
> jdbc:oracle:thin:@(description=(address_list=
>    (address=(protocol=tcp)(port=1521)(host=prodHost)))
> (connect_data=(INSTANCE_NAME=ORCL)))
>
> or
>
> postgres:
> "jdbc:postgresql://localhost/test?user=fred&password=secret&ssl=true"
>
> And then like wise on messaging front, systems like
>
> rabbitmq
> “amqp://myhost?heartbeat=10&connection_timeout=10000”
>
>
> I personally like the suggestion +1.
>
>
> Cheers
> Mike
>
>
> On 05/10/2017, 20:10, "Clebert Suconic" <clebert.suconic@gmail.com> wrote:
>
>     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cmccabe@apache.org> wrote:
>     > We used URIs as file paths in Hadoop.  I think it was a mistake, for a
>     > few different reasons.
>     >
>     > URIs are actually very complex.  You probably know about scheme, host,
>     > and port, but did you know about authority, user-info, query, fragment,
>     > scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
>     > isn't obvious (and it wouldn't be obvious in Kafka either).
>
>     URIs are just a hashmap of key=string.. just like Properties...
>
>     The Consumer and Producer is just having such hashMap.. and these
>     values are easy to translate to boolean, integer.. etc. We would just
>     need to add such mapping as part of this task when done. I don't see
>     anything difficult there.
>
>
>     >
>     > When you flip back and forth between URIs and strings (and you
>     > inevitably will do this, when serializing or sending things over the
>     > wire), you run into tons of really hard problems.  Should you preserve
>     > the "fragment" (the thing after the hash mark) for your URI, or not?  It
>     > may not do anything now, but maybe it will do something later.  URIs
>     > also have complex string escaping rules.  Parsing URIs is very messy,
>     > especially when you start talking about non-Java programming languages.
>
>
>     Why flip back and forth? URIs would generate the same HashMap that's
>     being generated today.. I don't see any mess here.
>     Besides... This would be an addition, not replacement...
>
>     And I'm talking only about the Java API now.
>
>     Again, All the properties on ProducerConfig and ConsumerConfig seems
>     easy to be mapped as primitive types (String, numbers.. booleans).
>
>     Serialization shouldn't be a problem there. it would generate the same
>     properties it's generated now.
>
>     >
>     > URIs are designed for a world where you talk to a single host over a
>     > single port.  That isn't the world distributed systems live in.  You
>     > don't want your clients to fail to bootstrap because the single server
>     > you specified is having a bad day, even when the other 8 servers are up.
>
>     I have seen a few projects using this style of URI: I would make it
>     doing the same here:
>
>     If you have multiple hosts:
>
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property1=value");
>
>     if you have a single host:
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka://host2:port?property1=value&property2=value2");
>
>
>     One example of an apache project using a similar approach is qpid-jms:
>     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failover-configuration-options
>
>
>     > The bottom line is that URIs are the wrong abstraction for the job.
>     > They just don't express what we really want, and they introduce a lot of
>     > complexity and ambiguity.
>
>     I have seen the opposite to be honest. this has been simpler for me
>     and users I know than using a HashMap.. .  users in my experience tend
>     to write this faster.
>
>     users can certainly put up with the HashMap.. but this is easier to
>     remember. I'm just proposing what I think it's a simpler API.
>
>
>
>
>     Perhaps we should move into the KIP discussion itself here.. I first
>     intended to start this thread to see if it would make sense or not...
>     But I don't have authorization to create the KIP page.. so again..
>     based on the contributing page.. can someone add me authorizations to
>     the WIKI space?
>
>
> The information contained in this email is strictly confidential and for the use of the
addressee only, unless otherwise indicated. If you are not the intended recipient, please
do not read, copy, use or disclose to others this message or any attachment. Please also notify
the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the
email and any copies of it. Opinions, conclusion (etc) that do not relate to the official
business of this company shall be understood as neither given nor endorsed by it. IG is a
trading name of IG Markets Limited (a company registered in England and Wales, company number
04008957) and IG Index Limited (a company registered in England and Wales, company number
01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both
IG Markets Limited (register number 195355) and IG Index Limited (register number 114059)
are authorised and regulated by the Financial Conduct Authority.



-- 
Clebert Suconic

Mime
View raw message