Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <BACA0B26-7E71-4F01-823E-3074E1A265FD@ig.com>
References: <CAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_Oyvtqw@mail.gmail.com>
 <CAOeJiJgGtUikqYRc6H6eCsQ9QtmD7rqXQ7kTScT9K7=uzyS_WA@mail.gmail.com>
 <CAKF+bsoUU1a4bdofLOCRU4yB246HGf9KJZZrB=DD+zGVvA66Qg@mail.gmail.com>
 <CAKF+bsoxgnDgfsm_BRgk-=Pvhe4izNetKYfB_ccBhcqffB-q=g@mail.gmail.com>
 <1507227655.3564592.1129128368.1AB9F357@webmail.messagingengine.com>
 <CAKF+bsot4XaAdB34wdWXT=e+25g1tzwmxGcYa7qm8dQU3rzUdQ@mail.gmail.com> <BACA0B26-7E71-4F01-823E-3074E1A265FD@ig.com>
From: Clebert Suconic <clebert.suconic@gmail.com>
Date: Thu, 5 Oct 2017 18:22:49 -0400
Message-ID: <CAKF+bspg0SxjTret99r0zKMWY7dN8dWv-R6D9U=bJXChmkh5Pw@mail.gmail.com>
Subject: Re: [DISCUSS] URIs on Producer and Consumer
To: dev@kafka.apache.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 05 Oct 2017 22:22:56 -0000

Another useful feature for this...

Say that I am writing an application... if I write this URI on any of
my internal properties.. I can tweak my Consumer or Producer without
changing any code.

Say, you store the URI on your project's configuration xml..read it
and start your consumer... later on you could just update the XML, and
your user wouldn't need to rewrite his application.


right now if you (as an user) want to do such thing.. you would have
to write the parser yourself.

On Thu, Oct 5, 2017 at 4:33 PM, Michael Pearce <Michael.Pearce@ig.com> wrot=
e:
> To me, this is a lot more in line with many other systems connections, to=
 have the ability to have a single connection string / uri, is this really =
that left field suggesting or wanting this?
>
> If anything this bring kafka more standardised approach imo, to have a un=
ified resource identifier, protocol name and a set schema for that.
>
> e.g.
> Database connection strings like
>
> oracle:
> jdbc:oracle:thin:@(description=3D(address_list=3D
>    (address=3D(protocol=3Dtcp)(port=3D1521)(host=3DprodHost)))
> (connect_data=3D(INSTANCE_NAME=3DORCL)))
>
> or
>
> postgres:
> "jdbc:postgresql://localhost/test?user=3Dfred&password=3Dsecret&ssl=3Dtru=
e"
>
> And then like wise on messaging front, systems like
>
> rabbitmq
> =E2=80=9Camqp://myhost?heartbeat=3D10&connection_timeout=3D10000=E2=80=9D
>
>
> I personally like the suggestion +1.
>
>
> Cheers
> Mike
>
>
> On 05/10/2017, 20:10, "Clebert Suconic" <clebert.suconic@gmail.com> wrote=
:
>
>     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cmccabe@apache.org> wro=
te:
>     > We used URIs as file paths in Hadoop.  I think it was a mistake, fo=
r a
>     > few different reasons.
>     >
>     > URIs are actually very complex.  You probably know about scheme, ho=
st,
>     > and port, but did you know about authority, user-info, query, fragm=
ent,
>     > scheme-specific-part?  Do you know what they do in Hadoop?  The map=
ping
>     > isn't obvious (and it wouldn't be obvious in Kafka either).
>
>     URIs are just a hashmap of key=3Dstring.. just like Properties...
>
>     The Consumer and Producer is just having such hashMap.. and these
>     values are easy to translate to boolean, integer.. etc. We would just
>     need to add such mapping as part of this task when done. I don't see
>     anything difficult there.
>
>
>     >
>     > When you flip back and forth between URIs and strings (and you
>     > inevitably will do this, when serializing or sending things over th=
e
>     > wire), you run into tons of really hard problems.  Should you prese=
rve
>     > the "fragment" (the thing after the hash mark) for your URI, or not=
?  It
>     > may not do anything now, but maybe it will do something later.  URI=
s
>     > also have complex string escaping rules.  Parsing URIs is very mess=
y,
>     > especially when you start talking about non-Java programming langua=
ges.
>
>
>     Why flip back and forth? URIs would generate the same HashMap that's
>     being generated today.. I don't see any mess here.
>     Besides... This would be an addition, not replacement...
>
>     And I'm talking only about the Java API now.
>
>     Again, All the properties on ProducerConfig and ConsumerConfig seems
>     easy to be mapped as primitive types (String, numbers.. booleans).
>
>     Serialization shouldn't be a problem there. it would generate the sam=
e
>     properties it's generated now.
>
>     >
>     > URIs are designed for a world where you talk to a single host over =
a
>     > single port.  That isn't the world distributed systems live in.  Yo=
u
>     > don't want your clients to fail to bootstrap because the single ser=
ver
>     > you specified is having a bad day, even when the other 8 servers ar=
e up.
>
>     I have seen a few projects using this style of URI: I would make it
>     doing the same here:
>
>     If you have multiple hosts:
>
>     KafkaConsumer consumer =3D new
>     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property=
1=3Dvalue");
>
>     if you have a single host:
>     KafkaConsumer consumer =3D new
>     KafkaConsumer("kafka://host2:port?property1=3Dvalue&property2=3Dvalue=
2");
>
>
>     One example of an apache project using a similar approach is qpid-jms=
:
>     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failo=
ver-configuration-options
>
>
>     > The bottom line is that URIs are the wrong abstraction for the job.
>     > They just don't express what we really want, and they introduce a l=
ot of
>     > complexity and ambiguity.
>
>     I have seen the opposite to be honest. this has been simpler for me
>     and users I know than using a HashMap.. .  users in my experience ten=
d
>     to write this faster.
>
>     users can certainly put up with the HashMap.. but this is easier to
>     remember. I'm just proposing what I think it's a simpler API.
>
>
>
>
>     Perhaps we should move into the KIP discussion itself here.. I first
>     intended to start this thread to see if it would make sense or not...
>     But I don't have authorization to create the KIP page.. so again..
>     based on the contributing page.. can someone add me authorizations to
>     the WIKI space?
>
>
> The information contained in this email is strictly confidential and for =
the use of the addressee only, unless otherwise indicated. If you are not t=
he intended recipient, please do not read, copy, use or disclose to others =
this message or any attachment. Please also notify the sender by replying t=
o this email or by telephone (+44(020 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusion (etc) that do not relate to the =
official business of this company shall be understood as neither given nor =
endorsed by it. IG is a trading name of IG Markets Limited (a company regis=
tered in England and Wales, company number 04008957) and IG Index Limited (=
a company registered in England and Wales, company number 01190902). Regist=
ered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both=
 IG Markets Limited (register number 195355) and IG Index Limited (register=
 number 114059) are authorised and regulated by the Financial Conduct Autho=
rity.


--=20
Clebert Suconic