From dev-return-56100-archive-asf-public=cust-asf.ponee.io@thrift.apache.org Sat May 18 06:39:18 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 05E1C18064C for ; Sat, 18 May 2019 08:39:17 +0200 (CEST) Received: (qmail 57813 invoked by uid 500); 18 May 2019 06:39:17 -0000 Mailing-List: contact dev-help@thrift.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@thrift.apache.org Delivered-To: mailing list dev@thrift.apache.org Received: (qmail 57802 invoked by uid 99); 18 May 2019 06:39:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 May 2019 06:39:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0280DC2D04 for ; Sat, 18 May 2019 06:39:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.003 X-Spam-Level: * X-Spam-Status: No, score=1.003 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Nc6bOVE3K_-Y for ; Sat, 18 May 2019 06:39:13 +0000 (UTC) Received: from oldblue2.lightblue.com (213.242-48-213.static.virginmediabusiness.co.uk [213.48.242.213]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id EB0A65F195 for ; Sat, 18 May 2019 06:39:12 +0000 (UTC) Received: from [192.168.220.13] by oldblue2.lightblue.com (GMS 15.01.3664/KW9347.00.a504e021) with ESMTP id debauaaa for dev@thrift.apache.org; Sat, 18 May 2019 07:37:30 +0100 Reply-To: From: "John Dougrez-Lewis" To: References: <025c01d50adb$bca08650$35e192f0$@com> <03ea01d50b41$6d299880$477cc980$@com> <04c101d50b9f$ec294190$c47bc4b0$@com> <88ED3ECFF82541E6A3570C8AF2EF5DB4@HAGGIS> In-Reply-To: <88ED3ECFF82541E6A3570C8AF2EF5DB4@HAGGIS> Subject: RE: THRIFT-66 - Bidirectional communication Date: Sat, 18 May 2019 07:39:06 +0100 Organization: Light Blue Ltd Message-ID: <096b01d50d44$63c85bf0$2b5913d0$@com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Content-Language: en-gb Thread-Index: AdUNAL47sx5qGoe6TH+gUdHl42QopwAPU09g Hi, > At least PUB/SUB can be implemented solely on the transport level. No = IDL change necessary. My understanding is that Thrift generally provides only synchronous RPC, = making with one synchronous return. How would multiple subsequent = Subscription updates be sent back to the Client? > async was already a keyword long ago. It's now called "oneway". But 'oneway' provides no guaranteed deterministic mechanism of = indicating back to the client any subsequent processing failures of the = call on the server side, so it appears very fragile and so would be = difficult to justify it being used in any serious professional = applications such as Financial Trading Systems.=20 > Not sure if I can follow. What exactly would that preprocessor do? = Generate IDL from IDL? Yes. >> [in/out/inout] parameter > There was some discussion about this a while ago, don't remember the = ticket.=20 > Technically that should be possible, since the data are transferred in = a=20 > struct anyway, so multiple out parameters (or even in/out) should = really not=20 > be that hard to implement. Yes, even with just a synchronous IDL extension, out parameters can be = always be rewritten as members of a struct in the returned values type. ------------------ In addition, Callback methods could be passed as arguments in the = A<=3D>B extended IDL and translated into interface methods on the B=3D>A = async response path, e.g. A<=3D>B IDL: [aync] methodReturnType Method(arg1, arg2, callbackType Arg) // define callbackType signature cbReturnType callbackType(cbArgtype1, cbArgtype2) =3D=3D=3D=3D=3D=3D=3D> generates: A=3D>B IDL // immediate return to provide handle for context for subsequent async = return // requires caller to supply callbackHandle to provide context for = callback handleType Method(arg1, arg2, callbackHandle) B=3D>A IDL // async return from method call void Method(handleType, methodReturnType) // call for async callback cbReturnType CallbackType(handleType, callbackHandle, cbArgtype1, = cbArgtype2) ----------------- Regards, John -----Original Message----- From: Jens Geyer [mailto:jensg@apache.org]=20 Sent: 17 May 2019 23:36 To: dev@thrift.apache.org Subject: Re: THRIFT-66 - Bidirectional communication Hi, > That gets you to the point where Thrift supports and generates=20 > bidirectional, async & pub/sub based on IDL At least PUB/SUB can be implemented solely on the transport level. No = IDL change necessary. > adding attributes [asyc] method/interface async was already a keyword long ago. It's now called "oneway". > pre-processing the extended IDL with a new pre-processor to generate = the=20 > representations of the service definitions for both sides in terms of = the=20 > current IDL Not sure if I can follow. What exactly would that preprocessor do? = Generate=20 IDL from IDL? As a general statement, I would recommend to strictly separate between = an=20 interface contract (which the IDL is and should be) and a concrete=20 implementation or configuration setup (which the IDL should not be). > [in/out/inout] parameter There was some discussion about this a while ago, don't remember the = ticket.=20 Technically that should be possible, since the data are transferred in a = struct anyway, so multiple out parameters (or even in/out) should really = not=20 be that hard to implement. I have no idea if we run into subtle problems = with certain languages and how that could be circumvented (by using = structs=20 maybe) and if in/out is really supported by all of them on a language = level,=20 we will find out. Have fun, JensG -----Urspr=C3=BCngliche Nachricht-----=20 From: John Dougrez-Lewis Sent: Thursday, May 16, 2019 6:29 AM To: 'James E. King III' Cc: dev@thrift.apache.org Subject: RE: THRIFT-66 - Bidirectional communication Yes, my suggestion is for an enhancement to Thrift to make it = bidirectional,=20 as follows: 1) use 2 connections, A =3D> B, and B =3D> A using what is already in = place. 2) write, by hand, a new, supporting choreography code to establish this = double connection, in the first instance for a single language 3) extend the IDL support this - adding attributes [asyc] = method/interface,=20 [pub/sub] method/interface, [in/out/inout] parameter 4) extend the IDL generation to implement this: i) pre-processing the extended IDL with a new pre-processor to=20 generate the representations of the service definitions for both sides = in=20 terms of the current IDL ii) generate the language dependent code required to hook up at of = the=20 two ends. 5) implement across multiple languages That gets you to the point where Thrift supports and generates=20 bidirectional, async & pub/sub based on IDL Then, optionally: 6) Having achieved a working implementation defined by an extended IDL,=20 consider re-implementing in terms of a single bidirectional transport = rather=20 than 2 existing unidirectional independent transports. The extended IDL defining a single async/pubsub A<=3D>B interface is = used to=20 generate 2 interfaces: an outgoing A=3D>B and an async response return = channel=20 B=3D>A, e.g.: A<=3D>B [async] returntype functionName (arg1, arg2) =3D> A=3D>B Handletype functionName (arg1, arg2) B=3D>A void functionName (Handletype, returntype) A<=3D>B [async] returntype functionName (arg1, arg2, [out] arg3) =3D> A=3D>B Handletype functionName (arg1, arg2) B=3D>A void functionName (Handletype, returntype, arg3) A<=3D>B [async] returntype functionName (arg1, arg2, [inout] arg3) =3D> A=3D>B Handletype functionName (arg1, arg2, arg3in) B=3D>A void functionName (Handletype, returntype, arg3out) -----Original Message----- From: James E. King III [mailto:jking@apache.org] Sent: 15 May 2019 19:02 To: jlewis@lightblue.com Cc: James E. King III; dev@thrift.apache.org Subject: Re: THRIFT-66 - Bidirectional communication Re: Visio - There are no "endpoint" implementations that allow for=20 bi-directional communication over a single transport connection today. Is that the Visio you were referring to? The only implementation of = that=20 concept is buried in the THRIFT-66 attachments, and only for C#, and it = was=20 done on a codebase about 8 years past... With today's thrift code if you want either end to be a client (make requests) or a server (reply to requests) you would need to separately=20 instantiate a client or server on each end and have them connect to=20 each-other, i.e. A ---> B (A sends requests to a thrift server on B, B replies, on a=20 transport) A <--- B (B sends requests to a thrift server on A, A = replies, on=20 a transport separate from the last one) It sounds like what you'd like to get to is: A <--> B (A and B can function as a client or as a server over the same=20 transport) Thrift cannot do the latter today, so I would recommend the former, = using=20 one transport for each direction. Given they will both be on the same system, if your languages support = it,=20 unix domain sockets are quite fast. Otherwise if you want a shared memory solution you need to write your = own=20 transport for that. - Jim On Wed, May 15, 2019 at 1:12 PM John Dougrez-Lewis = =20 wrote: > > Hi Jim, > > The "oneway" route looks a bit fragile in the face of failures in the=20 > subsequent server-side processing which then cannot be signalled back = to=20 > the client. > > The 2-way connection would be the way forward for me, particularly = since=20 > it would work with across multiple languages. > > My primary use case would be a simple language bridge mechanism for a=20 > library to allow processes coded in one language to call, with = potentially=20 > asynchronously returns, and pub/sub, to another process hosting a = library=20 > coded in another language running (in the first instance) on the same = box,=20 > communicating via IPC, preferably fast shared memory, but failing that = > sockets would do. > > You put a Visio diagram up back in 2010. Is the underlying source code = for=20 > that available now ? > > Rather than hand-rolling the 2-way connection setup/teardown and=20 > supporting code, for each and every language each time, it would be = nice=20 > if Framework code for that could be generated automatically from an=20 > enhanced and extended version of the IDL. > > Regards, > > John > > -----Original Message----- > From: James E. King III [mailto:jking@apache.org] > Sent: 15 May 2019 12:05 > To: dev@thrift.apache.org; jlewis@lightblue.com > Subject: Re: THRIFT-66 - Bidirectional communication > > Hello! > > Thrift is still a dedicated client/server model environment where = clients=20 > can request and servers reply. The easiest way to make it 2-way today = is=20 > to open a connection both ways. If you don't have firewalls in the = way=20 > then you can do this effectively. The more difficult and more correct = way=20 > to do it would be to rewrite the transport layer to use endpoints in = which=20 > each side can be a client and/or server for any number of services = (using=20 > TMultiplexedProtocol on top of another protocol, like = TBinaryProtocol).=20 > This design allows one end to be a "listener", one end to be an=20 > "initiator" (starts the connection), and after they connect they are = equal=20 > peers with the ability to request or reply of each-other. > > You can approximate asynchronous behavior by exclusively using = "oneway"=20 > requests in your design. I'd suggest avoiding use of oneway requests = with=20 > THttpProtocol varieties however as today there are some issues, since = Http=20 > transport requires a response to be sent, and "oneway" dictates there = is=20 > no reply, and most languages do not handle it well right now (there = are=20 > open backlog issues for this). > > For a matrix of supported languages, protocols, transports, and server = > types, see the file LANGUAGES.md at the root of the github repository. > > Another idea I was toying with a while ago was to add a message bus=20 > transport to Thrift which would allow for things like reliable = delivery=20 > and broadcast semantics but that also does not exist today. > > - Jim > > On Wed, May 15, 2019 at 1:05 AM John Dougrez-Lewis = =20 > wrote: > > > > Hi, > > > > > > > > I was looking for a mechanism to be able to provide > > language-agnostic API support to a hobby project I've been working = on=20 > > for some time. > > > > > > > > By following a trail of papers, books and references, I eventually > > came across Apache Thrift and have found and started going through > > Randy Abernethy's new book. > > > > > > > > > > > > Essentially what I was looking for was support for asynchronous > > calls, and by extension, pub/sub and two way communication across > > and between multiple languages over some channel, preferably IPC but = in=20 > > the worst case sockets. > > > > > > > > > > > > Having read the book, I can see that there is support for basic > > synchronous RPC between a client and a server over a significant > > number of languages and for just a very few languages, such as java, > > some element of support for asynchronous callbacks, and otherwise > > one-way methods that do not provide indication of subsequent = failure. > > > > > > > > It appeared to me one way of extending bi-directional asynchronous > > support would be to have the client to set itself up as a server for > > the server at the other end to connect to, and then it would just be > > a question of choreographing the setting up of a pair of RPC = channels. > > > > > > > > An asynchronous call could be implemented by providing a synchronous > > method that simply immediately returns a handle to the caller, and > > the server would then continue to process the call request on a > > background threadpool thread on the server, and the async result > > would then be signalled by a call from the server back to the client > > on the 2nd channel with the handle providing a context to lookup the = > > result. > > > > > > > > Pub/sub would just then be multiple calls from the server back to > > the client. > > > > > > > > The whole thing could sit on top of the existing unidirectional RPC > > implementation and provide full asynchronous calls & pub/sub across > > *ALL* supported languages at probably very little additional effort, > > with no changes to the existing code. > > > > > > > > You could then have a framework that extended the existing IDL to > > include decoration with attributes for async & pub/sub methods & = in/out=20 > > parameters. > > > > > > > > This extended IDL could then be pre-processed to generate > > client-server and server-client service definitions in the existing > > base IDL language, together with generating supporting glue code to > > compile to provide the support for hooking up the channels between = each=20 > > side. > > > > > > > > I note that THRIFT-66 was raised 10 years ago, but it looks like the > > C# code was never made available for release by Dell. > > > > > > > > I have some questions: > > > > > > > > 1) What is the current state of plans for this supporting this = sort=20 > > of > > functionality? What issues have been encountered ? > > > > > > > > 2) Is there a document/spreadsheet somewhere showing a matrix = of=20 > > what > > Transports and Protocols are supported for each language? > > > > > > > > > > > > Regards, > > > > > > > > John > > > > > > > > > > > > > > >