kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: How to use RPC mechanism in Kafka?
Date Mon, 22 Sep 2014 16:51:22 GMT
Agree with Svante, for your use case Kafka can be used to retain your
request messages, but it is not proper for RPC usages.

On Mon, Sep 22, 2014 at 6:06 AM, svante karlsson <saka@csi.se> wrote:

> 1 )  HA proxy -> node.js (rest api). Use aerospike as session store if you
> need one.
> 2)   Difficult since the prioritization must be passed to lower levels and
> that's usually "hard". (Get rid of this constraint and go for a SLA - like
> 99,9% within 50ms or something like that)
> 2b) Group you customers on different servers with different SLAs
> 3)   Log your requests in kafka from node.js (but do not expect replies on
> this path)
>
> /svante
>
>
>
> 2014-09-22 13:32 GMT+02:00 lavish goel <lavishgo@gmail.com>:
>
> > Thank you Svante for response.
> >
> > *Below are some requirements that makes us to consider Message Broker.*
> >     1. We have to handle 30,000 TPS.
> >     2. We need to prioritize the requests.
> >     3. Request Data should not be lost.
> >
> >
> > Thanks
> >
> > Regards
> > Lavish Goel
> >
> >
> >
> > On Mon, Sep 22, 2014 at 4:20 PM, svante karlsson <saka@csi.se> wrote:
> >
> > > Why do you want a message broker for RPC?
> > > What is "large amounts of requests"?
> > >
> > > /svante
> > >
> > >
> > >
> > >
> > >
> > > 2014-09-22 12:38 GMT+02:00 lavish goel <lavishgo@gmail.com>:
> > >
> > > > Thank you so much Svante for your response.
> > > >
> > > > The application which we are designing depends lot upon
> > request/response
> > > > mechanism. In that case should we move to some other message broker?
> If
> > > > yes, Can you please tell me the name which is best for this use case
> > and
> > > > can handle large amount of requests?
> > > > Is there any workaround in Kafka? If Yes, Please tell me.
> > > >
> > > > Thanks
> > > >
> > > > Warm Regards
> > > > Lavish Goel
> > > >
> > > >
> > > > On Mon, Sep 22, 2014 at 3:41 PM, svante karlsson <saka@csi.se>
> wrote:
> > > >
> > > > > Wrong use-case. Kafka is a queue (in normal case a TTL (time to
> live)
> > > on
> > > > > messages). There is no correlation between producers and consumers.
> > > There
> > > > > is no concept of a consumed message. There is no "request" and no
> > > > > "response".
> > > > >
> > > > > You can produce messages (in another topic) as result of your
> > > processing
> > > > > but you cant respond to a producer.
> > > > >
> > > > > /svante
> > > > >
> > > > >
> > > > > 2014-09-22 11:42 GMT+02:00 lavish goel <lavishgo@gmail.com>:
> > > > >
> > > > > > Thank you for your response. I have gone through the protocol
> wiki.
> > > > Now I
> > > > > > have some understanding of it.
> > > > > > Sorry for again asking the question.
> > > > > >
> > > > > > I want to know, Is it possible:
> > > > > >
> > > > > > Let say, I have producer PA,PB,PC. They send request messages
> A,B,C
> > > > > > respectively. Now these messages goes to topic. There are few
> > > consumers
> > > > > > that took message from topic and do some computation on message.
> > > > > > Let say after computation request A,B,C becomes AA,BB,CC
> > > respectively.
> > > > > Now
> > > > > > I want to send this response to Producer A,B,C respectively.
> > > > > >
> > > > > > Is it possible? if yes. Can You please explain in bit detail.
> > > > > >
> > > > > > Thank you so much
> > > > > >
> > > > > >
> > > > > > Regards
> > > > > > Lavish Goel
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 20, 2014 at 10:18 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Also you can take a look at the SimpleConsumer and SyncProducer
> > > > > > > implementation.
> > > > > > >
> > > > > > > On Sat, Sep 20, 2014 at 9:47 AM, Guozhang Wang <
> > wangguoz@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> You can take a look at the protocol wiki to understand
the
> > > request /
> > > > > > >> response data types. Kafka server accepts socket connection
> for
> > > > > > requests.
> > > > > > >>
> > > > > > >> Guozhang
> > > > > > >>
> > > > > > >> On Fri, Sep 19, 2014 at 10:42 PM, lavish goel <
> > lavishgo@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >>> Thank you for your response. I want to implement
> > request/response
> > > > > type
> > > > > > >>> model.
> > > > > > >>> For eg. I have a producer that publish a message(He
wants to
> > know
> > > > > some
> > > > > > >>> status) to topic. A consumer pulls the message
and processes
> > the
> > > > > > request.
> > > > > > >>> Now I want that, the response of this request should
go that
> > > > > producer.
> > > > > > Can
> > > > > > >>> you please tell me how can I implement this.
> > > > > > >>>
> > > > > > >>> Thanks
> > > > > > >>> Lavish
> > > > > > >>>
> > > > > > >>> On Sat, Sep 20, 2014 at 1:26 AM, Guozhang Wang
<
> > > wangguoz@gmail.com
> > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Do you mean that you want to know the protocol?
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> > > > > > >>>>
> > > > > > >>>> Guozhang
> > > > > > >>>>
> > > > > > >>>> On Fri, Sep 19, 2014 at 12:18 PM, lavish goel
<
> > > lavishgo@gmail.com
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>> > Hi,
> > > > > > >>>> >
> > > > > > >>>> > Please tell me how to use request/response
mechanism in
> > kafka?
> > > > > > >>>> > Thanks
> > > > > > >>>> > Lavish Goel
> > > > > > >>>> >
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> --
> > > > > > >>>> -- Guozhang
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> -- Guozhang
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message