openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos Santana <csantan...@gmail.com>
Subject Re: Persistent Kafka Connections
Date Wed, 09 Jan 2019 14:32:17 GMT
Is one of those things that works well for a hello world low scale
deployment.

In practice we saw a lot of actions running in parallel from multiple
tenants and all of these even each one doing their caching were opening too
many new socket connection to a kafka service and eventually the kafka
service started to throttle and rate limit the number of new connections.

This is the reason the code it’s there in the repo but deprecated and not
recommended. From talking to kafka experts in ibm it was recommended from
them that having stateless clients doing a lot of new connection it is a
antipattern

So they recommended to try to have a low number of connections open and
high thruput for each of these connection, this means if you wan to produce
from actions at scale and multitenant the actions would need to go thru
some type of broker via http and this broker is a stateful service that
maintains the socket connection to kafka service.

— Carlos

On Wed, Jan 9, 2019 at 8:12 AM Michele Sciabarra <michele@sciabarra.com>
wrote:

> Hello whiskers,
>
> I am working on a simple webchat based on OpenWhisk  (full disclosure; it
> is for the book on OpenWhisk for O'Reilly!) and I am using MessageHub.
>
> However in the documentation for the messaging package I read that the
> /whisk.system/messaging/messageHubProduce is deprecated, and the
> recommended way is to use a persistent connection to Kafka.
>
> This statement is puzzling me.  My first idea is "why do not they cache
> the connection?", but I checked the code for messageHubProduce.py here:
>
>
> https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/action/messageHubProduce.py
>
> and looks like the producer is cached, so I wonder if the deprecation of
> messageHubProduce is still valid.
>
> Furthermore, I wrote an action to send messages to MessageHub with the
> idea of implementing a cached connection, and apparently works:
>
>
> https://github.com/learning-apache-openwhisk/chapter11-messages/blob/master/src/persistent/main/send.go
>
> So the question is: is this the correct way to send messages to Kafka
> (caching connnections?) Can I recommend this practice in the book?
>
>
> --
>   Michele Sciabarra
>   michele@sciabarra.com
>
-- 
Carlos Santana
<csantana23@gmail.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message