kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wim Van Leuven <wim.vanleu...@highestpoint.biz>
Subject Re: Plans to extend streams?
Date Wed, 29 Nov 2017 14:56:09 GMT
What you are actually asking is if Kafka Streams should be reimplemented as
Apache Storm?
-wim

On Wed, 29 Nov 2017 at 15:10 Adrienne Kole <adriennekole1@gmail.com> wrote:

> Hi,
>
> The purpose of this email is to get overall intuition for the future  plans
> of streams library.
>
> The main question is that, will it be a single threaded application in the
> long run and serve microservices use-cases, or are there any plans to
> extend it to multi-node execution framework with less kafka dependency.
>
> Currently, each streams node 'talks' with kafka cluster and they can
> indirectly talk with each other again through kafka. However, especially if
> kafka is not in the same network with streams nodes (actually this can
> happen if they are in the same network as well) this will cause high
> network overhead and inefficiency.
>
> One solution for this (bypassing network overhead) is to deploy streams
> node on kafka cluster to ensure the data locality. However, this is not
> recommended as the library and kafka can affect each other's performance
> and  streams does not necessarily have to know the internal data
> partitioning of kafka.
>
> Another solution would be extending streams library to have a common
> runtime. IMO, preserving the current selling points of streams (like
> dynamic scale in/out) with this kind of extensions can be very good
> improvement.
>
> So my question is that, will streams in the long/short run, will extend its
> use-cases to massive and efficient stream processing (and compete with
> spark) or stay and strengthen its current position?
>
> Cheers,
> Adrienne
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message