kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Crayford <tcrayf...@heroku.com>
Subject Re: Plans to improve SSL performance in Kafka, for 0.10.x?
Date Tue, 06 Sep 2016 10:35:02 GMT
> We have a single broker

Kafka isn't really designed to run with a single broker setup. Whilst I
doubt it impacts your problems here, you should immediately bump your
broker count to 3 or more.

Our performance tests were run with the JVM producer and consumer, and we
didn't notice any real slowdown on that side either when enabling SSL. Just
to check the obvious: are your producers and consumers going over a fast
connection to the broker?

> This GC you notice, is it on the broker(s) or on the clients which hosts
the producer/consumers? Either way, this is a good hint and something I
hadn't considered in my setup. I'll see if I can get some GC logs/metrics
on our setup for this. Thanks for the input.

On the broker - we never looked at consumers or producers, but that might
be a good place to look.

On Tue, Sep 6, 2016 at 5:40 AM, Jaikiran Pai <jai.forums2013@gmail.com>
wrote:

> On Tuesday 06 September 2016 09:42 AM, Jaikiran Pai wrote:
>
>> Hi Todd,
>>
>> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
>>> SHA1PRNG.
>>>
>>
>> Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our
>> setup. However, right now, (intentionally) we have single broker within the
>> setup and based on the JIRA details, I believe this one won't affect us for
>> now.
>>
>
> I read that JIRA again and saw the stacktrace. I might be wrong when I
> said this probably doesn't affect us given that the code path appears to
> the same for other SSL interactions. I'm going to introduce this patch (or
> in fact the latest 0.10 branch) in our setup and see how it goes.
>
> -Jaikiran
>
>
>
>
>>
>>
>>> Right now, we're specifically avoiding moving consume traffic to SSL, due
>>> to the zero copy send issue. Now I've been told (but I have not
>>> investigated) that OpenSSL can solve this. It would probably be a good
>>> use
>>> of time to look into that further.
>>>
>>
>> Unfortunately, the way our application is written, we don't have the
>> ability to enable SSL for producers and disable it for consumers. Changing
>> the application to support something like that is doable, but I will have
>> to make sure that this is one of the reasons for the slowness.
>>
>>
>> Thanks for all your inputs. I was trying to avoid coming up with a
>> reproducer which closely matches our application usage scenario (given our
>> tight delivery timelines for the product), but from what you say and a
>> reply Tom Crayford in this thread, I am starting to feel that not everyone
>> is affected to the extent we are. I will see if I can narrow to this down
>> to something more concrete.
>>
>> -Jaikiran
>>
>>
>>
>>
>>
>>> That said, switching the message format to the newer option (KIP-31 I
>>> believe?) will result in the brokers not needing to recompress message
>>> batches that are produced. This should result in a significant reduction
>>> in
>>> CPU usage, which may offset the cost of SSL. We haven't had a chance to
>>> fully investigate this, however, as changing that config depends on the
>>> clients being updated to support the new format.
>>>
>>> -Todd
>>>
>>> On Sunday, September 4, 2016, Jaikiran Pai <jai.forums2013@gmail.com>
>>> wrote:
>>>
>>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
>>>> decided to start using SSL for Kafka communication between broker and
>>>> clients. Right now, we have a pretty basic setup with just 1 broker with
>>>> SSL keystore setup and the Java client(s) communicate using the
>>>> Producer/Consumer APIs against this single broker. There's no client
>>>> auth
>>>> (intentionally) right now. We also have plain text enabled for the
>>>> initial
>>>> testing.
>>>>
>>>> What we have noticed is that the consumer/producer performance when SSL
>>>> is
>>>> enabled is noticeably poor when compared to plain text. I understand
>>>> that
>>>> there are expected to be performance impacts when SSL is enabled but the
>>>> order of magnitude is too high and in fact it shows up in a very
>>>> noticeable
>>>> fashion in our product. I do have the numbers, but I haven't been able
>>>> to
>>>> narrow it down yet (because there's nothing that stands out, except that
>>>> it's slow). Our application code is exactly the same between non-SSL and
>>>> SSL usage.
>>>>
>>>> Furthermore, I'm aware of this specific JIRA in Kafka
>>>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
>>>> similar issue. So what I would like to know is, in context of Kafka
>>>> 0.10.x
>>>> releases and Java 8 support, are there any timelines that the dev team
>>>> is
>>>> looking for in terms of improving this performance issue (which I
>>>> believe
>>>> requires usage of OpenSSL or other non-JDK implementations of
>>>> SSLEngine)?
>>>> We would like to go for GA of our product in the next couple of months
>>>> and
>>>> in order to do that, we do plan to have Kafka over SSL working with
>>>> reasonably good performance, but the current performance isn't
>>>> promising.
>>>> Expecting this to be fixed in the next couple of months and have it
>>>> available in 0.10.x is probably too much to expect, but if we know the
>>>> plans around this, we should be able to come up with a plan of our own
>>>> for
>>>> our product.
>>>>
>>>>
>>>> -Jaikiran
>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message