kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Dupriez <alexandre.dupr...@gmail.com>
Subject Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM
Date Thu, 01 Aug 2019 08:42:16 GMT
Thanks Gokul and Stanislav for your answers.

I went through the PR 5940 [1]. Indeed Gokul, your reasoning echoes the
observations of Ismael about a potential inter-protocol layering violation.

As you said Stanislav, the server-side SSL engine responds with an alert
with code 80 (internal_error) from what I saw when reproducing the OOM.
Since the Alert is generated below the application layer, I am not sure
what could be done on the broker to handle the scenario more gracefully.
Interestingly, the SSL engine emits the possibility of receiving a
plaintext message in debug logs [2].

The idea was indeed to perform a simple check on the message size decoded
in NetworkReceive against a configurable value, and throw
an InvalidReceiveException in a similar fashion as what happens on the
broker, where a non-unlimited maxSize can be provided. Basically, mimic the
behaviour on the broker. The default value for the maximal request size on
the broker is 100 MB which you are suggesting to use client-side.

Adding a client configuration property for clients may be an overkill here.
What I am going to ask is naive but - is it theoretically possible for the
broker to legitimately send responses over 100 MB in size?

Thanks,
Alexandre

[1] https://github.com/apache/kafka/pull/5940
[2]

kafka-network-thread-0-ListenerName(SSL)-SSL-4, fatal error: 80: problem
unwrapping net record

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

kafka-network-thread-0-ListenerName(SSL)-SSL-4, SEND TLSv1.2 ALERT:  fatal,
description = internal_error

kafka-network-thread-0-ListenerName(SSL)-SSL-4, WRITE: TLSv1.2 Alert,
length = 2

kafka-network-thread-0-ListenerName(SSL)-SSL-4, called closeOutbound()

kafka-network-thread-0-ListenerName(SSL)-SSL-4, closeOutboundInternal()

kafka-network-thread-0-ListenerName(SSL)-SSL-4, called closeInbound()

kafka-network-thread-0-ListenerName(SSL)-SSL-4, fatal: engine already
closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before
receiving peer's close_notify: possible truncation attack?

[Raw write]: length = 7

0000: 15 03 03 00 02 02 50                               ......P


Le jeu. 1 août 2019 à 08:50, Stanislav Kozlovski <stanislav@confluent.io> a
écrit :

> Hey Alexandre, thanks for the KIP!
>
> I had personally hit the same problem and found it very annoying.
> Have you considered detecting such a scenario in the client and simply
> throwing an exception, rather than allocating any memory?
> I have an open PR that does just that -
> https://github.com/apache/kafka/pull/5940
> <https://github.com/apache/kafka/pull/5940/files>
>
> Best,
> Stanislav
>
> On Wed, Jul 31, 2019 at 10:35 AM Gokul Ramanan Subramanian <
> gokul2411s@gmail.com> wrote:
>
> > Hi Alex.
> >
> > A rejected alternative is to try to do SSL handshake from the plaintext
> > transport layer. This couples various transport layers and is inflexible
> to
> > adding new transport layers in the future. Further, if the plaintext
> > transport layer does SSL handshake, it will always get an error,
> > irrespective of whether or not the peer supports SSL. Therefore, the
> > plaintext transport layer would have to determine if the peer uses SSL or
> > not based on the type of error it gets back from the SSL handshake. This
> > fits right into the general anti-pattern of using exceptions as control
> > flow mechanisms.
> >
> > Another rejected alternative would be to have a single byte at the
> > transport layer represent the security protocol in use. This byte would
> > have to be present in every single message between clients and brokers
> and
> > between brokers and brokers. This change is backwards-incompatible and
> adds
> > overhead to Kafka. It is likely never going to get accepted by the
> > community.
> >
> > In the absence of a fool-proof way to do a handhskake across all security
> > protocols (plaintext, SSL, other future ones), I think that the proposed
> > KIP provides a good solution. Therefore, you have my +1. (Not sure if my
> +1
> > counts, since I am a Kafka noob).
> >
> > Thanks.
> > Gokul Ramanan Subramanian
> > Senior SDE, Amazon AWS
> >
> > On 28/Jul/19 05:43:19PM +0100, Alexandre Dupriez wrote:
> >  Hello,
> >
> >  I have created the KIP-498
> >  <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM
> > >
> >  to
> >  propose a minor addition to the producer configuration properties, in
> > order
> >  to protect against OOM when the response message decoded by a client
> >  indicates an "abnormally high" size to be allocated.
> >
> >  This follows this discussion thread
> >  <https://www.mail-archive.com/dev@kafka.apache.org/msg55658.html> and
> is
> >  tracked by KAFKA-4090 <https://issues.apache.org/jira/browse/KAFKA-4090
> >.
> >
> >  Please let me know what you think. I created this KIP even though the
> >  change seems minimal because it affects client configuration, which is
> >  mentioned as a type of change eligible for a KIP on this main wiki page
> >  <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> >  .
> >
> >  Thanks,
> >  Alexandre
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message