tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Stanchev <Gstanc...@serena.com>
Subject RE: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
Date Fri, 09 Oct 2015 16:39:57 GMT
Just for the record, https.protocols is a property used by the HttpsUrlConnection class. If
your app is using a client that doesn't rely on the internal Oracle HTTP client, it's better
to use " jdk.tls.client.protocols" which is read directly by the socket/SSL classes. Apache
Http Client is one client that does use sockets directly instead of HttpsUrlConnection.


Also, I want to mention that we have finally been able to isolate a really nasty problem with
Java SSL implementation and Windows's SSPI based SChannel acting as a server. The issue we
ran into is very similar as to what the OP reported, that’s why I asked him a while ago
on a separate thread about keystores if the server is IIS-based. The issue that we ran into
might sound esoteric but our product ran into it very often. We have a configuration where
we lock our intracomponent communication via 2way SSL. We have Java Tomcat-based server components
that talk to IIS based native components via web services over HTTP which we secure with 2way
SSL. When we upgraded to JRE 8 we started getting those connection resets in Tomcat when we
enabled the 2way SSL. Those were on the Java->IIS calls that work over regular SSL. On
the java side, if you turn on the ssl debug on, you'd see connection reset IOException thrown,
and on the Windows side, you'd see " The following fatal alert was generated: 20. The internal
error state is 960." In the System event log. This error is " TLS1_ALERT_BAD_RECORD_MAC" from
SSPI. A search in Oracle's bug database found a close resemblance to this issue: https://bugs.openjdk.java.net/browse/JDK-8068812.
But the reporter couldn't reproduce it and also was against another Oracle component Java
Web Download or something like this. So we started working with Oracle but the intermittent
and hard-to-reproduce nature of this issue hampered the investigation. As the OP mentioned
below, there are apocryphal reports for this all over the web but people cannot troubleshoot
it very well. Anyway here are my findings. In order for the issue to arise all those conditions
must be met:

[1] The client must start with ClientHello TLSv1.2
[2] The server must respond with ServerHello TLSv1 (so the server should not have TLS1.2 enabled)
[3] The cipher key exchange must NOT be ECDHE. I have not tried DHE as I could not make Windows
take it. According to this doc [1] it should have several DHE key exchange ciphers available
but I could not make it work. RSA key exchange exhibits the issue. I have been using “TLS_RSA_WITH_AES_256_CBC_SHA”
to reproduce.
[4] The initial connection must not be 2way-SSL
[5] The server must upgrade the connection after encrypted payload is read. In case of IIS,
it has 2way SSL setting on a Virtual Directory. The initial handshake is not 2way SSL, once
the HTTP request is read, it determines that the requested resource (/gsoap) is 2way SSL-protected
and upgrades the connection to 2way SSL.
[6] The server must kill the connection abruptly when error happens. This is important as
if the server terminates the connection gracefully and keeps it around Java is able to recover
since it reuses the cached connection. IIS does exactly this.
[7] The server must be Windows 7, Win2k8, Win2k8-R2, Win8. Might be other versions affected
but bug is not evident on Win2k12 or Win2k12-R2 or Win10. Perhaps it was fixed, perhaps those
versions support TLSv1.2

It might sound complicated but it is not, it is very common. All you have to do is to have
a Virtual Directory in IIS that requires 2way SSL and is set to "want". RSA-based key exchanges
are one of the most commonly used ciphers. The Windows platforms we tested with are still
widely used in the enterprises while the adoption rate of win2k12+ is still lagging (from
our experience with our customers). And you have equal "chance" of cipher support between
Windows servers and Java 8 clients to land on RSA based cipher (which exhibits the issue)
or ECDHE key exchange that is ok. Again, it might sound too complicated but it was blocker
for us to ship our product.

For us, to work around this issue is for the 2way SSL to turn off TLSv1.2 (via "-Djdk.tls.client.protocols=TLSv1"
or RSA key exchange algorithm (via the java security properties) on affected systems which
fixes it.

I have a lot more technical details that I am not sharing here. I have sent them to Oracle
so hopefully that gets resolved, but my guess is that is really an SSPI/SChannel issue in
which case someone needs to work with Microsoft and I don't know if my company can do that.
It boils down to - when all these conditions are met, is Java creating a bad MAC or SChannel
fails to verify that MAC correctly on SSL handshake step X.

Hopefully this can help someone else, as we have spent quite a bit of time on this issue.

George


TL;DR: There is a nasty bug in Java or Windows when calling SSPI-based app over 2way SSL that
you can work around by turning off TLSv1.2 or RSA key exchanges.

-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net] 
Sent: Monday, August 31, 2015 11:15 AM
To: Tomcat Users List
Subject: Re: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Diarmuid,

On 8/31/15 11:36 AM, dmccrthy wrote:
> To cut a long story short openssl helped. Using openssl -connect 
> showed the error below. When I added the -tls1 flag the error went 
> away.
> 
> 2104:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or 
> bad record mac:.\ssl\s3_pkt.c:532:
> 
> Configuring Tomcat with the JVM parameter below resolved the issue.
> 
> -Dhttps.protocols="TLSv1"

Yep, you have to use a TLS handshake; the older SSL3 handshake won't wor k.

Now... you *can* enable SSLv2Hello and then only support TLSv1+, but let's face it: SSL is
dead and clients should get with the program.

> This may limit our client Web application but I have very little 
> influence over the server-side application, so it'll have to do as a 
> solution for now.

Your client should not try to use SSL handshake unless you know it's required.

> So the issue seems to be a Java handshake error whereby it has a 
> problem downgrading from TLSv1.2 to TLSv1 during the handshake.

I don't know enough about TLS to know if there is an appreciable difference between the handshake
protocol between the two. I know that
SSLv2 was the lowest common denominator for a while, and most servers would enable it *just
for the handshake* but would refuse to actually negotiate an SSLv2 connection (SSLv2 was basically
DOA). These days, nobody should be using SSL at all, but I might understand wanting to use
SSL-compatible handshakes, so using "SSLv2Hello" but not supporting the SSLv3 protocol would
be the way to do that.

Again, we should just let SSL die. The more we push clients and servers to only negotiate
TLS from here on out, the better off Internet security will be overall.

> Why it works in our non-prod environment with the same versions of 
> everything is a mystery, but I can speculate at some difference in 
> underlying O/S settings, or perhaps a Java bug.

Or more likely a configuration option you didn't detect when you checked everything out.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJV5IudAAoJEBzwKT+lPKRY+xsP/RBgD3VErGPgLZIOu6GDdUn9
4e/yH5kVAdiuOcnn26En+OJKGONK5c9Hf9gW23utPw2ruG1KYwSkr9dJ4yFIiKxh
1wYhq4oTN3j8DwNb8C2Y522eyH9hRYo+W8S7er0pvklcGiImgx+vp/LYitM2dHeG
6MEPJ03+/5CLbrmYfLbcZvnPydqKYVJtIqozHGhrC5+fAbcFFsWBcwrfk0Menwji
wE4w7o5yi38/MtXX1b3FtMnvyjfiKHdPoKJ4PJv8KA3hKjWzEVLiehFGEQGb9VS6
uqgMLB5YR3QrI0yYN53soi4apgqktcfwGimK+B5Ar/2uvsarLvZgIF8uYuEHXTYY
bn+V7ngmRxuwX71VUJDP47MtwQsgxko4H1qa2jIh7Pp8kRHeHwKytJxOLPUepmKp
K1rmii+OtVPMQusXGVJedohXMwOCi3kUf9exmadnN02EgAPmIh/rfy7uBARfYJdC
CqjiQxmTCm5yKu1uB9PXFbZMiCFjq+Hw3g6/cVnn1X5nxlfkPBCAODk5UouRrO+w
oIErBEAbLSb7twWo8S6B9pupb/8C4RNvdhPk9pCfPf2cC+Ouybg4Sz/S4KSQ1N8P
0zfXVoDL4AJtl3k1ylXbfPVRE5SY+ZMqT4fC3odI1frWGI3oSnnXNQsqxr4ehmpo
PUDzJVZbKxLMruMEbnSh
=OrTI
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Mime
View raw message