tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Stanchev <Gstanc...@serena.com>
Subject RE: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
Date Tue, 13 Oct 2015 16:35:12 GMT
Hi Chris,

[1] states: " JDK 7-9 enables SSLv2Hello on the server side only.  (Will not send, but will
accept SSLv2Hellos)"

I've opened support case both MS and already there is a bug filed with Oracle on this and
really, to be absolutely certain if the issue is in Java or SChannel, one would have to write
a non-Java client that that mimics the handshake messages send from Java with something like
OpenSSL or NSS or whatever and see if the bug replicates. Writing a Java/Java server client
could still leave some doubts as one can argue the code reuse could mask the problem - it
works but the bug on the client side is hidden by the server containing similar/same bug so
the MACs check out...

Unfortunately I don't have the time to invest in this more than I already had. And if MS support
engineers can pass it on to someone from the windows core team may be we can have some movement
forward. 

George


[1] http://www.oracle.com/technetwork/java/javase/documentation/cve-2014-3566-2342133.html

-----Original Message-----
From: George Stanchev [mailto:Gstanchev@serena.com] 
Sent: Tuesday, October 13, 2015 10:26 AM
To: Tomcat Users List
Subject: RE: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac

Hi Chris,

How do you force Java 8 to use SSLv2Hello? So far I am unable to force it to go down to use
SSLv2Hello. It would've been nice to be able to control the first handshake message version
(TLSv1 or TLSv1.2 or SSLv2Hello), but I have not been able to do that...

George

-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Tuesday, October 13, 2015 7:42 AM
To: Tomcat Users List
Subject: Re: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

George,

On 10/9/15 12:46 PM, George Stanchev wrote:
> One more clarification: on point [6] below I stated that Java is able 
> to recover with a retry on a cached connection. Unfortunately that is 
> only valid for higher level classes like HttpUrlConnection which makes
> 1 retry on IOException (and only on a GET and some other conditions). 
> The lower-level sockets just throw and that’s it...

Can you confirm whether using SSLv2Hello also solves the problem? Or do you have a requirement
that nothing less than TLSv1 may even handshake properly?

- -chris

> -----Original Message----- From: George Stanchev 
> [mailto:Gstanchev@serena.com] Sent: Friday, October 09, 2015 10:40 AM
> To: Tomcat Users List Subject: RE: [OT] Tomcat 7.0.55/Jre 7u67:
> SEND TLSv1 ALERT: fatal, description = bad_record_mac
> 
> Just for the record, https.protocols is a property used by the 
> HttpsUrlConnection class. If your app is using a client that doesn't 
> rely on the internal Oracle HTTP client, it's better to use "
> jdk.tls.client.protocols" which is read directly by the socket/SSL 
> classes. Apache Http Client is one client that does use sockets 
> directly instead of HttpsUrlConnection.
> 
> 
> Also, I want to mention that we have finally been able to isolate a 
> really nasty problem with Java SSL implementation and Windows's SSPI 
> based SChannel acting as a server. The issue we ran into is very 
> similar as to what the OP reported, that’s why I asked him a while ago 
> on a separate thread about keystores if the server is IIS-based. The 
> issue that we ran into might sound esoteric but our product ran into 
> it very often. We have a configuration where we lock our 
> intracomponent communication via 2way SSL. We have Java Tomcat-based 
> server components that talk to IIS based native components via web 
> services over HTTP which we secure with 2way SSL. When we upgraded to 
> JRE 8 we started getting those connection resets in Tomcat when we 
> enabled the 2way SSL. Those were on the
> Java->IIS calls that work over regular SSL. On the java side, if
> you turn on the ssl debug on, you'd see connection reset IOException 
> thrown, and on the Windows side, you'd see " The following fatal alert 
> was generated: 20. The internal error state is 960." In the System 
> event log. This error is "
> TLS1_ALERT_BAD_RECORD_MAC" from SSPI. A search in Oracle's bug 
> database found a close resemblance to this issue:
> https://bugs.openjdk.java.net/browse/JDK-8068812. But the reporter 
> couldn't reproduce it and also was against another Oracle component 
> Java Web Download or something like this. So we started working with 
> Oracle but the intermittent and hard-to-reproduce nature of this issue 
> hampered the investigation. As the OP mentioned below, there are 
> apocryphal reports for this all over the web but people cannot 
> troubleshoot it very well. Anyway here are my findings. In order for 
> the issue to arise all those conditions must be met:
> 
> [1] The client must start with ClientHello TLSv1.2 [2] The server must 
> respond with ServerHello TLSv1 (so the server should not have
> TLS1.2 enabled) [3] The cipher key exchange must NOT be ECDHE. I have 
> not tried DHE as I could not make Windows take it. According to this 
> doc [1] it should have several DHE key exchange ciphers available but 
> I could not make it work. RSA key exchange exhibits the issue. I have 
> been using “TLS_RSA_WITH_AES_256_CBC_SHA” to reproduce. [4] The 
> initial connection must not be 2way-SSL [5] The server must upgrade 
> the connection after encrypted payload is read.
> In case of IIS, it has 2way SSL setting on a Virtual Directory. The 
> initial handshake is not 2way SSL, once the HTTP request is read, it 
> determines that the requested resource (/gsoap) is 2way SSL-protected 
> and upgrades the connection to 2way SSL. [6] The server must kill the 
> connection abruptly when error happens. This is important as if the 
> server terminates the connection gracefully and keeps it around Java 
> is able to recover since it reuses the cached connection. IIS does 
> exactly this. [7] The server must be Windows 7, Win2k8, Win2k8-R2, 
> Win8. Might be other versions affected but bug is not evident on
> Win2k12 or Win2k12-R2 or Win10.
> Perhaps it was fixed, perhaps those versions support TLSv1.2
> 
> It might sound complicated but it is not, it is very common. All you 
> have to do is to have a Virtual Directory in IIS that requires 2way 
> SSL and is set to "want". RSA-based key exchanges are one of the most 
> commonly used ciphers. The Windows platforms we tested with are still 
> widely used in the enterprises while the adoption rate of win2k12+ is 
> still lagging (from our experience with our customers). And you have 
> equal "chance" of cipher support between Windows servers and Java 8 
> clients to land on RSA based cipher (which exhibits the issue) or 
> ECDHE key exchange that is ok. Again, it might sound too complicated 
> but it was blocker for us to ship our product.
> 
> For us, to work around this issue is for the 2way SSL to turn off
> TLSv1.2 (via "-Djdk.tls.client.protocols=TLSv1" or RSA key exchange 
> algorithm (via the java security properties) on affected systems which 
> fixes it.
> 
> I have a lot more technical details that I am not sharing here. I have 
> sent them to Oracle so hopefully that gets resolved, but my guess is 
> that is really an SSPI/SChannel issue in which case someone needs to 
> work with Microsoft and I don't know if my company can do that. It 
> boils down to - when all these conditions are met, is Java creating a 
> bad MAC or SChannel fails to verify that MAC correctly on SSL 
> handshake step X.
> 
> Hopefully this can help someone else, as we have spent quite a bit of 
> time on this issue.
> 
> George
> 
> 
> TL;DR: There is a nasty bug in Java or Windows when calling SSPI-based 
> app over 2way SSL that you can work around by turning off TLSv1.2 or 
> RSA key exchanges.
> 
> -----Original Message----- From: Christopher Schultz 
> [mailto:chris@christopherschultz.net] Sent: Monday, August 31, 2015
> 11:15 AM To: Tomcat Users List Subject: Re: [OT] Tomcat 7.0.55/Jre
> 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
> 
> Diarmuid,
> 
> On 8/31/15 11:36 AM, dmccrthy wrote:
>> To cut a long story short openssl helped. Using openssl -connect 
>> showed the error below. When I added the -tls1 flag the error went 
>> away.
> 
>> 2104:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption
>> failed or bad record mac:.\ssl\s3_pkt.c:532:
> 
>> Configuring Tomcat with the JVM parameter below resolved the issue.
> 
>> -Dhttps.protocols="TLSv1"
> 
> Yep, you have to use a TLS handshake; the older SSL3 handshake won't 
> wor k.
> 
> Now... you *can* enable SSLv2Hello and then only support TLSv1+, but 
> let's face it: SSL is dead and clients should get with the program.
> 
>> This may limit our client Web application but I have very little 
>> influence over the server-side application, so it'll have to do as a 
>> solution for now.
> 
> Your client should not try to use SSL handshake unless you know it's 
> required.
> 
>> So the issue seems to be a Java handshake error whereby it has a 
>> problem downgrading from TLSv1.2 to TLSv1 during the handshake.
> 
> I don't know enough about TLS to know if there is an appreciable 
> difference between the handshake protocol between the two. I know that
> SSLv2 was the lowest common denominator for a while, and most servers 
> would enable it *just for the handshake* but would refuse to actually 
> negotiate an SSLv2 connection (SSLv2 was basically DOA). These days, 
> nobody should be using SSL at all, but I might understand wanting to 
> use SSL-compatible handshakes, so using "SSLv2Hello" but not 
> supporting the SSLv3 protocol would be the way to do that.
> 
> Again, we should just let SSL die. The more we push clients and 
> servers to only negotiate TLS from here on out, the better off 
> Internet security will be overall.
> 
>> Why it works in our non-prod environment with the same versions of 
>> everything is a mystery, but I can speculate at some difference in 
>> underlying O/S settings, or perhaps a Java bug.
> 
> Or more likely a configuration option you didn't detect when you 
> checked everything out.
> 
> -chris
> 
> ---------------------------------------------------------------------
>
> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
>
> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
>
> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJWHQo3AAoJEBzwKT+lPKRYyikQAIrCIi/xUNf6MBzJAGT1zLk0
lmxNVDcve7NrLcESBWXlCZl2iDNrDS1jxqLG8Nv6aZu64Jkz4tRh2b+nGXIXzrhc
NCejO7t4bWqqrucsjjT5SDJn2GCJEz2+usMvWTUXH+YUhJNRuXC/6xAjSP5W3vlO
TffM5U0kiQFHUWfHrcn2kuVWgvD7uA9oSE4vhb7qilypWR+0F0jw6BQOSkzpg9S7
zT3Pe2Mpl5M0au+zqDP9kQTk3E784OAyinfVz7lWsh8hd4ZdFz0cDr3SZSpFaMXI
po2yf2I8lF1ocrlC7fG7PDwtDnBRLVTo/ZB7hYHG9yWvWzp96KzHi+IZ1QfpjozR
oqL4SJFZmREAGZ17ry8AT3spp2L3Axp7gVBX+mkseUhZwAesf18OgPUlgvXaLkG2
yy2BYmiGwfyyItg+ipgH6XmOl0iptOTy8wrg3rtNwlzX21ZrhdRRWprjdDc550jp
Bt+MJw7YiLDyG5F4gSlXe2PI9PrpJ7/Ed2+vW6noJGPaPMnjctksCSH6jl60SUmn
0m9v8jrGxqTV/0ib+fkgIdmJZE7dMPQkvMQG+Skhx0H2MObJeM30ILGFSS8J17hi
W29muAGCuD5cy1PTfivRDxQZf8JHVjZ6wmUC2h4NclTamPClnqPxtXW7AmDwkCzj
Nfynh7OSvHG2bJdaFGYI
=8/jo
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Mime
View raw message