Monitoring Fails Due to SSL Handshake or Authentication Failures

  • kb1186
  • Nov. 3, 2016 - Last Update: Jun. 27, 2018
  • 2 min read

What to do if monitoring your systems with BMC TrueSight Capacity Optimization or BMC PATROL/TrueSight Operations Management fails due to SSL handshake.

Troubleshooting Java SSL Handshake Close_notify

Related Topics

Problem

Because manufacturers and Java increased the security level of SSL connections, the monitoring of your systems may fail due to a SSL handshake failure or if your certificates do not comply with the java security constraints. The following errors will occur:

  • EXCEPTION CertificateException : Certificates does not conform to algorithm constraints
  • Error: Received fatal alert: close_notify
  • Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Solution

When using Java 1.6

Java 1.6 sends a SSLv2 Hello message during the SSL negotiation (“handshake”). Because this message is not supported by most secured services, the handshake will fail. To solve this issue, upgrade to the latest version of java available.

When using Java 1.8 or higher

  1. Use the testssl.sh utility or any other SSL scanner to test the SSL encryption and verify the server preferences. The command to be run is ./testssl.sh <hostname>:<port>.

    If you are testing the connection against:

    • an SMI-S provider: Use port 5989
    • an HTTP API: Use port 443.
    In the example below, we tested the SSL encryption of a NetApp Filer configured in 7-mode and with Data ONTAP API v7.3.3 installed:  
![Running the testssl.sh utility to test the SSL encryption of a NetApp Filer configured in 7-mode](/kb/1186/testssl.png)

    As you can see in the screenshot above: * the negotiated protocol is TLSv1 * the server key size is 512 bit * the signature algorithm is MD5.

  2. Open the java.security file to verify that the server preferences comply with the current java constraints. The java.security file is to be found in:

    • opt/bmc/BCO/jre/lib/security/ (for BMC TrueSight Capacity Optimization)
    • %JAVA_home%/lib/security (for BMC PATROL/BMC TrueSight Operations Management while using java 1.8)
    • %JAVA_home%/conf/security (for BMC PATROL/BMC TrueSight Operations Management while using java 9 and higher)

  3. Search for the following lines:

  • jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

  • jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768

  • jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

    Which means that we will face a handshake failure since our signature algorithm (MD5) is disabled and our server key size is lower than 1024.

  1. If the server preferences do not match the java security constraints, update the java.security file accordingly and save your changes. In our example, we will remove MD5 from the jdk.certpath.disabledAlgorithms and jdk.jar.disabledAlgorithms lines and set the key size to 511.

In this case, we are adapting the java constraints to a specific API. You may have to modify them again if another certificate requires lower properties. You can disable those constraints by commenting them out using # symbol.

  1. If you are using:
    • BMC TrueSight Capacity Optimization, restart the scheduler
    • BMC PATROL/TrueSight Operations Management, restart the PATROL agent.
Troubleshooting Java SSL Handshake Close_notify