I ran into a fun problem recently with Zabbix and the zabbixapi gem. During puppet runs, each puppetdb record for a Zabbix_host resource is pushed through the zabbixapi, to create or update the host in the Zabbix system. When this happened, an interesting error crops up:
Error: /Stage[main]/Zabbix::Resources::Web/Zabbix_host[kickstart.example.com]: Could not evaluate: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
If you google for that, you’ll find a lot of different errors and causes described across a host of systems. Puppet itself is one of those systems, but it’s not the only one. All of the systems have something in common: Ruby. What they rarely have is actual resolution, though. Possible causes include time out of sync between nodes, errors with the certificates and stores on the client or server side, and of course a bunch of “it works now!” with no explanation what changed. To confuse matters even more, the Zabbix web interface works just fine in the latest browsers, so the SSL issue seems restricted to zabbixapi.
To find the cause, we looked at recent changes. The apache SSLProtocols were changed recently, which shows up in a previous puppet run’s output:
- SSLProtocol ALL -SSLv2 -SSLv3 + SSLProtocol ALL -SSLv2 -SSLv3 -TLSV1
The TLSv1 protocol was disabled. This is generally speaking a recommended practice – there are numerous attacks against SSL3/TLS1.0 and it will no longer allowed in PCI-DSS3.0 after June 30, 2016 (page 2). While the monitoring system generally does not fall within the PCI-DSS scope, it is common to enforce the same base apache configurations throughout the system. Unfortunately, we need Zabbix to work! This clearly looks like an issue with the Ruby gem and its ability to communicate with the HTTPS server using the right protocol.
I found a Stack Overflow question where someone asked How to set TLS context options in Ruby (like OpenSSL::SSL::SSL_OP_NO_SSLv2). The top answer includes a command to run in the interactive ruby shell, irb, that enumerates the options available in Ruby’s own net/https module. I ran this command on the Zabbix server and rearranged the results order slightly to put the SSL Protocols at the front:
irb(main):001:0> require 'net/https' => true irb(main):002:0> OpenSSL::SSL.constants.grep(/OP_/) => ["OP_NO_SSLv3", "OP_NO_SSLv2", "OP_NO_TLSv1", "OP_TLS_D5_BUG", "OP_SSLREF2_REUSE_CERT_TYPE_BUG", "OP_EPHEMERAL_RSA", "OP_ALL", "OP_PKCS1_CHECK_1", "OP_SSLEAY_080_CLIENT_DH_BUG", "OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG", "OP_SINGLE_DH_USE", "OP_NETSCAPE_DEMO_CIPHER_CHANGE_BUG", "OP_DONT_INSERT_EMPTY_FRAGMENTS", "OP_NO_TICKET", "OP_MSIE_SSLV2_RSA_PADDING", "OP_TLS_ROLLBACK_BUG", "OP_NETSCAPE_CHALLENGE_BUG", "OP_SINGLE_ECDH_USE", "OP_NETSCAPE_CA_DN_BUG", "OP_TLS_BLOCK_PADDING_BUG", "OP_MICROSOFT_BIG_SSLV3_BUFFER", "OP_CIPHER_SERVER_PREFERENCE", "OP_MICROSOFT_SESS_ID_BUG", "OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION", "OP_PKCS1_CHECK_2"]
If SSLv2, SSLv3, and TLSv1 are disabled, there are no more protocols available to talk with the server. How can that be? TLS v1.1 and 1.2 are not that new. Unfortunately, the system ruby is not new either!
$ ruby -v ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
The Zabbix server is running on EL6, which has a positively ancient version of Ruby installed that does not have TLSv1.1 and 1.2 support. That seems like a simple cause! Let’s look at EL7, which includes Ruby 2.0.0, and see how it fares:
irb(main):001:0> require 'net/https' => true irb(main):002:0> OpenSSL::SSL.constants.grep(/OP_/) => [:OP_NO_SSLv2, :OP_NO_SSLv3, :OP_NO_TLSv1, :OP_NO_TLSv1_1, :OP_NO_TLSv1_2, :OP_MICROSOFT_SESS_ID_BUG, :OP_NETSCAPE_CHALLENGE_BUG, :OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG, :OP_SSLREF2_REUSE_CERT_TYPE_BUG, :OP_MICROSOFT_BIG_SSLV3_BUFFER, :OP_MSIE_SSLV2_RSA_PADDING, :OP_SSLEAY_080_CLIENT_DH_BUG, :OP_TLS_D5_BUG, :OP_TLS_BLOCK_PADDING_BUG, :OP_DONT_INSERT_EMPTY_FRAGMENTS, :OP_ALL, :OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION, :OP_SINGLE_ECDH_USE, :OP_SINGLE_DH_USE, :OP_EPHEMERAL_RSA, :OP_CIPHER_SERVER_PREFERENCE, :OP_TLS_ROLLBACK_BUG, :OP_NO_TICKET, :OP_NO_COMPRESSION, :OP_PKCS1_CHECK_1, :OP_PKCS1_CHECK_2, :OP_NETSCAPE_CA_DN_BUG, :OP_NETSCAPE_DEMO_CIPHER_CHANGE_BUG]
A test installation of Zabbix on EL7 with the same Apache SSL settings has no issues. The zabbixapi gem is able to use TLSv1.1 or 1.2 to communicate with the apache server and no errors are generated.
The fix is relatively easy, and isn’t in Zabbix or the zabbixapi ruby gem: Upgrade to a newer system Ruby. I could do that with the EL6 SCL version of Ruby or by redeploying Zabbix on an EL7 node, or switching to another distribution entirely. That’s much more helpful than bogus concerns about time synchronization and system certs, and I hope it helps anyone else struggling with a similar issue.