ADUFRAY

BEAST, CRIME, and POODLE… Ugh

18 Oct 2014

Note: see updated article from 2014-DEC-24.

With all the talk about BEAST, CRIME, and POODLE, I thought it was time to revisit the SSL configuration of my blog. My blog is much easier to test with than a production web stack.

If you haven’t familiarized yourself with the excellent Qualys SSL Labs Server Test tool, do so. It is a great resource and will quickly become invaluable part of your toolkit. I’m a bit surprised they didn’t start rate limiting me after so many tests! Here are the results of my testing, which earned this site an A+ rating.

Note: For any of my recommendations to work, you must be using OpenSSL 1.0.1j (openssl-1.0.1e-30.el6_6.2 on Red Hat Enterprise Linux (RHEL) variants) or later. I also had problems with the version of nginx in EPEL, so I updated to nginx-1.6.2-1 in the official repo.

Certificate

This is probably the easiest part. Google and Microsoft have deprecated certificates signed with SHA1 and are very forcefully recommending administrators reissue their certificates using SHA2 for the signature algorithm. This is really simple and only involves creating a new certificate signing request (CSR) and submitting it. HOWEVER, as we’ll find out later in the Key Exchange section, you’ll want to make sure you’re using a 4096-bit or greater key. The default and most common key size is 2048-bit, so now would be a good time to update.

Generate the key:
$ openssl genrsa -out /etc/pki/tls/private/www.example.com.key 4096

Generate the signing request
$ openssl req -new -key /etc/pki/tls/private/www.example.com.key \
              -out /etc/pki/tls/certs/www.example.com.csr -sha512

Submit the CSR to your certificate authority (CA) for signing and you’ve got the certificate part handled. Make a combined certificate file including your server’s certificate and then any intermediate certificates. Do not include the root CA, though — it’s unnecessary and will generate a warning at SSL labs.

Combine the certificates
$ cat www_example_com.crt intermediate1.crt intermediate2.crt > www.example.com.combined.crt

Add these lines to your nginx config

ssl_certificate /etc/pki/tls/certs/www.example.com.combined.crt;
ssl_certificate_key /etc/pki/tls/private/www.example.com.key;

Protocol Support

This is where we start making some tough decisions. SSLv2 is considered completely broken, you should definitely not be using it. SSLv3 is now essentially broken and it is recommended you disable it. However, if you disable SSLv3, you will block access to your site from legacy systems like Windows XP. Depending on your userbase, this may be a dealbreaker for you. If you want to be very agressive, disable all protocols except TLSv1.2 — however, you’ll be limiting yourself quite a bit more at that point.

Here are the relevant configurations.

Maximum accessibility, least security

ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

Recommended, good mix of both accessibility and security

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Aggressive security

ssl_protocols TLSv1.2;

Key Exchange

Key Exchange is impacted both by your certificate’s key as well as the cipher suites we choose later. Choosing specific ciphers and using a strong key, however, is not enough. We also have to generate some random data beforehand to strengthen the key exchange mechanisms. The Diffie-Hellman parameters take an especially long time (about an hour on the new Mac Pro).

Generate the Diffie-Hellman parameters
$ openssl dhparam -out /etc/pki/tls/certs/dhparams-4096.pem 4096

Then add these to your nginx config

ssl_dhparam /etc/pki/tls/certs/dhparams-4096.pem;
ssl_ecdh_curve secp384r1;

The second line strengthens the elliptic curve algorithms that will be used later.

Cipher Strength

Chosing the right ciphers and, more importantly, in the right order, is slightly tedious, but not particularly difficult. I sorted the ciphers by AES method (AES-GCM is better than AES-CBC, for example), AES key length (256-bit only), hash family (SHA384, SHA256, and then SHA). I also preferred ECDHE over DHE for the key exchange (the EC parameters are stronger than the DH parameters). OpenSSL supports a wide variety of cipher suites. I essentially limited myself to 256-bit length keys which filtered the vast majority out. However, if you also include 128-bit AES and Camellia ciphers, you get a list that looks like this:

ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD

ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD

ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384

DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256

ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256

ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA1
ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256) Mac=SHA1

ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA1
ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA1
ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA1
ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1

AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
CAMELLIA256-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(256) Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1

The ones without ECDHE, ECDH, or DH in the beginning to not use ephemeral keys and thus do NOT support perfect forward secrecy. I removed those from my list, but you could also just move them to the bottom.

If we want to be accessible we need to add RC4 cipher suites to the list. (There is a discussion about the relative strength of RC4 vs 3DES, so perhaps this recommendation will need to be updated)

Ideally we would put RC4 ciphers at the bottom of the list, preferring to use our secure cipher suites listed above. Unfortuantely due to a flaw in SSLv3, not prioritizing RC4 above CBC-based algorithms leave us vulnerable to the POODLE attack. Since the verbose cipher list above tells us which protocol a given cipher suite belongs to, we can move any SSLv3 CBC suite below RC4-SHA. Here’s the order I ended up with:

ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD

ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD

ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384

DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256

ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256

ECDH-ECDSA-RC4-SHA      SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=RC4(128)  Mac=SHA1
ECDH-RSA-RC4-SHA        SSLv3 Kx=ECDH/RSA Au=ECDH Enc=RC4(128)  Mac=SHA1
ECDHE-ECDSA-RC4-SHA     SSLv3 Kx=ECDH     Au=ECDSA Enc=RC4(128)  Mac=SHA1
ECDHE-RSA-RC4-SHA       SSLv3 Kx=ECDH     Au=RSA  Enc=RC4(128)  Mac=SHA1

ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA1
ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256) Mac=SHA1

ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA1
ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA1
ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA1
ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1

AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD

RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1

AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
CAMELLIA256-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(256) Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1

Summary

Here are the nginx configurations we have ended up with. Both will get ‘A+’ ratings (if you use Strict Transport Security, optional — ‘A’ otherwise), the latter of the two will get a 100% on all sections.

Modestly secure, while supporting all browsers and SSLv3:

ssl on;

ssl_certificate /etc/pki/tls/certs/www.example.com.combined.crt;
ssl_certificate_key /etc/pki/tls/private/www.example.com.key;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

ssl_ciphers ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:ECDH-ECDSA-AES256-SHA:ECDH-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:ECDH-ECDSA-AES128-SHA:ECDH-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:RC4-SHA:AES256-SHA256:AES128-SHA256:AES256-SHA:CAMELLIA256-SHA:AES128-SHA;

ssl_ecdh_curve secp384r1;

# only enable this if you run an SSL-only site!
add_header Strict-Transport-Security "max-age=31536000";

ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

Aggressively secure, not worrying so much about legacy clients (preffered, but by enabling TLSv1 and TLSv1.1 you can reach many more clients):

ssl on;

ssl_certificate /etc/pki/tls/certs/www.example.com.combined.crt;
ssl_certificate_key /etc/pki/tls/private/www.example.com.key;
ssl_protocols TLSv1.2;
# strong ciphers, 256 only, only with foward secrecy (breaks winxp, java, old android)
ssl_ciphers ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDH-ECDSA-AES256-SHA:ECDH-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-CAMELLIA256-SHA;
ssl_dhparam /etc/pki/tls/certs/dhparam-4096.pem;
ssl_ecdh_curve secp384r1;

# only enable this if you run an SSL-only site!
add_header Strict-Transport-Security "max-age=31536000";

ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

Note: see updated article from 2014-DEC-24.

Setting up Check_MK with Multisite on CentOS 6.5

12 Aug 2014

I’ve been using Nagios for the better part of a decade. It’s an incredibly powerful monitoring platform that’s highly extensible. About 4 or so years ago I discovered an excellent replacement for Nagios’s (outdated) interface, NPCD, and many other Nagios plugins: Check_MK. Check_MK is very light-weight and scales excellently. It can also be made to run entirely over SSH, which makes dealing with corporate firewalls a piece of cake.

Setting up a basic installation is fairly straight-forward and covered thoroughly in Check_MK’s online documentation. I’m not going to bore you with it. What I found less-intuitive was configuring the distriubuted single-pane-of-glass interface known as Multisite. Multisite lets you consolidate several Nagios+Check_MK installations into a single view. Unfortunately, by default it’s all over cleartext and requires you to expose xinetd to the world - gross. Getting Multisite to tunnel over SSH is not difficult, but also not documented. I found several threads in the check_mk mailing list where users asked how to do it, but no one ever had a solution. Here’s mine.

Prerequisites

Start with two CentOS systems, built using the Minimal package group. One system, we’ll call it master.example.com, is going to be your single-pane-of-glass. The second system, slave.example.com, will be your remote site you want to view.

As of CentOS 6, the mod_python package is no longer included in base, which means EPEL is required. If you’re installing this on RHEL 6, don’t forget to subscribe to the rhel-x86_64-server-optional-6 repo.

# curl -O http://[mirror]/fedora/epel/6/i386/epel-release-6-8.noarch.rpm
# yum localinstall epel-release-6-8.noarch.rpm

Next, let’s install the required packages.

# yum install -y gcc gcc-c++ man make httpd gd-devel perl wget   \
                 samba-client postgresql-devel openssh-clients   \
                 openldap-devel net-snmp net-snmp-utils          \
                 bind-utils mysql mysql-devel rpcbind mod_python \
                 mod_ssl php rrdtool-perl perl-Time-HiRes php-gd

If you want Perl Net::SNMP checks, RADIUS checks, and fping, you’ll also need to install these optional packages:

# yum install -y perl-Net-SNMP radiusclient-ng-devel fping

Monitoring Software Installation

Once you’ve got all the required software, go ahead and download the source packages for Nagios, nagios-plugins, check_mk, and pnp4nagios:

nagios-3.5.1.tar.gz - Skip to download

nagios-plugins-2.0.3.tar.gz - Handy to have

check_mk-1.2.5i5p2.tar.gz - Live dangerously: get the innovation release

pnp4nagios-0.6.24.tar.gz - Pretty graphs

I configure each in the above order, and I install them all into /usr/local/, which is probably a holdover from my long-time romance with FreeBSD. We’ll start with Nagios:

First create the Nagios user. For some reason this has been broken in the source package for several years, and no one has bothered to fix it.

# useradd -r -d /var/log/nagios -s /bin/sh -G apache -c "nagios" nagios

Then the standard unpack, configure, and make commands, plus a bunch of extra makes:

# tar -zxvf nagios-3.5.1.tar.gz
# cd nagios
# ./configure
# make all
# make install
# make install-init
# make install-commandmode
# make install-config
# make install-webconf
# make install-exfoliation

Nagios-plugins is pretty simple, just watch out for any plugins that get skipped due to dependencies, just in case you actually want them.

# tar -zxvf nagios-plugins-2.0.3.tar.gz
# cd nagios-plugins-2.0.3
# ./configure
# make
# make install

Check_MK is a little bit different. It comes with a setup.sh script that walks you through the configuration directories, then compiles itself and performs the install. Here are the answers I use to get everything installed under /usr/local/check_mk/:

# tar -zxvf check_mk-1.2.5i5p2.tar.gz
# cd check_mk-1.2.5i5p2
# ./setup.sh

Executable programs             /usr/local/bin
Check_MK configuration          /usr/local/check_mk/etc
Check_MK software               /usr/local/check_mk
documentation                   /usr/local/check_mk/doc
check manuals                   /usr/local/check_mk/doc/checks
working directory of Check_MK   /usr/local/check_mk/var/lib
extensions for agents           /usr/local/check_mk
configuration dir for agents    /usr/local/check_mk/etc
Name of Nagios user             nagios
User of Apache process          apache
Common group of Nagios+Apache   nagios
Nagios binary                   /usr/local/nagios/bin/nagios
Nagios main configuration file  /usr/local/nagios/etc/nagios.cfg
Nagios object directory         /usr/local/nagios/etc/check_mk.d
Nagios startskript              /etc/init.d/nagios
Nagios command pipe             /usr/local/nagios/var/rw/nagios.cmd
Check results directory         /usr/local/nagios/var/spool/checkresults
Nagios status file              /usr/local/nagios/var/status.dat
Path to check_icmp              /usr/local/nagios/libexec/check_icmp
URL Prefix for Web addons       /[SITE NAME]/    !! CHANGE THIS TO YOUR SITE NAME !!
Apache config dir               /etc/httpd/conf.d
HTTP authentication file        /usr/local/nagios/etc/htpasswd.users
HTTP AuthName                   Nagios Access
PNP4Nagios templates            /usr/local/pnp4nagios/share/templates
RRD files                       /usr/local/check_mk/pnp-rraconf
rrdcached socket                /tmp/rrdcached.sock
compile livestatus module       yes
Nagios / Icinga version         3.5.1
check_mk's binary modules       /usr/local/check_mk/lib
Unix socket for Livestatus      /usr/local/nagios/var/rw/live
Backends for other systems      /usr/local/check_mk/share/livestatus
Install Event Console           no

Pay very close attention to the URL Prefix for Web addons configuration value. This is going to be the key value for your site name in later parts of the Multisite configuration. To keep with my example, my two installations will use master and slave as the values here.

Lastly, we set up PNP4Nagios, which gives us the awesome RRD graphs for all our services. It also users the same site prefix, so be mindful:

# tar -zxvf pnp4nagios-0.6.24.tar.gz
# cd pnp4nagios-0.6.19
# ./configure --with-base-url=/[SITE NAME]/pnp4nagios
# make all
# make fullinstall

Configuration Files

Now, let’s get configuring! For brevity, I’ve summarized my changes below:

/usr/local/nagios/etc/nagios.cfg:
    # Comment out the localhost config:
    #cfg_file=/usr/local/nagios/etc/objects/localhost.cfg

    cfg_dir=/usr/local/nagios/etc/check_mk.d
    broker_module=/usr/local/check_mk/lib/livestatus.o /usr/local/nagios/var/rw/live
    broker_module=/usr/local/pnp4nagios/lib/npcdmod.o config_file=/usr/local/pnp4nagios/etc/npcd.cfg
    use_syslog=0
    check_for_updates=0
    process_performance_data=1
    admin_email=root@localhost
    admin_pager=root@localhost

You’ll have to read Check_MK’s online documentation to understand what each option is doing, but this is a pretty good starter config. Each system will monitor hosts only accessible by it. For example, if your master site was local to you in San Francisco, it might monitor all your California resources and not have access to resources in your remote office branch in New York (thus the slave server).

Check_MK supports all kinds of configuration options, most of which rely on the tags defined in the all_hosts variable. You can set a different data-aquisition method for all kinds of systems (in my example, executing the local Check_MK agent directly & remotely via SSH). You could also use SNMP or UNIX sockets.

/usr/local/check_mk/etc/main.mk:
    all_hosts = [ 
      'master.example.com|local',
      'host-monitored-by-master.example.com|ssh',
    ]

    datasource_programs = [
      ( "/usr/bin/sudo /usr/local/check_mk/agents/check_mk_agent.linux", [ 'local' ], ALL_HOSTS ),
      ( "ssh -i ~nagios/.ssh/id_rsa nagios@<IP> sudo /usr/local/bin/check_mk_agent", [ 'ssh' ], ALL_HOSTS ),
    ]

    ipaddresses = {
      "master.example.com" : "127.0.0.1",
    }

    extra_service_conf["normal_check_interval"] = [
      ( '5', ALL_HOSTS, [ "" ] ),
    ]

    extra_host_conf["max_check_attempts"] = [
      ( '3', ALL_HOSTS ),
    ]

System Configuration and Cleanup

That’s basically it for configuration files to get started, now let’s configure the systems themselves.

Enable services to start at boot:

# chkconfig httpd on
# chkconfig nagios on
# chkconfig npcd on

Fix various SELinux (read: disable) and permissions:

# mkdir /usr/local/check_mk/var/lib/web/admin
# chown apache:nagios /usr/local/check_mk/var/lib/web/admin
# chmod 770 /usr/local/check_mk/var/lib/web/admin
# setenforce 0
# echo 'SELINUX=permissive' >> /etc/sysconfig/selinux

Add some sudo permissions for the nagios user:

# echo 'Defaults:nagios !requiretty' >> /etc/sudoers
# echo 'nagios ALL = (root) NOPASSWD: /usr/local/check_mk/agents/check_mk_agent.linux' >> /etc/sudoers

Create an inventory of services to monitor on our systems:

# check_mk -I master.example.com host-monitored-by-master.example.com

Rebuild and restart Nagios / Check_MK

# check_mk -R

Start the other services:

# service httpd start
# service npcd start

Create an HTTPD user and password (use the same htpasswd file on both systems):

# htpasswd -c -s /usr/local/nagios/etc/htpasswd.users nagiosadmin

Now to complete the installation of PNP4nagios, open http://master.example.com/[SITE NAME]/pnp4nagios/ in your browser, then remove or rename the installation script:

# mv /usr/local/pnp4nagios/share/install.php{,.orig}

Multisite Configuration

Once you repeat the above steps for the slave.example.com server, you’re ready to configure Multisite. You’ll want to create an SSH public/private keypair for the nagios user. This keypair will be used to establish the SSH tunnel, but nothing else. We’ll lock it down to keep things safe. The keypair should only reside on master.example.com, and we’ll copy the public key into slave.example.com’s authorized_keys file.

Master:

# su - nagios
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen -t rsa -b 4096 -f ./id_rsa
$ chmod 400 id_rsa
$ chmod 444 id_rsa.pub

Slave:

# su - nagios
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ vi authorized_keys
    command="exit",no-pty,permitopen="localhost:80",permitopen="localhost:6557",permitopen="localhost:2000",permitopen="localhost:2001" ssh-rsa AAAAAAA..long-key..ZZZZZ

The options leading the authorized_keys file keeps the nagios user from being able to do pretty much anything except get our host data and forward a few ports. The only command it can run is “exit”, it can’t open a psuedo-terminal, and it can only forward the ports we’ve specified. Ports 80 (HTTP), 6557 (check_mk), 2000 & 2001 (SSH tunnel status checks).

We’re going to start by setting up xinetd on slave.example.com — don’t worry, it’ll only listen on localhost.

# yum install -y xinetd
# cat << 'END' > /etc/xinetd.d/livestatus
    service livestatus {
      bind            = 127.0.0.1
      type            = UNLISTED
      port            = 6557
      socket_type     = stream
      protocol        = tcp
      wait            = no
      cps             = 100 3
      instances       = 500
      per_source      = 250
      flags           = NODELAY
      user            = nagios
      server          = /usr/local/bin/unixcat
      server_args     = /usr/local/nagios/var/rw/live
      only_from       = 127.0.0.1 ::1
      disable         = no
      log_type        = SYSLOG daemon info
    }
  END 
# chkconfig xinetd on
# service xinetd start

Now on master.example.com we’re going to setup autossh to establish and maintain the SSH tunnel.

# yum install -y autossh
# su - nagios
$ cat << 'END' > ~/autossh.bash
    #!/bin/bash

    autossh -f -M 2000 -i ~nagios/.ssh/id_rsa  -L 8081:localhost:80 -L 6558:localhost:6557 -N nagios@slave.example.com
  END
$ chmod a+x autossh.bash
$ crontab -e
    @reboot bash ~/autossh.bash
$ ./autossh.bash

Lastly we need to tell Multisite about our other site and set up HTTPD to proxy the requests through our tunnel.

# vi /usr/local/check_mk/etc/multisite.mk
   sites = {
     "master" : {
       "alias" : "Master Site"
     },
     "slave": {
       "alias" : "Slave Site",
       "socket": "tcp:localhost:6558",
       "url_prefix": "/slave/",
     },
   }

# vi /etc/httpd/conf/httpd.conf
   <Location /slave>
       RewriteEngine On
       RewriteRule ^/.+/slave/(.*) http://localhost:8081/slave/$1 [P]
   </Location>

Go ahead and restart Nagios/Check_MK and HTTPD, and you should be all set.

# check_mk -R
# service httpd restart

If all went according to plan, you should be able to go to http://master.example.com/master/check_mk/ and see the systems monitored by both the master and slave servers! You should probably go ahead and configure HTTPD to use SSL, as well as configure iptables or another suitable software/hardware filewall to limit traffic appropriately.

Canon, uniFLOW, and RightFax: Oh My!

17 Jul 2014

My organization recently purchased around 100 high end Canon multi-function printers, along with a print queue management suite called uniFLOW. uniFLOW gives us a single, roaming print queue, secure print (badge required to get your print job!), and excellent print accounting metrics. With the aforementioned badge integration, we know exactly who is printing all those color copies of their fantasy team’s roster.

It is difficult to believe that here in 2014 organizations are still required to support faxing, but here we are. Even as part of an all new printer deployment, I find myself spending an inordinate amount of time configuring, troubleshooting, and supporting FAX.

The Canon multi-function printers with uniFLOW support simple scan to e-mail and copy functions, but fax is slightly more complicated. It’s less significantly less intuitive, for starters. There is no fax icon or menu action on the large touch screen display. Instead, you have to use the Scan to E-mail process. At which point, you e-mail the phone number you’d like to fax, and hit send. Easy enough to remember, but getting the back end configuration right to make it work is another challege. Here’s my configuration, in case someone else has the same difficulty.

First, make sure you’re running a recent version of RightFax. RightFax is software that allows an organization to deliver faxes to e-mail, obviating the need for every employee to have a personal fax machine — and most importantly, vice-versa: you can deliver faxes from e-mail to a fax number. Your organization may use something different, but I imagine the set up will be similar.

It took a bit of testing with RightFax’s Exchange connector and Wireshark to determine the correct To: header syntax, but here it is:

IMCEARFAX-Walk+20up+40_FN=AAABBBCCCC@domain.com

RightFax requires a recipient name and obviously a fax number destination. The above formatting includes both, broken down as follows:

IMCEARFAX-               RightFax prefix
Walk+20up                Recipient name - +20 being the hexadecimal ASCII code for a space
+40                      ASCII for the at-symbol
_FN=AAABBBCCCC           Fax number equals AAA-BBB-CCC

Now that we know the syntax, we need a way to translate a string of only numbers into that address — it would be an onerus requirement for an end-user to have to type in that ridiculously long e-mail address every time they wanted to send a fax. Thankfully, uniFLOW gives us everything we need to be successful.

Create an XML file on the uniFLOW server, e.g. C:\fax-to-email.xml:

<MOMEMAILCONVERSION>
  <RULE NAME="SENDER">%o</RULE>
  <RULE NAME="RECIPIENT">IMCEARFAX-Walk+20up+40_FN=/{\d+}/@FAXSERVER.domain.com</RULE>
  <RULE NAME="BODY">%o</RULE>
</MOMEMAILCONVERSION>

Edit the FAXSERVER.domain.com to be the FQDN of your RightFax server, then go into the uniFLOW server configuration interface, navigate to Printer -> Printer -> (The printer you want to configure) -> Next -> Device Agents tab. Scroll down to the bottom section titled Other / MIND SMTP Control. Here you can configure how the printer will handle different destination addresses. Set the Email Conversions field value to:

^{\d+}$:C:\fax-to-email.xml

Click Save, and then go send a fax!

Setting up jabberd2 on RHEL

31 May 2013

Setting up a simple Jabber service on Red Hat Enterprise Linux (or any of its derivatives) is very straight-forward. If you install the Extra Packages for Enterprise Linux (EPEL) repository, you’ll have access to jabberd2.

The Quick StartGuide for RPM documentation provided by the jabberd2 project is almost all you need to get it up and going.

# yum install jabberd
# chkconfig jabberd on

The documentation tells you to switch to MySQL, but this is unnecessary. The default configuration uses an SQLite database for user registration and session storage, so you can set that up instead. To do so, simply initialize the database:

# sqlite3 /var/lib/jabberd/db/sqlite.db < /usr/share/jabberd/db-setup.sqlite

Then all that’s left is setting up your ID entries in c2s.xml and sm.xml:

c2s.xml:
    <local>
      <id register-enable='mu' password-change='mu'>domain.com</id>
      ...
    </local>

sm.xml:
    <local>
      <id>domain.com</id>
      ...
    </local>

Now you’re all set:

# service jabberd start

This, of course, only sets up the most basic implementation. There are a few extra steps I took to have a slightly more secure setup. I also wanted to host multiple domains for friends to use. By default:

To fix the encrypted password issue, you have to use MySQL instead of SQLite. There’s an open bug on GitHub to implement this feature, but it’s still waiting for patches. It’s pretty easy to change to MySQL, though. The database setup script, /usr/share/jabberd/db-setup.mysql, wants to create the database for you, which can cause problems if you’ve already created one, so go ahead and comment out the first few lines:

-- CREATE DATABASE jabberd2;
-- USE jabberd2;

Then you can create the database, its user, and run the script:

# mysql -u root -p
> CREATE DATABASE jabber;
> GRANT ALL ON jabber.* to 'jabber'@'localhost' IDENTIFIED BY 'password';
> QUIT
# mysql -u jabber -p jabber < /usr/share/jabberd/db-setup.mysql

Then edit both c2s.xml and sm.xml again, changing the DB driver to use MySQL instead of SQLite:

c2s.xml:
    <authreg>
      ...
      <module>mysql</module>
      ...
      <mysql>
        <host>localhost</host>
        <port>3306</port>

        <dbname>jabber</dbname>

        <user>jabber</user>
        <pass>password</pass>

        <password_type>
          <crypt/>
        </password_type>
      </mysql>
      ...
    </authreg>

sm.xml:
    <storage>
      ...
      <driver>mysql</driver>
      ...
      <mysql>
        <host>localhost</host>
        <port>3306</port>

        <dbname>jabber</dbname>

        <user>jabber</user>
        <pass>password</pass>

        <transactions/>
      </mysql>
      ...
    </storage>

This gives us a strong database backend with passwords that are somewhat securely stored. The hashing method uses the crypt(3) function to generate a salted MD5 hash. Without digging into the source code, I can’t tell how many rounds of MD5 it is, but it’s sort of irrelevant: MD5 is broken and unsuited for password hashing. Still, better than plaintext — basically only keeps honest people honest. Moving on…

To set up TLS/SSL you have a decision to make: self-signed or legitimate certificate. I prefer legitimate because these days you can get SSL certificates very cheaply (or even free). It’s a bit easier to set up self-signed though:

# openssl genrsa -out ./jabber.key 2048
# openssl req -new -key ./jabber.key -x509 -sha1 -out ./jabber.crt
# cat jabber.crt jabber.key >> combined.crt
# rm jabber.crt jabber.key

jabberd2 requires that they key (and any intermediate certificates) be present in the PEM file. I don’t like this approach — the key should not be readable by anyone but root. Unfortunately the jabberd2 processes aren’t clever enough read in the key as root and then drop privileges — patches are welcome, I’m sure.

If you want to go the legitimate certificate route, create a key and certificate signing request:

# openssl genrsa -out ./jabber.key 2048
# openssl req -new -key ./jabber.key -sha1 -out ./jabber.csr

Send the CSR to your certificate authority. They should send you back a certificate and one or more intermediate / root certificates. Your combined certificate should look something like this:

# cat certificate.crt intermediate.crt root.crt key.key > combined.crt

To install the SSL certificate (either self-signed or legitimate), you need to edit the <id> tag in c2s.xml:

<local>
  <id register-enable='mu' pemfile='/path/to/combined.crt' require-starttls='mu' password-change='mu'>domain.com</id>
  ...
</local>

My server requires all client connections to be encrypted. If you want to support TLS, but don’t care whether or not the connections are encrypted, just remove the require-starttls parameter.

Next up is setting the IP address to listen on. This is a very quick edit to c2s.xml and s2s.xml:

<local>
  ...
  <ip>0.0.0.0</ip>
  ...
</local>

Adding an administrator is similarly easy. Just edit the aci section of sm.xml:

<aci>
  ...
  <acl type='all'>
    <jid>admin@domain.com</jid>
  <acl>
  ...
</aci>

Securing the inter-service communication is fairly easy, but a little cumbersome. You need to edit all the service configuration files (c2s.xml, s2s.xml, and sm.xml) as well as the router configuration (router.xml) and the router users file (router-users.xml).

For each service, create a user entry in router-users.xml:

<users>
  <user>
    <name>c2s</name>
    <secret>random_password</secret>
  </user>
  <user>
    <name>s2s</name>
    <secret>random_password</secret>
  </user>
  <user>
    <name>sm</name>
    <secret>random_password</secret>
  </user>
</users>

Then edit router.xml to give access to each of these users:

<aci>
  <acl type='all'>
    <user>c2s</user>
    <user>s2s</user>
    <user>sm</user>
  </acl>
  ...
</aci>

While you’re there, go ahead and enable SSL:

<local>
  ...
  <pemfile>/path/to/combined.crt</pemfile>
  ...
</local>

You could probably be more granular in the ACLs, but I don’t know enough about what permissions are needed for each process. Next, you have to edit each service’s configuration file and change the user & password settings — make sure you match the passwords correctly to what you defined in router-users.xml:

<router>
  ...
  <user>c2s OR s2s OR sm</user>
  <pass>corresponding password</pass>
  <pemfile>/path/to/combined.crt</pemfile>
  ...
</router>

Now all your services should be using a unique password to communicate. In addition, the traffic will be encrypted.

All that remains is to set up multiple domains. Searching around the net I couldn’t find very much documentation, but thankfully it’s very easy. In previous versions of jabberd2, you needed to run an sm process for each separate domain. That no longer seems to be the case. You just need to edit c2s.xml, sm.xml, and configure appropriate SRV records. In c2s.xml, just add another ID entry for the second domain:

<local>
  ...
  <id register-enable='mu' pemfile='/path/to/combined.crt' require-starttls='mu' password-change='mu'>domain1.com</id>
  <id register-enable='mu' pemfile='/path/to/combined.crt' require-starttls='mu' password-change='mu'>domain2.com</id>
  ...
</local>

Pretty much the same thing in sm.xml:

<local>
  <id>domain1.com</id>
  <id>domain2.com</id>
</local>

Then add the SRV records to your DNS:

_xmpp-client._tcp.domain1.com. IN    SRV 5    5 5222 domain1.com.
_xmpp-server._tcp.domain1.com. IN    SRV 5    5 5269 domain1.com.

Do the same thing for domain2.com, leaving domain1.com as the destination at the end:

_xmpp-client._tcp.domain2.com. IN    SRV 5    5 5222 domain1.com.
_xmpp-server._tcp.domain2.com. IN    SRV 5    5 5269 domain1.com.

This configuration will let you use Jabber IDs for either domain1.com or domain2.com. Just make sure in your client’s configuration you put in the correct value for the Connect Server — in this example, that would be domain1.com.

Once you’ve created your accounts using a Jabber client, you might consider going back to c2s.xml and removing the register-enable parameter from the id tag. Otherwise any Internet user can create accounts on your server.

Link: The Barman by John Gallagher

11 May 2013

Totally sponsoring this. Can’t wait to program in all my complicated drinks so my wife can make them for me!