Symptom: Your Zimbra server certificate won't update from e.g LETSENCRYPT.
Cause/repairs:
- Expired IdenTrust root certificate
- Certbot no longer support your OS
This article is for Zimbra server admins. In case you use an older but well fenced off server OS (like CentOS 6.6) together with Letsencrypt and Certbot you might hit two problems:
IdenTrust root cert expired.
In Zimbra the root certificate is added to the chain like
cat /etc/letsencrypt/live/[ServerName]/chain.pem identrust.pem >zimbrachain.pem
(identrust.pem is here the previously downloaded root cert). Then zimbrachain.pem is later handed to zmcertmgr for verification and deployment. You can see that it has expired:
# openssl verify identrust.pem
identrust.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 0 depth lookup:certificate has expired
OK
You might think you can download a new root cert from Identrust and all would be well. Nope.
Letsencrypt now hands out their own root certificate though. You can find it here: isgrootx1.pem. Replacing the original identrust certificate with the new one won't work either though, but you'll need it later.
Certbot no longer support your OS.
All this work trying to get Certbot to run again turned out to be futile, but I'll walk trough the steps anyway because it also show how to do the long needed update of the system.
Running certbot manually might produce some strange error.
/usr/local/bin/certbot-auto renew
WARNING: couldn't find Python 3.5+ to check for updates.
Bootstrapping dependencies for Legacy RedHat-based OSes that will use Python3... (you can skip this with --no-bootstrap)
yum is hashed (/usr/bin/yum)
To use Certbot on this operating system, packages from the SCL repository need to be installed.
Enable the SCL repository and try running Certbot again.
Ok, so actually we'll need Python3 to do the bootstrapping. What Python version and OS do I have now again?
#python3 --version
Python 3.4.8
#cat /etc/system-release
CentOS release 6.6 (Final)
Certbot needs at least Python3.5, so I'll have to update. Trying to install centos-release-scl:
yum install centos-release-scl
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
Ouch, something is wrong with yum. Turned out the solution is simple; download centos6-eol.repo (I didn't use wget but had to scp it to the server) and copy it to /etc/yum.repos.d as CentOS-Base.repo:
#cp centos6-eol.repo /etc/yum.repos.d/CentOS-Base.repo
# yum update
You'll be presented with a scary list of things to update / install. Do it. Then, to make sure, run yum update again. After that you should be able to run
# yum install centos-release-scl
You might get
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/centos-sclo-rh/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: centos-sclo-rh
In which case you might have to edit relevant files in /etc/yum.repos.d, the sections
edit [centos-sclo-rh]
baseurl=http://vault.centos.org/centos/6.10/sclo/$basearch/rh/
edit [centos-sclo-sclo]
baseurl=http://vault.centos.org/centos/6.10/sclo/$basearch/sclo/
All right. When yum update works again, and the scl repo is installed, you should be able to install Python36:
# yum install rh-python36
# scl enable rh-python36 bash
# python3 --version
Python 3.6.12
Yes! Now lets run certbot. Should be happy now, yes?
/usr/local/bin/certbot-auto renew
Bootstrapping dependencies for Legacy RedHat-based OSes that will use Python3... (you can skip this with --no-bootstrap)
yum is /usr/bin/yum
... blabla....Upgrading certbot-auto 1.3.0 to 1.20.0...
Replacing certbot-auto...
Your system is not supported by certbot-auto anymore.
Certbot cannot be installed.
Please visit https://certbot.eff.org/ to check for other alternatives.
Fsck! All in vain. Certbot won't work on CentOS 6.6. I actually tried to downgrade certbot but that didn't work either.
Installing acme.sh
So I found the acme.sh script. This is not doing all the invasive bootstrapping as certbot do, and doesnt have messy Python dependencies. I installed as root, and it was fairly simple
# sudo -s
# cd ~
# curl https://get.acme.sh | sh -s email="This email address is being protected from spambots. You need JavaScript enabled to view it."
... blabla...
# cd ~/.acme.sh
# ./acme.sh --set-default-ca --server letsencrypt
# ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt # yum install socat
The last line is to ensure socat is installed, because acme.sh needs it in standalone mode. That was the first part of the installation.
But this is Zimbra, so we'll have to do some additional tap-dancing to deploy the certificate once it's downloaded. There are some deploy scripts installed together with acme.sh, but none for Zimbra.
First, we need a deploy script and a deploy hook. The script is to be placed in the ".acme.sh/deploy" folder, and should end in ".sh" (though when specified on the command line the extension should be left out).
Save this as ~/.acme.sh/deploy/zimbra.sh (edit the relevant fields "my.server.domain"):
#!/bin/bash
# acme.sh, run as root, standalone (Zimbra need to be stopped to use port 80).
# Install:
# yum install socat
# isrg root X1 need to be downloaded separately (once):
# wget -O /root/.acme.sh/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem.txt
# curl https://get.acme.sh | sh -s email="root@my.server.domain"
# cd ~/.acme.sh
# ./acme.sh --set-default-ca --server letsencrypt
# ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt
#
# I removed the pre-installed crontab for root:
# crontab -r
#
# Manual run:
# To save --pre-hook:
# ./acme.sh --issue --pre-hook /root/.acme.sh/zimbra-stop.sh --standalone -d my.server.domain --force
#
# To deploy and save --deploy-hook:
# ./acme.sh --deploy --deploy-hook zimbra -d my.server.domain # ######## Public functions #####################
function mail_ok () {
/opt/zimbra/postfix/sbin/sendmail -froot@my.server.domain -F'root@my.server.domain' -t <<EOF
To: root@my.server.domain
Subject: Zimbra mail server certificate update $(date +%B)
Zimbra mail server certificate has been renewed.
$(cat lasterr.txt)
EOF
}
function mail_err () {
/opt/zimbra/postfix/sbin/sendmail -fThis email address is being protected from spambots. You need JavaScript enabled to view it. -F'root@my.server.domain' -t <<EOF
To: root@my.server.domain
Subject: Zimbra mail server certificate update $(date +%B) failed
Zimbra mail server certificate update error:
$*
$(cat lasterr.txt)
EOF
}
#domain keyfile certfile cafile fullchain
zimbra_deploy() {
_cdomain="$1"
_ckey="$2"
_ccert="$3"
_cca="$4"
_cfullchain="$5"
#isrg root X1 need to be downloaded separately (once)
# wget -O /root/.acme.sh/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem.txt
ISG_X1="$(dirname "$_cca")/../isrgrootx1.pem"
# Zimbra jetty is stopped when entering this, because acme.sh is run in standalone and need port 80.
# append root pem so verifycrt can walk the chain
cat "$_cfullchain" "$ISG_X1" > "${_cca}.real"
chown zimbra:zimbra "$_ckey" "$_ccert" "${_cca}.real"
_debug "Verifying..."
/opt/zimbra/bin/zmcertmgr verifycrt comm "$_ckey" "$_ccert" "${_cca}.real" > lasterr.txt || {
echo "Verification of the issued certificate failed."
su - zimbra -c "zmcontrol restart" >> lasterr.txt
mail_err "/opt/zimbra/bin/zmcertmgr verifycrt comm $_ckey $_ccert ${_cca}.real\n failed."
return 1
}
# ----- Backup
cp -a /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.$(date "+%Y%m%d")
# ----- Copy private key
cp -f "$_ckey" /opt/zimbra/ssl/zimbra/commercial/commercial.key
# ----- Deploy
rm -f lasterr.txt
_debug "Deploy..."
/opt/zimbra/bin/zmcertmgr deploycrt comm "$_ccert" "${_cca}.real" > lasterr.txt || {
echo "Installation of the issued certificate failed." su - zimbra -c "zmcontrol restart" >> lasterr.txt mail_err "/opt/zimbra/bin/zmcertmgr deploycrt comm $_ckey $_ccert ${_cca}.real\n failed."
return 1
}
# ----- Restart the server
_debug "Restarting Zimbra..."
/usr/bin/openssl x509 -enddate -noout -in "$_ccert" >> lasterr.txt
echo "" >> lasterr.txt
time su - zimbra -c "zmcontrol restart" >> lasterr.txt
sleep 5
su - zimbra -c "zmcontrol status" >> lasterr.txt
mail_ok
_debug "Done."
return 0
}
We also need a pre-hook script to stop Jetty (which is occupying port 80 which acme.sh needs when downloading the cert).
Place this in the directory ".acme.sh" as "zimbra-stop.sh":
#!/bin/bash
#
# Stop the email server/Jetty
#
su - zimbra -c "zmmailboxdctl stop"
Also, remember to place the " isgrootx1.pem" certificate in the same directory (you probably downloaded it above).
# wget -O /root/.acme.sh/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem.txt
Acme.sh saves the pre-hook and deploy hook in its settings, but it do it when you download a cert and deploy it. It does not change the settings if no certificate is downloaded, therefore we need the --force switch below.
Try it out:
# ./acme.sh --issue --pre-hook /root/.acme.sh/zimbra-stop.sh --standalone -d my.server.domain --force
Remove the "--pre-hook /root/.acme.sh/zimbra-stop.sh" part if you just want to test the cert download without having to restart Zimbra. But you'll need to shut down Zimbra (or at least Jetty) or acme.sh won't be able to use port 80.
To test the deployment:
# ./acme.sh --deploy --deploy-hook zimbra -d my.server.domain
Beware that LDAP need to be running when you deploy (this is why zimbra-stop.sh only shuts down Jetty), or you'll see an error message in a line saying:
** Saving server config key zimbraSSLCertificate...failed. ** Saving server config key zimbraSSLPrivateKey...failed.
It will work anyway though.
To automate this, acme.sh has already placed a crontab for your user. Check it out:
# crontab -l
21 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
I wanted it to use a crontab in /etc/cron.d, so I created something like "cert-cron" in "/etc/cron.d":
#
# Regular cron job for zimbra mailserver cert renewal
#
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
0 4 1,7,14,21 * * root [ -x "/root/.acme.sh"/acme.sh ] && "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
Then removed the one installed by acme.sh (this is in /var/spool/cron/root, check first that you don't delete other entries):
# crontab -r
This should be it. Hopefully you'll have the latest and greatest updates for the CentOS6.6 and a functioning cert download. Since certificates rely on outside sources we'll now have to wait until it breaks again, maybe Letsencrypt go out of service or something else. But hopefully it will run for a couple of more years.