echo "

Script to clean up all the ARF/OpenScap compliance reports in Satellite

Since we only need to know the last compliance check I made a script to clean up all the previous reports before the next compliance check runs.

#!/bin/bash
#this script removes all the arf reports from the satellite server
###

#settings
USER=ronly
PASS=xxxxxxxxxxx
URI=https://localhost

#check amount of reports
while [ $(curl -k -u $USER:$PASS $URI/api/v2/compliance/arf_reports/ | python -m json.tool | grep \"\total\": | cut -f2 -d":" | cut -f1 -d"," | sed "s/ //g") -gt 0 ]; do
        #fetch reports
        for i in $(curl -k -u $USER:$PASS $URI/api/v2/compliance/arf_reports/ | python -m json.tool | grep \"\id\": | cut -f2 -d":" | cut -f1 -d"," | sed "s/ //g")
        #delete reports
        do
                curl -k -u $USER:$PASS -i -H "Content-Type: application/json" -X DELETE $URI/api/v2/compliance/arf_reports/$i
        done
done

To manually rerun the benchmark on all machines I use following ansible command

ansible all -m shell -a 'eval $(grep foreman_scap_client /var/spool/cron/root | cut -f6-7 -d" " | sed '/^$/d')'

 Update: Red Hat published my script https://access.redhat.com/solutions/3040861

 Update: After Satellite 6.3 the location for the cron rule has changed to /etc/cron.d/foreman_scap_client_cron

MDADM and a single LVM to boot from (on a Debian based system)

Yesterday I moved a customer's server to new disks. To get some extra features like snapshotting I opted to convert the current static disk layout to LVM on top of MDADM. As a bonus I did the entire sync online to a new disk connected to my laptop which acted as a degraded mirror.

Yes I know GPT is the way to go but I started this move at 7 PM to minimise business impact and since the new disks are still only 500G I went for MBR which was already in place. For GPT "RAID" check this answer

This tutorial can also be used to move a non-RAID server to a RAID setup or to move a server to a new machine that will have MDADM RAID.

First create the partition table

parted /dev/sdz 
mklabel msdos
mkpart primary ext2 0% 100%
set 1 lvm on
quit

Create the MDADM mirror and LVM group + partitions (I will only use 2 in this tutorial but you can use this as a guide to get a better solution for a professional environment)

mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sdz1 missing
pvcreate /dev/md0
vgcreate lvm /dev/md0
lvcreate -L 500M lvm -n boot
lvcreate -L 20G lvm -n root
mkfs.ext4 /dev/mapper/lvm-boot
mkfs.ext4 /dev/mapper/lvm-root

 Next mount the new partitions and start cloning the old system

mount /dev/mapper/lvm-root /mnt
mkdir /mnt/boot
mount /dev/mapper/lvm-boot /mnt/boot
rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} user@src /mnt/
*here you should stop all services on the src and rerun the rsync command for a final sync*

Next we will set up the bootloader and adapt the system files

for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
sudo cp /etc/resolv.conf /mnt/etc/resolv.conf
sudo chroot /mnt
blkid
*get the ID's for the LVM paritions*
vi /etc/fstab
*replace the ID's of the old /, /boot, ...*
apt-get install lvm2 mdadm
mdadm --examine --scan >> /etc/mdadm.conf
*delete the previous lines starting with ARRAY, if any*
vi /etc/mdadm/mdadm.conf
*now update initramfs to make sure it contains MDADM and LVM2 support and install grub to the new disk*
update-initramfs -k all -u -v
grub-install /dev/sdz

At this point you should be able to shut down the src server and boot from the new disk after replacement (or if you use this to move a server by starting the dst server). If you are able to boot succesfully and have the second disk already in place we are now going to "restore" the mirror.

*copy the MBR*
dd if=/dev/sdz of=/dev/sdy bs=512 count=1
partprobe /dev/sdy
*add the disk to the MDADM RAID to start rebuilding*
mdadm --manage /dev/md0 --add /dev/sdy1
*check if the rebuild is started*
cat /proc/mdstat
*just to make sure reinstall grub on boot disks, choose both /dev/sdz en /dev/sdy*
dpkg-reconfigure grub-pc

Another reboot after the rebuild is finished can be good to verify everything.

Zimbra to Zimbra migration using Zextras and SSHFS

This is only an extension to a normal installation. So you should have your new Zimbra environment ready to move the data of your current Zimbra environment to it.

In the past we did an in-place upgrade of our Zimbra and the Ubuntu it was residing on. The last upgrade from 8.5.1 on Ubuntu 14.04 to 8.7.1 on Ubuntu 16.04 was such a drama that I started looking for an easier solution that would give me a fresh Zimbra with all my settings, accounts, mails, calendar items ... without too much fuss.

To prevent inconsistency please stop your old Zimbra and afterwards make a snapshot and/or backup before you continue.

/etc/init.d/zimbra stop

OR

su - zimbra -c "zmcontrol stop"

Also stop and disable fetchmail if you're using it

/etc/init.d/fetchmail stop 
sed -i "s/START_DAEMON=yes/START_DAEMON=no/g" /etc/default/fetchmail

Next change the IP because the Zimbra will need to be started again to do the export

sed -i "s/old_ip/temporary_ip/g" /etc/hosts /etc/network/interfaces
reboot

Next on both Zimbras create a folder to store the migration data on

su - zimbra -c "mkdir /opt/zimbra/backup/exports /opt/zimbra/backup/zextras"

Next mount this folder as the user zimbra on the new Zimbra (don't set a password on the zimbra user, just create a key pair and share the SSH public key). This will save you time to move the data.

 sshfs zimbra@temporary_ip:/opt/zimbra/backup/exports/ /opt/zimbra/backup/exports/

Next install the migration tool on the old Zimbra

cd /tmp && wget https://download.zextras.com/zextras_migration_tool-latest.tgz
tar xvzf zextras_migration_tool-*.tgz
cd zextras_migration_tool-*/
./install all

Now open the Zimbra admin page (https://temporary_ip:7071) and start exporting the domains and user you want to keep. Through ZeXtras -> ZxMig -> Start Migration

undefined

undefinedundefined

 Change the folder to the newly created /opt/zimbra/backup/exports

undefined

Select the domains you want to move

undefined

Under ZxNotifications check for any error during the export

undefined

When this finishes you can start importing on the new Zimbra. First install the Zextra suite (this is free for 30 days and we will remove it after we finish to get rid of warnings afterwards)

cd /tmp && wget http://download.zextras.com/zextras_suite-latest.tgz 
tar xvzf zextras_suite-*.tgz
cd zextras_suite-*/
./install all

Open the Zimbra admin page (https://new_ip:7071) and open the ZeXtas -> ZxBackup -> select Import Backup

 undefinedundefinedundefined

Change the path to /opt/zimbra/backup/exports

undefined

Select the domains you want to import

undefinedundefined

Select the accounts you want to import

undefined

And check if the restore started without warnings

undefined

After this finished you can clean up and give your new Zimbra the IP of the old one to start sending and receiving mails again.

If you had an SSL certificate installed now is the time to move that as well. Do this as user zimbra!

On the old zimbra

rsync -au /opt/zimbra/ssl/ zimbra@new_ip:/tmp/ssl/

On the new Zimbra

cp /tmp/ssl/zimbra/commercial/commercial.key /opt/zimbra/ssl/zimbra/commercial/ 
./zmcertmgr deploycrt comm /tmp/ssl/zimbra/commercial/commercial.crt /tmp/ssl/zimbra/commercial/commercial_ca.crt

In any case I would also enable HTTP -> HTTPS redirection

zmprov ms "FQDN" zimbraReverseProxyMailMode redirect

Let's finish:

FIRST: shutdown the old Zimbra machine and make sure it won't boot by itself. Don't forget to copy any backup settings, fetchmail config, cron jobs, ...

On the new Zimbra

cd /tmp/zextras_suite-*/ 
./install -u all
sed -i "s/new_ip/old_ip/g" /etc/hosts /etc/network/interfaces
reboot

Now your new Zimbra should be available with all the settings, accounts and mails just like it was when you stopped your old Zimbra.

Upgrading an existing Zimbra environment from Ubuntu 14.04 (ZCS 8.6.0) to Ubuntu 16.04 (ZCS 8.7.1)

Since I got some complaints about the current environment which I checked and according to the changelog these issues should be fixed by now. I'm going to perform an in-place upgrade of my current Zimbra 8.6.0 on Ubuntu 14.04 to Zimbra 8.7.1 on Ubuntu 16.04. The system runs in an OpenVZ container which gives us some extra flexibility.

 

First of all create a backup or a snapshot after you stopped the Zimbra services.

/etc/init.d/zimbra stop

Next create a "snapshot". Since this hypervisor doesn't have a snapshot solution I specifically choose to use the local (SSD) storage without compression to get the backup to finish as fast as possible, we can't afford unnecessary downtime and in the next generation of our servers we will be using file system based snasphots (LVM or ZFS) which makes waiting for a backup to finish something of the past. I created my snapshot using the web interface of Proxmox 3.4 but this is what happens in the background after stopping the container.

vzdump 113 --remove 0 --mode stop --storage local --node duff-prox-01

 

To prevent Zimbra from accepting mails. I changed the IP of my VPS during the entire upgrade. Next restart Zimbra and make sure it still runs fine.

sed -i "s/172.16.4.10/172.16.4.99/g" /etc/network/interfaces /etc/hosts
service networking restart
/etc/init.d/zimbra restart

 

Next install and enable the Zimbra proxy and memcached if you didn't had them configured in the past. See https://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy_and_memcached

cd /tmp/ && wget https://files.zimbra.com/downloads/8.6.0_GA/zcs-8.6.0_GA_1153.UBUNTU14_64.20141215151116.tgz
tar xvzf zcs-8.6.0_GA_1153.UBUNTU14_64.20141215151116.tgz
cd zcs-8.6.0_GA_1153.UBUNTU14_64.20141215151116/packages
dpkg -i dpkg -i zimbra-proxy_8.6.0.GA.1153.UBUNTU14.64_amd64.deb zimbra-memcached_8.6.0.GA.1153.UBUNTU14.64_amd64.deb
su - zimbra
./libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x both -H zimbra.ampersant
./libexec/zmproxyconfig -e -m -o -i 7143:143:7993:993 -p 7110:110:7995:995 -H zimbra.ampersant
zmprov ms zimbra.ampersant zimbraMailReferMode reverse-proxied
zmmailboxdctl restart
zmprov ms zimbra.ampersant +zimbraServiceEnabled memcached
zmcontrol restart

 

Now upgrade Zimbra to the latest edition available for Ubuntu 14.04

Note! Zimbra introduced their own repository so before starting the upgrade make sure you can install GPG keys from keyserver.ubuntu.com:11371 and that your server can surf the web (through the proxy if any). And just to be sure, rebuild the apt cache:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9BE6ED79
sudo apt-get clean
cd /var/lib/apt
sudo mv lists lists.old
sudo mkdir -p lists/partial
sudo apt-get clean
sudo apt-get update
cd /tmp/ && wget https://files.zimbra.com/downloads/8.7.1_GA/zcs-8.7.1_GA_1670.UBUNTU14_64.20161025045105.tgz
tar xvzf zcs-8.7.1_GA_1670.UBUNTU14_64.20161025045105.tgz
cd zcs-8.7.1_GA_1670.UBUNTU14_64.20161025045105
./install.sh

If you run in to any other issues please first check the installation log

tail -f /tmp/install.log.*

If you successfully finish the upgrade stop Zimbra before you upgrade Ubuntu

/etc/init.d/zimbra stop

 

Next upgrade Ubuntu (this application is in the update-manager-core package)

do-release-upgrade

 

Only for when you are running a very old kernel! Revert back to upstart

apt-get install upstart-sysv

 

Get the latest build of Zimbra for Ubuntu 16.04

cd /tmp/ && wget https://files.zimbra.com/downloads/8.7.1_GA/zcs-8.7.1_GA_1670.UBUNTU16_64.20161025045114.tgz

 

Next extract the installation files

tar -xvzf zcs-8.7.1_GA_1670.UBUNTU16_64.20161025045114.tgz

 

And start the repair of the old install to be able to do an in-place upgrade. You will get some alerts and we work around them anyway

cd zcs-8.7.1_GA_1670.UBUNTU16_64.20161025045114
echo "deb [arch=amd64] https://repo.zimbra.com/apt/87 xenial zimbra
deb-src [arch=amd64] https://repo.zimbra.com/apt/87 xenial zimbra" > /etc/apt/sources.list.d/zimbra.list
apt-get update
apt-get install zimbra-core-components
apt-get -f install
dpkg -i packages/*.deb
apt-get -f install
./install.sh --softwareonly #DO NOT CHECK DATABASE INTEGRITY!

Started 1:10 -> finished

Benchmarking Linux bonding with cheap NICs

Since a couple of years I have been using bonding to achieve load balancing, higher bandwidth and redundancy on Linux (and BSD, Windows, Solaris, ...). At home this can be rather challenging since lots of consumer switches don't understand all the bonding methods and Realtek adapters have the unpleasant habit of keeping the spoofed MAC address after a reboot which messes things up.

For this set up I will use a bunch of cheap Realtek and Intel NICs to connect 2 PC's using 4 NICs on every side to benchmark the actual speed.

I already mentioned that my Realtek adapters tend to remember the spoofed MAC address during reboot which causes UDEV to give the adapters a new name and rendering your config worthless. To prevent this I added some lines inside /etc/network/interfaces that start and stop the bond and load and unload the bonding module which causes the NIC to have it's factory MAC address back before rebooting.

This is my basic config using bonding mode 0 (round-robin). I only changed the mode= or in case of mode=2 and mode=4 with L3+L4 hashing I uncommented the xmit_hash_policy variable. All benchmarked modes give fault tolerance and load balancing (albeit not always visible in the results)

# The primary network interface
auto bond0
iface bond0 inet static
pre-up modprobe bonding mode=0 miimon=100 #xmit_hash_policy=layer3+4
pre-up ip link set enp4s0 master bond0
pre-up ip link set enp6s0 master bond0
pre-up ip link set enp8s0 master bond0
pre-up ip link set enp9s0 master bond0
mtu 9000
address 10.0.0.1
netmask 255.255.255.0
up /bin/true
down /bin/true
post-down ip link set enp4s0 nomaster
post-down ip link set enp6s0 nomaster
post-down ip link set enp8s0 nomaster
post-down ip link set enp9s0 nomaster
post-down ip link set dev bond0 down
post-down rmmod bonding

Mode=0 is only good for connecting 2 systems with multiple NICs directly to each other to achieve a single connection with a higher bandwidth. In switched environments or even worse in a bridge this method will really mess up your connections. In a switched connection you will see random packet loss and out of order packets. In a bridge you are lucky if you even get some packages over.

To get the most out of this I set the MTU to 9000 (jumbo frames) and connected 2 systems directly to each other. My NICs are all auto sensing so I didn't have to use crossed cables.

I used these scripts to run multiple instances of iperf (network benchmark tool) in parallel https://sandilands.info/sgordon/multiple-iperf-instances

method mode 0 (Round Robin) mode 2 (XOR) mode 2 (XOR L3+L4) mode 4 (LACP L2) mode 4 (LACP L3+L4)
single connection speed 3.14 Gbits/sec 719 Mbits/sec 661 Mbits/sec 730 Mbits/sec 725 Mbits/sec
total speed of 4 simultaneous connections 3.157 Gbits/sec 722 Mbits/sec 2.107 Gbit/s 735 Mbits/sec 1.484 Gbits/sec
ethtool advertised speed 4000Mb/s 4000Mb/s 4000Mb/s 4000Mb/s 4000Mb/s
Where to use inter system connection only

inter system connection OR on a switch that only supports static LAGs

inter system connection OR on a L3 switch than only supports static LAGs (...)

inter system connection OR on a L2 managed switch that supports LACP inter system connection OR on a L3 switch that supports LACP

As you can see mode=0 wins the benchmark but as I already said that comes with a price. Netgear for instance will recommend mode=2 for unmanaged switches or switches that only can handle static LAGs and mode=4 for switches that support LACP.

But in real life you should always use LACP since it is the most robust method out there and if you really need higher single connection speed you will have to invest in 10, 25, 40 or 100Gbit connections.

LACP can combine any form of hashing. You can combine hashing the MAC address (L2), the IP address (L3) or the UDP/TCP session (L4). This isn't only the case for Linux. Solaris and derivates like SmartOS just as well give you the option to combine any of these 3 hashing methods in your LACP aggregate. In Solaris L4 hashing is the default one, in Linux it is L2.

You can see in my result that running multiple iperf sessions on different ports really does make a difference if you use L4 hashing since we have 4 different TCP sessions. In the real world LACP will rarely be used for single connection configurations. If you use LACP underneath a bridge for your hypervisor for instance, since all your VMs and containers will have their own MAC address, IP address and use their own UDP/TCP sessions all your physical connections will be actually used and you will get a higher total bandwidth (albeit hard to benchmark) but you will never get more than 1 Gbit per session.

To finish, one mode that doesn't have anything to do with load balancing but can be nice is mode=1. I used this in the past to set up a fiber connection as primary link with a WiFi link as backup. In case the fiber stops working, traffic would be send over the WiFi link. Of course this kind of behavior can be achieved by using STP as well if you have a managed switch on both ends.

Home