Errors and solutions when starting a vserver Guest
- Some Vservers can't access or lost Internet!
- Login by ssh or su is slow
- Errors when starting a 32-bit vserver on XFS
- Subversion clients complain
- /proc gets unmounted, host go bananas
Some Vservers can't access or lost Internet!
Assume you have say 8 public IP addresses that has been assigned by your ISP. In some cases, one or all of the guest systems (and maybe even the main server handler) will not be able to reach Internet.
The reason can be that the upstream gateway need to update its arp cache. This is more likely if you move one guest vserver from one main server (with one MAC address) to another (with another mac address).
There are two solutions:
1. If behind an ubiquity firewall
Log in to the firewall trough ssh and type
sudo -s arping -s <vserver guest ip> <isp gateway ip>
2. If the server handler is directly connected to Internet
Assuming the main server runs debian, open a root shell on the main server/server handler and type
arping -S <vserver guest ip> -B
Login by ssh or su is slow
You might experience a long delay when logging in by ssh or using su.
You can fix this by (in the guest system):
pam-auth-update
(uncheck "elogind Session Management")
Errors when starting a 32-bit vserver on XFS
I recently started up a new server. It has a large hard drive array (>5TByte) and an internal uSD-card. I installed debian Buster on the uSD-card, changed the kernel for vserver support and partitioned the huge mechanical raid array sda as
Device Start End Sectors Size Type
/dev/sda1 2560 1953279 1950720 952.5M Linux swap
/dev/sda2 1953280 22924799 20971520 10G Linux filesystem
/dev/sda3 22924800 11720797326 11697872527 5.5T Linux filesystem
Since the OS is running from a uSD card and I wanted to avoid degeneration of the flash, I moved /var to /dev/sda2 (formatted as ext4). This causes minimal writes to the uSD card:
init 1
mkfs -t ext4 /dev/sda2
mount /dev/sda2 /mnt/var
cp -apx /var/* /mnt/var
echo "/dev/sda2 /var/ ext4 defaults 0 0">>/etc/fstab
umount /mnt/var
exit
The /dev/sda3, formatted as XFS, was mounted to /vservers:
echo "/dev/sda3 /vservers xfs defaults 0 1">>/etc/fstab # THIS IS WRONG, SEE BELOW!
modprobe -v xfs
mkfs.xfs -f /dev/sda3
mount /dev/sda3 /vservers
I then transferred some vservers there using rsync,
apt-get install rsync
cd /vservers
rsync -aPxvze "ssh -c aes128-ctr" --whole-file --stats --numeric-ids remote-vserver:/srv/* .
When I started up one of the vservers, a 32-bit installation of "mailman", I got errors like
cron: "'/var/spool/cron' is not a directory, bailing out.".
Apache2 also fails, and "apt-get update" gives "E: Lists directory /var/lib/apt/lists/partial is missing." (it's not).
After some help from IRC Bertl_oO and Guy- to track down the bug using strace, i could see from the strace that stat64 returned
stat64("/var/spool/cron", {st_dev=makedev(8, 3), st_ino=4832143994, st_mode=S_IFDIR|0755, st_nlink=3, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=30, st_atime=2022/10/24-11:17:56, st_mtime=2010/01/30-21:19:38, st_ctime=2022/10/24-11:17:47}) = 0
Notice the st_ino. The reason seem to be that the directory /var/spool/cron had an inode requiring 33 bits to store. n=log(4832143994)/log(2)=32.17 so 33 bits are required. As a quick test I moved the vserver to an ext4 partition and it then started just fine.
Solution
One solution might be to use the mount option "inode32" to xfs:
echo "/dev/sda3 /vservers xfs defaults,inode32 0 1">>/etc/fstab
or simply use ext4 for 32-bit vservers.
Subversion clients complain
I did not really document this problem, so this is just a hint of what is going on. It might have been caused by running a 32-bit vserver guest on a 64-bit XFS file system (see above).
After installing the new buster vserver kernel image, I started up an old guest system hosting a Subversion server (using apache + ssl, mod-svn).
However, I got weird errors and complaints when committing to repositories.
I have not yet found the solution, but I believe something has changed in the file system, e.g the kernel returning an uint32 instead of an uint16 when asking for a file parameter.
A working solution is to rebuild the svn guest system from scratch. The problem disappeared when using buster as guest system and the latest Apache + mod-svn.
Here is some quick commands to get started. On the fresh Buster guest:
apt-get install subversion libsvn-dev subversion-tools
mkdir -p /var/www/svn
cd /var/www/svn
svnadmin create --fs-type fsfs /var/www/svn/test-repo
chown -R www-data:www-data /var/www/svn
chmod -R 755 /var/www/svn
apt-get install apache2 apache2-utils libapache2-mod-svn
a2enmod dav
a2enmod dav_svn
mcedit /etc/apache2/mods-enabled/dav_svn.conf
Add the following entry to the file:
########
<Location /svn>
DAV svn
SVNParentPath /var/www/svn
AuthType Basic
AuthName "SVN test Repo"
AuthUserFile /etc/svnusers
Require valid-user
</Location>
########
Then:
systemctl restart apache2
htpasswd -cm /etc/svnusers svn1
(browse to the server to see that it's working). To set up SSL:
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
mcedit /etc/apache2/conf-available/ssl-params.conf
#######
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
# Disable preloading HSTS for now. You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
# Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
# Requires Apache >= 2.4.11
SSLSessionTickets Off
#######
cp /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-available/default-ssl.conf.bak
mcedit /etc/apache2/sites-available/default-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin This email address is being protected from spambots. You need JavaScript enabled to view it.
ServerName server_domain_or_IP
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
</VirtualHost>
</IfModule>
a2enmod ssl
a2enmod headers
a2ensite default-ssl
a2enconf ssl-params
apache2ctl configtest
Proc gets unmounted, host go bananas
If you're running Debian Buster you might have to "down-grade" (replace) util-vserver. Otherwise, when starting a guest, you might see:
# vserver buildbuster642 start Using makefile-style concurrent boot in runlevel 3.
ERROR: could not open /proc/stat: No such file or directory
Starting enhanced syslogd: rsyslogdrsyslogd: imklog: cannot open kernel log (/proc/kmsg): No such file or directory.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
.
Starting periodic command scheduler: cron.
vshelper.init: can not determine xid of vserver 'buildbuster642'; returned value was ''
This usually means that you're using an init-less init-style, but the
guest isn't configured to start any service. Try enabling a service,
changing the init-style, or making the contexts persistent.
An error occured after executing the vserver startup sequence. This
means that some processes may exist in the created context and the
manual execution of
/usr/sbin/vserver '/etc/vservers/buildbuster642' stop
is recommended to fix this.
and, at this point you can see that /proc has got unmounted! Try e.g ls /proc. It will come up empty. To fix it temporary, use "mount -t proc none /proc".
Other things has happened as well though, like /dev and /sys being unmounted. There will now also be a problem with a dangling symlink. In /var, there should be the symlinks
/var:
run -> /run
lock -> /run/lock
In /run, there should be the directory "lock":
/run/lock
If not, make sure to create e.g the directory /run/lock if it's missing.
If you try to start the guest again you will get
vcontext: vc_ctx_create(): File exists
An error occured while executing the vserver startup sequence; when
there are no other messages, it is very likely that the init-script
(/usr/lib/init/rc 3) failed.
Common causes are:
* /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' build
method knows how to deal with this, but on existing installations,
appending 'true' to this file will help.
Failed to start vserver 'buildbuster642'
This is because it's running, but you can't stop it with "vserver buildbuster642 stop". You will have to do this:
#vserver-stat
open(memory.stat): No such file or directory
CTX PROC VSZ RSS userTIME sysTIME UPTIME NAME
2000 4 157.9M 0 0m00s00 0m00s00 21m16s13
Notice the context CTX. Now:
chcontext --xid 2000 -- ps -ax
will list what is running on the guest. Example:
# chcontext --xid 2000 -- ps -ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:01 /sbin/init
9326 ? Ssl 0:00 /usr/sbin/rsyslogd
9335 ? Ss 0:00 /usr/sbin/cron
9665 ? R+ 0:00 ps -ax
Now just kill cron and rsyslogd:
chcontext --xid 2000 -- kill 9335 9326
And you will see that vserver-stat returns that no servers are running.
Make sure to reboot after this to recover /dev and /sys.
Then, before trying to restart the guest, do the "downgrading" and patch listed here: Installing VSERVER on Buster and running buster guest(s) (see "Fixing what doesn't work out of the box").