Errors and solutions when starting a vserver Guest

 

 

Some Vservers can't access or lost Internet!

Assume you have say 8 public IP addresses that has been assigned by your ISP. In some cases, one or all of the guest systems (and maybe even the main server handler) will not be able to reach Internet.

The reason can be that the upstream gateway need to update its arp cache. This is more likely if you move one guest vserver from one main server (with one MAC address) to another (with another mac address).

There are two solutions:

1. If behind an ubiquity firewall

Log in to the firewall trough ssh and type

sudo -s
arping -s <vserver guest ip> <isp gateway ip>

 

2. If the server handler is directly connected to Internet

Assuming the main server runs debian, open a root shell on the main server/server handler and type

arping -S <vserver guest ip> -B

 

 

Login by ssh or su is slow

You might experience a long delay when logging in by ssh or using su.

You can fix this by (in the guest system):

pam-auth-update

(uncheck "elogind Session Management") 

 

Errors when starting a 32-bit vserver on XFS

I recently started up a new server. It has a large hard drive array (>5TByte) and an internal uSD-card. I installed debian Buster on the uSD-card, changed the kernel for vserver support and partitioned the huge mechanical raid array sda as

Device        Start         End     Sectors   Size Type
/dev/sda1      2560     1953279     1950720 952.5M Linux swap
/dev/sda2   1953280    22924799    20971520    10G Linux filesystem
/dev/sda3  22924800 11720797326 11697872527   5.5T Linux filesystem

Since the OS is running from a uSD card and I wanted to avoid degeneration of the flash, I moved /var to /dev/sda2 (formatted as ext4). This causes minimal writes to the uSD card:

init 1
mkfs -t ext4 /dev/sda2
mount /dev/sda2 /mnt/var

cp -apx /var/* /mnt/var
echo "/dev/sda2 /var/ ext4 defaults 0 0">>/etc/fstab
umount /mnt/var
exit

The /dev/sda3, formatted as XFS, was mounted to /vservers:

echo "/dev/sda3 /vservers xfs defaults  0 1">>/etc/fstab     # THIS IS WRONG, SEE BELOW!
modprobe -v xfs
mkfs.xfs -f /dev/sda3

mount /dev/sda3 /vservers

I then transferred some vservers there using rsync,

apt-get install rsync
cd /vservers
rsync -aPxvze "ssh -c aes128-ctr"  --whole-file --stats --numeric-ids remote-vserver:/srv/* .

When I started up one of the vservers, a 32-bit installation of "mailman", I got errors like

cron: "'/var/spool/cron' is not a directory, bailing out.".

Apache2 also fails, and "apt-get update" gives "E: Lists directory /var/lib/apt/lists/partial is missing." (it's not).

After some help from IRC Bertl_oO and Guy- to track down the bug using strace, i could see from the strace that stat64 returned

stat64("/var/spool/cron", {st_dev=makedev(8, 3), st_ino=4832143994, st_mode=S_IFDIR|0755, st_nlink=3, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=30, st_atime=2022/10/24-11:17:56, st_mtime=2010/01/30-21:19:38, st_ctime=2022/10/24-11:17:47}) = 0

Notice the st_ino. The reason seem to be that the directory /var/spool/cron had an inode requiring 33 bits to store. n=log(4832143994)/log(2)=32.17 so 33 bits are required. As a quick test I moved the vserver to an ext4 partition and it then started just fine.

Solution

One solution might be to use the mount option "inode32" to xfs:

echo "/dev/sda3 /vservers xfs defaults,inode32  0 1">>/etc/fstab

 

or simply use ext4 for 32-bit vservers.
 

 

Subversion clients complain

I did not really document this problem, so this is just a hint of what is going on. It might have been caused by running a 32-bit vserver guest on a 64-bit XFS file system (see above).

After installing the new buster vserver kernel image, I started up an old guest system hosting a Subversion server (using apache + ssl, mod-svn).
However, I got weird errors and complaints when committing to repositories.
I have not yet found the solution, but I believe something has changed in the file system, e.g the kernel returning an uint32 instead of an uint16 when asking for a file parameter.

A working solution is to rebuild the svn guest system from scratch. The problem disappeared when using buster as guest system and the latest Apache + mod-svn.

Here is some quick commands to get started. On the fresh Buster guest:

apt-get install subversion libsvn-dev subversion-tools

mkdir -p /var/www/svn
cd /var/www/svn
svnadmin create --fs-type fsfs /var/www/svn/test-repo
chown -R www-data:www-data /var/www/svn
chmod -R 755 /var/www/svn

apt-get install apache2 apache2-utils libapache2-mod-svn

a2enmod dav
a2enmod dav_svn

mcedit /etc/apache2/mods-enabled/dav_svn.conf

Add the following entry to the file:

########
<Location /svn>
DAV svn
SVNParentPath /var/www/svn
AuthType Basic
AuthName "SVN test Repo"
AuthUserFile /etc/svnusers
Require valid-user
</Location>
########

Then: 

systemctl restart apache2

htpasswd -cm /etc/svnusers svn1

(browse to the server to see that it's working). To set up SSL:

sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt

mcedit /etc/apache2/conf-available/ssl-params.conf


#######
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
# Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
# Requires Apache >= 2.4.11
SSLSessionTickets Off
#######

cp /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-available/default-ssl.conf.bak
mcedit /etc/apache2/sites-available/default-ssl.conf

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin This email address is being protected from spambots. You need JavaScript enabled to view it.
                ServerName server_domain_or_IP

                DocumentRoot /var/www/html

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on

                SSLCertificateFile      /etc/ssl/certs/apache-selfsigned.crt
                SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key

                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>

        </VirtualHost>
</IfModule>


a2enmod ssl
a2enmod headers
a2ensite default-ssl
a2enconf ssl-params
apache2ctl configtest


 

Proc gets unmounted, host go bananas

If you're running Debian Buster you might have to "down-grade" (replace) util-vserver. Otherwise, when starting a guest, you might see:

# vserver buildbuster642 start
Using makefile-style concurrent boot in runlevel 3.
ERROR: could not open /proc/stat: No such file or directory
Starting enhanced syslogd: rsyslogdrsyslogd: imklog: cannot open kernel log (/proc/kmsg): No such file or directory.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
.
Starting periodic command scheduler: cron.
vshelper.init: can not determine xid of vserver 'buildbuster642'; returned value was ''

This usually means that you're using an init-less init-style, but the
guest isn't configured to start any service. Try enabling a service,
changing the init-style, or making the contexts persistent.

An error occured after executing the vserver startup sequence. This
means that some processes may exist in the created context and the
manual execution of

  /usr/sbin/vserver '/etc/vservers/buildbuster642' stop

is recommended to fix this.

and, at this point you can see that /proc has got unmounted! Try e.g ls /proc. It will come up empty. To fix it temporary, use "mount -t proc none /proc".

Other things has happened as well though, like /dev and /sys being unmounted. There will now also be a problem with a dangling symlink. In /var, there should be the symlinks

/var:
run -> /run
lock -> /run/lock

In /run, there should be the directory "lock":

/run/lock

If not, make sure to create e.g the directory /run/lock if it's missing.

 

If you try to start the guest again you will get 

vcontext: vc_ctx_create(): File exists

An error occured while executing the vserver startup sequence; when
there are no other messages, it is very likely that the init-script
(/usr/lib/init/rc 3) failed.

Common causes are:
* /etc/rc.d/rc on Fedora Core 1 and RH9 fails always; the 'apt-rpm' build
  method knows how to deal with this, but on existing installations,
  appending 'true' to this file will help.


Failed to start vserver 'buildbuster642'

This is because it's running, but you can't stop it with "vserver buildbuster642 stop". You will have to do this:

#vserver-stat

open(memory.stat): No such file or directory
CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
2000     4 157.9M     0    0m00s00   0m00s00  21m16s13

Notice the context CTX. Now:

chcontext --xid 2000 -- ps -ax

will list what is running on the guest. Example:

# chcontext --xid 2000 -- ps -ax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:01 /sbin/init
 9326 ?        Ssl    0:00 /usr/sbin/rsyslogd
 9335 ?        Ss     0:00 /usr/sbin/cron
 9665 ?        R+     0:00 ps -ax

Now just kill cron and rsyslogd:

chcontext --xid 2000 -- kill 9335 9326

And you will see that vserver-stat returns that no servers are running.

 

Make sure to reboot after this to recover /dev and /sys.

Then, before trying to restart the guest, do the "downgrading" and patch listed here: Installing VSERVER on Buster and running buster guest(s) (see "Fixing what doesn't work out of the box").

 

 

 

 

 

 

 

 

 

 

Add comment

Security code
Refresh