Cloudstack 4.22 upgrade: SystemVM template update fails

[bradh352]

problem

After upgrading from cloudstack 4.21.0 to 4.22.0 on Ubuntu 24.04 using the official packages, it was expected the systemvm template would be installed for 4.22, however an error is shown:

versions

4.22.0 on Ubuntu 24.04. KVM hypervisor.

The steps to reproduce the bug

1. Start with a working 4.21.0 cluster
2. Upgrade all nodes to 4.22
3. Observe systemvm template was not updated

What to do about it?

No response

[bradh352]

Tried to manually install it, seems like it worked, but UI still shows the error.

root@node1:~# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /var/lib/cloudstack/mnt/244735810609859.71d49452 -u http://download.cloudstack.org/systemvm/4.22/systemvmtemplate-4.22.0-x86_64-kvm.qcow2.bz2 -h kvm -s 'Test123$' -F
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
--2025-11-10 13:52:28--  http://download.cloudstack.org/systemvm/4.22/systemvmtemplate-4.22.0-x86_64-kvm.qcow2.bz2
Resolving download.cloudstack.org (download.cloudstack.org)... 89.222.120.2, 89.222.120.6, 89.222.120.9, ...
Connecting to download.cloudstack.org (download.cloudstack.org)|89.222.120.2|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 518074341 (494M) [application/x-bzip2]
Saving to: ‘/usr/share/cloudstack-common/scripts/storage/secondary/8471b9d8-6fec-40ed-ad91-8f76de48a2e9.qcow2’

/usr/share/cloudstack-common/scripts/storage/secon 100%[==============================================================================================================>] 494.07M  25.1MB/s    in 19s

2025-11-10 13:52:47 (26.4 MB/s) - ‘/usr/share/cloudstack-common/scripts/storage/secondary/8471b9d8-6fec-40ed-ad91-8f76de48a2e9.qcow2’ saved [518074341/518074341]

Uncompressing to /usr/share/cloudstack-common/scripts/storage/secondary/8471b9d8-6fec-40ed-ad91-8f76de48a2e9.qcow2.tmp (type bz2)...could take a long time
Moving to /var/lib/cloudstack/mnt/244735810609859.71d49452/template/tmpl/1/219///8471b9d8-6fec-40ed-ad91-8f76de48a2e9.qcow2...could take a while
Successfully installed system VM template  and template.properties to /var/lib/cloudstack/mnt/244735810609859.71d49452/template/tmpl/1/219/

[bradh352]

I restarted all my management nodes .... still nothing, can't delete the bad template either....

[weizhouapache]

the systemvm template should be installed automatically

did you specify the checksum value ?

[bradh352]

All I did was change my repo file and do a normal apt-get dist-upgrade to pull in the latest package versions. I then logged into the UI and saw that error message.

I did not specify a checksum anywhere.

I do see the same error with an updated timestamp when I try to forcibly install the template as per the above comment using the cloud-install-sys-tmplt script. The timestamp updates so I know its a different error but the same message.

Its odd that one checksum is super long and the other is a shorter more normal md5 checksum length.

[weizhouapache]

All I did was change my repo file and do a normal apt-get dist-upgrade to pull in the latest package versions. I then logged into the UI and saw that error message.

I did not specify a checksum anywhere.

I do see the same error with an updated timestamp when I try to forcibly install the template as per the above comment using the cloud-install-sys-tmplt script. The timestamp updates so I know its a different error but the same message.

Its odd that one checksum is super long and the other is a shorter more normal md5 checksum length.

which repo do you use ?

you can register a SYSTEM template on UI.
I do not recommend to us cloud-install-sys-tmplt as it will overwrite the last SYSTEM template on secondary storage, but the template image on primary storage is not changed.

4.22 uses SHA-512 sum instead of MD5 sum. The mismatch might be because the SSVM is not recreated or live-patched.

[bradh352]

I'm using the official repo at download.cloudstack.org:

root@node1:~# cat /etc/apt/sources.list.d/cloudstack.sources
# Cloudstack repository list
# https://download.cloudstack.org/ubuntu
X-Repolib-Name: Cloudstack
Types: deb
URIs: https://download.cloudstack.org/ubuntu
Suites: noble
Components: 4.22
Signed-By: /etc/apt/trusted.gpg.d/cloudstack.asc

I just deleted the ssvm to see what would happen, it does show 4.22 in the UI, but its still using the 4.20.2 template as far as I can tell when inspecting it.

Just to be 100% clear, I did NOT run the cloud-install-sys-tmplt until I saw the error in the UI saying the 4.22 template had failed. Nor did I attempt any other method of initializing the system VM.

[weizhouapache]

I'm using the official repo at download.cloudstack.org:

root@node1:~# cat /etc/apt/sources.list.d/cloudstack.sources
# Cloudstack repository list
# https://download.cloudstack.org/ubuntu
X-Repolib-Name: Cloudstack
Types: deb
URIs: https://download.cloudstack.org/ubuntu
Suites: noble
Components: 4.22
Signed-By: /etc/apt/trusted.gpg.d/cloudstack.asc

I just deleted the ssvm to see what would happen, it does show 4.22 in the UI, but its still using the 4.20.2 template as far as I can tell when inspecting it.

is there a template named systemvm-kvm-4.22.0-x86_64 ?

[bradh352]

is there a template named systemvm-kvm-4.22.0-x86_64 ?

Yes, that is the one with the issue:

[weizhouapache]

is there a template named systemvm-kvm-4.22.0-x86_64 ?

Yes, that is the one with the issue:

ok, need some investigation

can you manually update the checksum of the template to the md5sum value (050ebxxxx) in vm_template table ?

[bhouse-nexthop]

ok, need some investigation

can you manually update the checksum of the template to the md5sum value (050ebxxxx) in vm_template table ?

I've done that, as requested, what do I need to trigger it to attempt to use it?

[bradh352]

It looks like restarting the management nodes caused it to re-evaluate. I waited 30 minutes and nothing happened but after restarting the management nodes it picked it up.

[weizhouapache]

It looks like restarting the management nodes caused it to re-evaluate. I waited 30 minutes and nothing happened but after restarting the management nodes it picked it up.

restarting ssvm should help.

is the issue fixed now ? @bradh352

[weizhouapache]

I tested a 4.20->4.22 upgrade, it worked

do you have multiple management servers ? @bradh352

[bradh352]

My cluster is healthy, yes. Doesn't explain what caused the issue, however. Now that I know how to fix it if it happens again, I'm going to upgrade another cluster today to see if it happens there as well. It is also running 4.21.

[weizhouapache]

My cluster is healthy, yes. Doesn't explain what caused the issue, however. Now that I know how to fix it if it happens again, I'm going to upgrade another cluster today to see if it happens there as well. It is also running 4.21.

if there are multiple management servers, it would be good to share the upgrade steps