Table of Contents

Upgrade Guide

Upgrading a standalone Engine, local database environment

1. Upgrading from 4.2 to oVirt 4.3

Upgrading your environment from 4.2 to 4.3 involves the following steps:

1.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

1.2. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

1.3. Upgrading the Engine from 4.2 to 4.3

You need to be logged into the machine that you are upgrading.

If the upgrade fails, the engine-setup command attempts to restore your oVirt Engine installation to its previous state. For this reason, do not remove the previous version’s repositories until after the upgrade is complete. If the upgrade fails, the engine-setup script explains how to restore your installation.

Procedure
  1. Enable the oVirt 4.3 repositories:

    All other repositories remain the same across oVirt releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Run engine-setup and follow the prompts to upgrade the oVirt Engine:

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
  4. Update the base operating system:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the upgrade.

The Engine is now upgraded to version 4.3.

You can now update the hosts.

1.4. Updating All Hosts in a Cluster

You can update all hosts in a cluster instead of updating hosts individually. This is particularly useful during upgrades to new versions of oVirt. See https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/cluster_upgrade/README.md for more information about the Ansible role used to automate the updates.

Update one cluster at a time.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines are shut down during the update, unless you choose to skip that host instead.

Procedure
  1. In the Administration Portal, click Compute  Clusters and select the cluster. The Upgrade status column shows if an upgrade is available for any hosts in the cluster.

  2. Click Upgrade.

  3. Select the hosts to update, then click Next.

  4. Configure the options:

    • Stop Pinned VMs shuts down any virtual machines that are pinned to hosts in the cluster, and is selected by default. You can clear this check box to skip updating those hosts so that the pinned virtual machines stay running, such as when a pinned virtual machine is running important services or processes and you do not want it to shut down at an unknown time during the update.

    • Upgrade Timeout (Minutes) sets the time to wait for an individual host to be updated before the cluster upgrade fails with a timeout. The default is 60. You can increase it for large clusters where 60 minutes might not be enough, or reduce it for small clusters where the hosts update quickly.

    • Check Upgrade checks each host for available updates before running the upgrade process. It is not selected by default, but you can select it if you need to ensure that recent updates are included, such as when you have configured the Engine to check for host updates less frequently than the default.

    • Reboot After Upgrade reboots each host after it is updated, and is selected by default. You can clear this check box to speed up the process if you are sure that there are no pending updates that require a host reboot.

    • Use Maintenance Policy sets the cluster’s scheduling policy to cluster_maintenance during the update. It is selected by default, so activity is limited and virtual machines cannot start unless they are highly available. You can clear this check box if you have a custom scheduling policy that you want to keep using during the update, but this could have unknown consequences. Ensure your custom policy is compatible with cluster upgrade activity before disabling this option.

  5. Click Next.

  6. Review the summary of the hosts and virtual machines that will be affected.

  7. Click Upgrade.

You can track the progress of host updates:

  • in the Compute  Clusters view, the Upgrade Status column shows Upgrade in progress.

  • in the Compute  Hosts view

  • in the Events section of the Notification Drawer (EventsIcon).

You can track the progress of individual virtual machine migrations in the Status column of the Compute  Virtual Machines view. In large environments, you may need to filter the results to show a particular group of virtual machines.

1.5. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

1.6. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

1.7. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

If you previously upgraded to 4.2 without replacing SHA-1 certificates with SHA-256 certificates, you must do so now.

1.8. Replacing SHA-1 Certificates with SHA-256 Certificates

oVirt 4.4 uses SHA-256 signatures, which provide a more secure way to sign SSL certificates than SHA-1. Newly installed systems do not require any special steps to enable oVirt’s public key infrastructure (PKI) to use SHA-256 signatures.

Preventing Warning Messages from Appearing in the Browser

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Define the certificate that should be re-signed:

    # names="apache"
  4. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  5. Restart the httpd service:

    # systemctl restart httpd
  6. Connect to the Administration Portal to confirm that the warning no longer appears.

  7. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

Replacing All Signed Certificates with SHA-256

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Re-sign the CA certificate by backing it up and creating a new certificate in ca.pem.new:

    # cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem."$(date +"%Y%m%d%H%M%S")"
    # openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
  4. Replace the existing certificate with the new certificate:

    # mv /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/ca.pem
  5. Define the certificates that should be re-signed:

    # names="engine apache websocket-proxy jboss imageio-proxy"

    If you replaced the oVirt Engine SSL Certificate after the upgrade, run the following instead:

    # names="engine websocket-proxy jboss imageio-proxy"

    For more details see Replacing the oVirt Engine CA Certificate in the Administration Guide.

  6. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  7. Restart the following services:

    # systemctl restart httpd
    # systemctl restart ovirt-engine
    # systemctl restart ovirt-websocket-proxy
    # systemctl restart ovirt-imageio
  8. Connect to the Administration Portal to confirm that the warning no longer appears.

  9. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

  10. Enroll the certificates on the hosts. Repeat the following procedure for each host.

    1. In the Administration Portal, click Compute  Hosts.

    2. Select the host and click Management  Maintenance and OK.

    3. Once the host is in maintenance mode, click Installation  Enroll Certificate.

    4. Click Management  Activate.

2. Upgrading from 4.3 to oVirt 4.4

Upgrading your environment from 4.3 to 4.4 involves the following steps:

2.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

You can now update the Engine to the latest version of 4.3.

2.2. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

You can now upgrade the Engine to 4.4.

2.3. Upgrading the oVirt Engine from 4.3 to 4.4

oVirt Engine 4.4 is only supported on Enterprise Linux 8.2 or later. You need to do a clean installation of Enterprise Linux 8.2 and oVirt Engine 4.4, even if you are using the same physical machine that you use to run oVirt Engine 4.3.

The upgrade process requires restoring oVirt Engine 4.3 backup files onto the oVirt Engine 4.4 machine.

Prerequisites
  • All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

  • If you use an external CA to sign HTTPS certificates, follow the steps in Replacing the oVirt Engine CA Certificate in the Administration Guide. The backup and restore include the 3rd-party certificate, so you should be able to log in to the Administration portal after the upgrade. Ensure the CA certificate is added to system-wide trust stores of all clients to ensure the foreign menu of virt-viewer works. See BZ#1313379 for more information.

Connected hosts and virtual machines can continue to work while the Engine is being upgraded.

Procedure
  1. Log in to the Engine machine.

  2. Back up the oVirt Engine 4.3 environment.

    # engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log
  3. Copy the backup file to a storage device outside of the oVirt environment.

  4. Install Enterprise Linux 8.2 or later. See Performing a standard RHEL installation for more information.

  5. Complete the steps to install oVirt Engine 4.4, including running the command yum install rhvm, but do not run engine-setup. See one of the Installing oVirt guides for more information.

  6. Copy the backup file to the oVirt Engine 4.4 machine and restore it.

    # engine-backup --mode=restore --file=backup.bck --provision-all-databases

    If the backup contained grants for extra database users, this command creates the extra users with random passwords. You must change these passwords manually if the extra users require access to the restored system. See https://access.redhat.com/articles/2686731.

  7. Install optional extension packages if they were installed on the oVirt Engine 4.3 machine.

    # yum install ovirt-engine-extension-aaa-ldap ovirt-engine-extension-aaa-misc ovirt-engine-extension-logger-log4j

    The configuration for these package extensions must be manually reapplied because they are not migrated as part of the backup and restore process.

  8. Decommission the oVirt Engine 4.3 machine if a different machine is used for oVirt Engine 4.4. Two different Engines must not manage the same hosts or storage.

  9. Run engine-setup to configure the Engine.

    # engine-setup

The oVirt Engine 4.4 is now installed, with the cluster compatibility version set to 4.2 or 4.3, whichever was the preexisting cluster compatibility version. Now you need to upgrade the hosts in your environment to oVirt 4.4, after which you can change the cluster compatibility version to 4.4.

You can now update the hosts.

2.4. Migrating hosts and virtual machines from oVirt 4.3 to 4.4

You can migrate hosts and virtual machines from oVirt 4.3 to 4.4 such that you minimize the downtime of virtual machines in your environment.

This process requires migrating all virtual machines from one host so as to make that host available to upgrade to oVirt 4.4. After the upgrade, you can reattach the host to the Engine.

When installing or reinstalling the host’s operating system, oVirt strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss.

Prerequisites
  • Hosts for oVirt 4.4 require Enterprise Linux 8.2 or later. A clean installation of Enterprise Linux 8.2 or later, or oVirt Node 4.4 is required, even if you are using the same physical machine that you use to run hosts for oVirt 4.3.

  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the hosts belong is set to 4.2 or 4.3. All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

Procedure
  1. Pick a host to upgrade and migrate that host’s virtual machines to another host in the same cluster. You can use Live Migration to minimize virtual machine downtime. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide.

  2. Put the host into maintenance mode and remove the host from the Engine. For more information, see Removing a Host in the Administration Guide.

  3. Install Enterprise Linux 8.2 or later, or oVirt Node 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  4. Install the appropriate packages to enable the host for oVirt 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  5. Add this host to the Engine, assigning it to the same cluster. You can now migrate virtual machines onto this host. For more information, see Adding Standard Hosts to the Engine in one of the Installing oVirt guides.

Repeat these steps to migrate virtual machines and upgrade hosts for the rest of the hosts in the same cluster, one by one, until all are running oVirt 4.4.

2.5. Upgrading oVirt Node while preserving local storage

Environments with local storage cannot migrate virtual machines to a host in another cluster because the local storage is not shared with other storage domains. To upgrade oVirt Node 4.3 hosts that have a local storage domain, reinstall the host while preserving the local storage, create a new local storage domain in the 4.4 environment, and import the previous local storage into the new domain.

Prerequisites
  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the host belongs is set to 4.2 or 4.3.

Procedure
  1. Ensure that the local storage on the oVirt Node 4.3 host’s local storage is in maintenance mode before starting this process. Complete these steps:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the Details pane and select the storage domain in the results list.

    3. Click Maintenance.

  2. Reinstall the oVirt Node, as described in Installing oVirt Node in the Installation Guide.

    When selecting the device on which to install oVirt Node from the Installation Destination screen, do not select the device(s) storing the virtual machines. Only select the device where the operating system should be installed.

    If you are using Kickstart to install the host, ensure that you preserve the devices containing the virtual machines by adding the following to the Kickstart file, replacing `device` with the relevant device.

    # clearpart --all --drives=device

    For more information on using Kickstart, see Kickstart references in Red Hat Enterprise Linux 8 Performing an advanced RHEL installation.

  3. On the reinstalled host, create a directory, for example /data in which to recover the previous environment.

    # mkdir /data
  4. Mount the previous local storage in the new directory. In our example, /dev/sdX1 is the local storage:

    # mount /dev/sdX1 /data
  5. Set the following permissions for the new directory.

    # chown -R 36:36 /data
    # chmod -R 0755 /data
  6. oVirt recommends that you also automatically mount the local storage via /etc/fstab in case the server requires a reboot:

    # blkid | grep -i sdX1
    /dev/sdX1: UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" TYPE="ext4"
    # vi /etc/fstab
    UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" /data    ext4    defaults     0       0
  7. In the Administration Portal, create a data center and select Local in the Storage Type drop-down menu.

  8. Configure a cluster on the new data center. See Creating a New Cluster in the Administration Guide for more information.

  9. Add the host to the Engine. See Adding Standard Hosts to the oVirt Manager in one of the Installing oVirt guides for more information.

  10. On the host, create a new directory that will be used to create the initial local storage domain. For example:

    # mkdir -p /localfs
    # chown 36:36 /localfs
    # chmod -R 0755 /localfs
  11. In the Administration Portal, open the Storage tab and click New Domain to create a new local storage domain.

  12. Set the name to localfs and set the path to /localfs.

  13. Once the local storage is active, click Import Domain and set the domain’s details. For example, define Data as the name, Local on Host as the storage type and /data as the path.

  14. Click OK to confirm the message that appears informing you that storage domains are already attached to the data center.

  15. Activate the new storage domain:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the details pane and select the new data storage domain in the results list.

    3. Click Activate.

  16. Once the new storage domain is active, import the virtual machines and their disks:

    1. In the Storage tab, select data.

    2. Select the VM Import tab in the details pane, select the virtual machines and click Import. See Importing Virtual Machines from a Data Domain in the Virtual Machine Management Guide for more details.

  17. Once you have ensured that all virtual machines have been successfully imported and are functioning properly, you can move localfs to maintenance mode.

  18. Click the Storage tab and select localfs from the results list.

    1. Click the Data Center tab in the details pane.

    2. Click Maintenance, then click OK to move the storage domain to maintenance mode.

    3. Click Detach. The Detach Storage confirmation window opens.

    4. Click OK.

You have now upgraded the host to version 4.4, created a new local storage domain, and imported the 4.3 storage domain and its virtual machines.

You can now update the cluster compatibility version.

2.6. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

You can now update the cluster compatibility version for virtual machines in the cluster.

2.7. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

You can now update the data center compatibility version.

2.8. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

Upgrading a standalone Engine remote database environment

3. Upgrading a Remote Database Environment from 4.2 to oVirt 4.3

Upgrading your environment from 4.2 to 4.3 involves the following steps:

3.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

You can now update the Engine to the latest version of 4.2.

3.2. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

3.3. Upgrading remote databases from PostgreSQL 9.5 to 10

oVirt 4.3 uses PostgreSQL 10 instead of PostgreSQL 9.5. If your databases are installed locally, the upgrade script automatically upgrades them from version 9.5 to 10. However, if either of your databases (Engine or Data Warehouse) is installed on a separate machine, you must perform the following procedure on each remote database before upgrading the Engine.

  1. Stop the service running on the machine:

    • When upgrading the Engine database, stop the ovirt-engine service on the Engine machine:

      # systemctl stop ovirt-engine
    • When upgrading the Data Warehouse database, stop the ovirt-engine-dwhd service on the Data Warehouse machine:

      # systemctl stop ovirt-engine-dwhd
  2. Enable the required repository to receive the PostgreSQL 10 package:

  3. Install the PostgreSQL 10 packages:

    # yum install rh-postgresql10 rh-postgresql10-postgresql-contrib
  4. Stop and disable the PostgreSQL 9.5 service:

    # systemctl stop rh-postgresql95-postgresql
    # systemctl disable rh-postgresql95-postgresql
  5. Upgrade the PostgreSQL 9.5 database to PostgreSQL 10:

    # scl enable rh-postgresql10 -- postgresql-setup --upgrade-from=rh-postgresql95-postgresql --upgrade
  6. Start and enable the rh-postgresql10-postgresql.service and check that it is running:

    # systemctl start rh-postgresql10-postgresql.service
    # systemctl enable rh-postgresql10-postgresql.service
    # systemctl status rh-postgresql10-postgresql.service

    Ensure that you see output similar to the following:

    rh-postgresql10-postgresql.service - PostgreSQL database server
       Loaded: loaded (/usr/lib/systemd/system/rh-postgresql10-postgresql.service;
    enabled; vendor preset: disabled)
       Active: active (running) since ...
  7. Copy the pg_hba.conf client configuration file from the PostgreSQL 9.5 environment to the PostgreSQL 10 environment:

    # cp -p /var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf  /var/opt/rh/rh-postgresql10/lib/pgsql/data/pg_hba.conf
  8. Update the following parameters in /var/opt/rh/rh-postgresql10/lib/pgsql/data/postgresql.conf:

    listen_addresses='*'
    autovacuum_vacuum_scale_factor=0.01
    autovacuum_analyze_scale_factor=0.075
    autovacuum_max_workers=6
    maintenance_work_mem=65536
    max_connections=150
    work_mem = 8192
  9. Restart the PostgreSQL 10 service to apply the configuration changes:

    # systemctl restart rh-postgresql10-postgresql.service

You can now upgrade the Engine to 4.3.

3.4. Upgrading the Engine from 4.2 to 4.3

Follow these same steps when upgrading any of the following:

  • the oVirt Engine

  • a remote machine with the Data Warehouse service

You need to be logged into the machine that you are upgrading.

If the upgrade fails, the engine-setup command attempts to restore your oVirt Engine installation to its previous state. For this reason, do not remove the previous version’s repositories until after the upgrade is complete. If the upgrade fails, the engine-setup script explains how to restore your installation.

Procedure
  1. Enable the oVirt 4.3 repositories:

    All other repositories remain the same across oVirt releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Run engine-setup and follow the prompts to upgrade the oVirt Engine, the remote database or remote service:

    # engine-setup

    During the upgrade process for the Engine, the engine-setup script might prompt you to disconnect the remote Data Warehouse database. You must disconnect it to continue the setup.

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
  4. Update the base operating system:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the upgrade.

The Engine is now upgraded to version 4.3.

3.4.1. Completing the remote Data Warehouse database upgrade

Complete these additional steps when upgrading a remote Data Warehouse database from PostgreSQL 9.5 to 10.

Procedure
  1. The ovirt-engine-dwhd service is now running on the Engine machine. If the ovirt-engine-dwhd service is on a remote machine, stop and disable the ovirt-engine-dwhd service on the Engine machine, and remove the configuration files that engine-setup created:

    # systemctl stop ovirt-engine-dwhd
    # systemctl disable ovirt-engine-dwhd
    # rm -f /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/*
  2. Repeat the steps in Upgrading the Engine from 4.2 to 4.3 on the machine hosting the ovirt-engine-dwhd service.

You can now update the hosts.

3.5. Updating All Hosts in a Cluster

You can update all hosts in a cluster instead of updating hosts individually. This is particularly useful during upgrades to new versions of oVirt. See https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/cluster_upgrade/README.md for more information about the Ansible role used to automate the updates.

Update one cluster at a time.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines are shut down during the update, unless you choose to skip that host instead.

Procedure
  1. In the Administration Portal, click Compute  Clusters and select the cluster. The Upgrade status column shows if an upgrade is available for any hosts in the cluster.

  2. Click Upgrade.

  3. Select the hosts to update, then click Next.

  4. Configure the options:

    • Stop Pinned VMs shuts down any virtual machines that are pinned to hosts in the cluster, and is selected by default. You can clear this check box to skip updating those hosts so that the pinned virtual machines stay running, such as when a pinned virtual machine is running important services or processes and you do not want it to shut down at an unknown time during the update.

    • Upgrade Timeout (Minutes) sets the time to wait for an individual host to be updated before the cluster upgrade fails with a timeout. The default is 60. You can increase it for large clusters where 60 minutes might not be enough, or reduce it for small clusters where the hosts update quickly.

    • Check Upgrade checks each host for available updates before running the upgrade process. It is not selected by default, but you can select it if you need to ensure that recent updates are included, such as when you have configured the Engine to check for host updates less frequently than the default.

    • Reboot After Upgrade reboots each host after it is updated, and is selected by default. You can clear this check box to speed up the process if you are sure that there are no pending updates that require a host reboot.

    • Use Maintenance Policy sets the cluster’s scheduling policy to cluster_maintenance during the update. It is selected by default, so activity is limited and virtual machines cannot start unless they are highly available. You can clear this check box if you have a custom scheduling policy that you want to keep using during the update, but this could have unknown consequences. Ensure your custom policy is compatible with cluster upgrade activity before disabling this option.

  5. Click Next.

  6. Review the summary of the hosts and virtual machines that will be affected.

  7. Click Upgrade.

You can track the progress of host updates:

  • in the Compute  Clusters view, the Upgrade Status column shows Upgrade in progress.

  • in the Compute  Hosts view

  • in the Events section of the Notification Drawer (EventsIcon).

You can track the progress of individual virtual machine migrations in the Status column of the Compute  Virtual Machines view. In large environments, you may need to filter the results to show a particular group of virtual machines.

3.6. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

3.7. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

3.8. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

If you previously upgraded to 4.2 without replacing SHA-1 certificates with SHA-256 certificates, you must do so now.

3.9. Replacing SHA-1 Certificates with SHA-256 Certificates

oVirt 4.4 uses SHA-256 signatures, which provide a more secure way to sign SSL certificates than SHA-1. Newly installed systems do not require any special steps to enable oVirt’s public key infrastructure (PKI) to use SHA-256 signatures.

Preventing Warning Messages from Appearing in the Browser

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Define the certificate that should be re-signed:

    # names="apache"
  4. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  5. Restart the httpd service:

    # systemctl restart httpd
  6. Connect to the Administration Portal to confirm that the warning no longer appears.

  7. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

Replacing All Signed Certificates with SHA-256

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Re-sign the CA certificate by backing it up and creating a new certificate in ca.pem.new:

    # cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem."$(date +"%Y%m%d%H%M%S")"
    # openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
  4. Replace the existing certificate with the new certificate:

    # mv /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/ca.pem
  5. Define the certificates that should be re-signed:

    # names="engine apache websocket-proxy jboss imageio-proxy"

    If you replaced the oVirt Engine SSL Certificate after the upgrade, run the following instead:

    # names="engine websocket-proxy jboss imageio-proxy"

    For more details see Replacing the oVirt Engine CA Certificate in the Administration Guide.

  6. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  7. Restart the following services:

    # systemctl restart httpd
    # systemctl restart ovirt-engine
    # systemctl restart ovirt-websocket-proxy
    # systemctl restart ovirt-imageio
  8. Connect to the Administration Portal to confirm that the warning no longer appears.

  9. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

  10. Enroll the certificates on the hosts. Repeat the following procedure for each host.

    1. In the Administration Portal, click Compute  Hosts.

    2. Select the host and click Management  Maintenance and OK.

    3. Once the host is in maintenance mode, click Installation  Enroll Certificate.

    4. Click Management  Activate.

4. Upgrading a Remote Database Environment from 4.3 to oVirt 4.4

Upgrading your environment from 4.3 to 4.4 involves the following steps:

4.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

You can now update the Engine to the latest version of 4.3.

4.2. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

You can now upgrade the Engine to 4.4.

4.3. Upgrading the oVirt Engine from 4.3 to 4.4

oVirt Engine 4.4 is only supported on Enterprise Linux 8.2 or later. You need to do a clean installation of Enterprise Linux 8.2 and oVirt Engine 4.4, even if you are using the same physical machine that you use to run oVirt Engine 4.3.

The upgrade process requires restoring oVirt Engine 4.3 backup files onto the oVirt Engine 4.4 machine.

Prerequisites
  • All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

  • If you use an external CA to sign HTTPS certificates, follow the steps in Replacing the oVirt Engine CA Certificate in the Administration Guide. The backup and restore include the 3rd-party certificate, so you should be able to log in to the Administration portal after the upgrade. Ensure the CA certificate is added to system-wide trust stores of all clients to ensure the foreign menu of virt-viewer works. See BZ#1313379 for more information.

Connected hosts and virtual machines can continue to work while the Engine is being upgraded.

Procedure
  1. Log in to the Engine machine.

  2. Back up the oVirt Engine 4.3 environment.

    # engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log
  3. Copy the backup file to a storage device outside of the oVirt environment.

  4. Install Enterprise Linux 8.2 or later. See Performing a standard RHEL installation for more information.

  5. Complete the steps to install oVirt Engine 4.4, including running the command yum install rhvm, but do not run engine-setup. See one of the Installing oVirt guides for more information.

  6. Copy the backup file to the oVirt Engine 4.4 machine and restore it.

    # engine-backup --mode=restore --file=backup.bck --provision-all-databases

    If the backup contained grants for extra database users, this command creates the extra users with random passwords. You must change these passwords manually if the extra users require access to the restored system. See https://access.redhat.com/articles/2686731.

  7. Install optional extension packages if they were installed on the oVirt Engine 4.3 machine.

    # yum install ovirt-engine-extension-aaa-ldap ovirt-engine-extension-aaa-misc ovirt-engine-extension-logger-log4j

    The configuration for these package extensions must be manually reapplied because they are not migrated as part of the backup and restore process.

  8. Decommission the oVirt Engine 4.3 machine if a different machine is used for oVirt Engine 4.4. Two different Engines must not manage the same hosts or storage.

The oVirt Engine 4.4 is now installed, with the cluster compatibility version set to 4.2 or 4.3, whichever was the preexisting cluster compatibility version.

Now you need to upgrade the remote databases in your environment.

4.4. Upgrading Remote Databases from PostgreSQL 10 to 12

oVirt 4.4 uses PostgreSQL 12, while oVirt 4.3 uses PostgreSQL 10. If either of your databases (Engine or Data Warehouse) is installed on a separate machine, you must perform the following procedure on each remote database before upgrading the Engine.

This procedure uses the pg_dump and pg_restore commands. For a complete list of options for these commands, enter pg_dump --help or pg_restore --help on a machine with PostgreSQL installed.

  1. Back up all remote databases. Use the following command on each remote database machine:

    # pg_dump -f backup-file database-name

    The default name for the engine database is engine, and the default name for the Data Warehouse database is ovirt-engine-history.

    For example, on the machine hosting the Data Warehouse database, enter a command similar to the following:

    # pg_dump -f dwh-backup.dump ovirt-engine-history
  2. Copy the backup files to an external location such as a USB drive.

    This process requires installing Enterprise Linux 8 on the Engine machine and all hosts, so make sure that you copy the backup file to a safe location.

  3. Install Enterprise Linux 8 on each database server, with the oVirt repository enabled.

    For detailed installation instructions, see Performing a standard EL installation.

    The server must meet the minimum Engine requirements.

  4. Enable the correct repositories on each database server. For detailed instructions, see Enabling the oVirt Engine Repositories for oVirt 4.4.

    Updates to the oVirt Engine are released through the Content Delivery Network.

  5. Install the PostgreSQL server package:

    # yum install postgresql postgresql-server postgresql-contrib
  6. Restore the backup on the data warehouse database machine and the engine database machine:

    # pg_restore -d database-name backup-file
  7. On the Engine machine, run engine-setup to configure the Engine:

    # engine-setup

You can now update the hosts.

4.5. Migrating hosts and virtual machines from oVirt 4.3 to 4.4

You can migrate hosts and virtual machines from oVirt 4.3 to 4.4 such that you minimize the downtime of virtual machines in your environment.

This process requires migrating all virtual machines from one host so as to make that host available to upgrade to oVirt 4.4. After the upgrade, you can reattach the host to the Engine.

When installing or reinstalling the host’s operating system, oVirt strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss.

Prerequisites
  • Hosts for oVirt 4.4 require Enterprise Linux 8.2 or later. A clean installation of Enterprise Linux 8.2 or later, or oVirt Node 4.4 is required, even if you are using the same physical machine that you use to run hosts for oVirt 4.3.

  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the hosts belong is set to 4.2 or 4.3. All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

Procedure
  1. Pick a host to upgrade and migrate that host’s virtual machines to another host in the same cluster. You can use Live Migration to minimize virtual machine downtime. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide.

  2. Put the host into maintenance mode and remove the host from the Engine. For more information, see Removing a Host in the Administration Guide.

  3. Install Enterprise Linux 8.2 or later, or oVirt Node 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  4. Install the appropriate packages to enable the host for oVirt 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  5. Add this host to the Engine, assigning it to the same cluster. You can now migrate virtual machines onto this host. For more information, see Adding Standard Hosts to the Engine in one of the Installing oVirt guides.

Repeat these steps to migrate virtual machines and upgrade hosts for the rest of the hosts in the same cluster, one by one, until all are running oVirt 4.4.

4.6. Upgrading oVirt Node while preserving local storage

Environments with local storage cannot migrate virtual machines to a host in another cluster because the local storage is not shared with other storage domains. To upgrade oVirt Node 4.3 hosts that have a local storage domain, reinstall the host while preserving the local storage, create a new local storage domain in the 4.4 environment, and import the previous local storage into the new domain.

Prerequisites
  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the host belongs is set to 4.2 or 4.3.

Procedure
  1. Ensure that the local storage on the oVirt Node 4.3 host’s local storage is in maintenance mode before starting this process. Complete these steps:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the Details pane and select the storage domain in the results list.

    3. Click Maintenance.

  2. Reinstall the oVirt Node, as described in Installing oVirt Node in the Installation Guide.

    When selecting the device on which to install oVirt Node from the Installation Destination screen, do not select the device(s) storing the virtual machines. Only select the device where the operating system should be installed.

    If you are using Kickstart to install the host, ensure that you preserve the devices containing the virtual machines by adding the following to the Kickstart file, replacing `device` with the relevant device.

    # clearpart --all --drives=device

    For more information on using Kickstart, see Kickstart references in Red Hat Enterprise Linux 8 Performing an advanced RHEL installation.

  3. On the reinstalled host, create a directory, for example /data in which to recover the previous environment.

    # mkdir /data
  4. Mount the previous local storage in the new directory. In our example, /dev/sdX1 is the local storage:

    # mount /dev/sdX1 /data
  5. Set the following permissions for the new directory.

    # chown -R 36:36 /data
    # chmod -R 0755 /data
  6. oVirt recommends that you also automatically mount the local storage via /etc/fstab in case the server requires a reboot:

    # blkid | grep -i sdX1
    /dev/sdX1: UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" TYPE="ext4"
    # vi /etc/fstab
    UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" /data    ext4    defaults     0       0
  7. In the Administration Portal, create a data center and select Local in the Storage Type drop-down menu.

  8. Configure a cluster on the new data center. See Creating a New Cluster in the Administration Guide for more information.

  9. Add the host to the Engine. See Adding Standard Hosts to the oVirt Manager in one of the Installing oVirt guides for more information.

  10. On the host, create a new directory that will be used to create the initial local storage domain. For example:

    # mkdir -p /localfs
    # chown 36:36 /localfs
    # chmod -R 0755 /localfs
  11. In the Administration Portal, open the Storage tab and click New Domain to create a new local storage domain.

  12. Set the name to localfs and set the path to /localfs.

  13. Once the local storage is active, click Import Domain and set the domain’s details. For example, define Data as the name, Local on Host as the storage type and /data as the path.

  14. Click OK to confirm the message that appears informing you that storage domains are already attached to the data center.

  15. Activate the new storage domain:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the details pane and select the new data storage domain in the results list.

    3. Click Activate.

  16. Once the new storage domain is active, import the virtual machines and their disks:

    1. In the Storage tab, select data.

    2. Select the VM Import tab in the details pane, select the virtual machines and click Import. See Importing Virtual Machines from a Data Domain in the Virtual Machine Management Guide for more details.

  17. Once you have ensured that all virtual machines have been successfully imported and are functioning properly, you can move localfs to maintenance mode.

  18. Click the Storage tab and select localfs from the results list.

    1. Click the Data Center tab in the details pane.

    2. Click Maintenance, then click OK to move the storage domain to maintenance mode.

    3. Click Detach. The Detach Storage confirmation window opens.

    4. Click OK.

You have now upgraded the host to version 4.4, created a new local storage domain, and imported the 4.3 storage domain and its virtual machines.

You can now update the cluster compatibility version.

4.7. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

You can now update the cluster compatibility version for virtual machines in the cluster.

4.8. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

You can now update the data center compatibility version.

4.9. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

Upgrading a self-hosted engine environment

5. Upgrading a Self-Hosted Engine from 4.2 to oVirt 4.3

Upgrading a self-hosted engine environment from version 4.2 to 4.3 involves the following steps:

5.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

5.2. Enabling Global Maintenance Mode

You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Engine virtual machine.

Procedure
  1. Log in to one of the self-hosted engine nodes and enable global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. Confirm that the environment is in maintenance mode before proceeding:

    # hosted-engine --vm-status

    You should see a message indicating that the cluster is in maintenance mode.

5.3. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

5.4. Upgrading the Engine from 4.2 to 4.3

You need to be logged into the machine that you are upgrading.

If the upgrade fails, the engine-setup command attempts to restore your oVirt Engine installation to its previous state. For this reason, do not remove the previous version’s repositories until after the upgrade is complete. If the upgrade fails, the engine-setup script explains how to restore your installation.

Procedure
  1. Enable the oVirt 4.3 repositories:

    All other repositories remain the same across oVirt releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Run engine-setup and follow the prompts to upgrade the oVirt Engine:

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
  4. Update the base operating system:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the upgrade.

The Engine is now upgraded to version 4.3.

5.5. Disabling Global Maintenance Mode

Procedure
  1. Log in to the Engine virtual machine and shut it down.

  2. Log in to one of the self-hosted engine nodes and disable global maintenance mode:

    # hosted-engine --set-maintenance --mode=none

    When you exit global maintenance mode, ovirt-ha-agent starts the Engine virtual machine, and then the Engine automatically starts. It can take up to ten minutes for the Engine to start.

  3. Confirm that the environment is running:

    # hosted-engine --vm-status

    The listed information includes Engine Status. The value for Engine status should be:

    {"health": "good", "vm": "up", "detail": "Up"}

    When the virtual machine is still booting and the Engine hasn’t started yet, the Engine status is:

    {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}

    If this happens, wait a few minutes and try again.

You can now update the self-hosted engine nodes, and then any standard hosts. The procedure is the same for both host types.

5.6. Updating All Hosts in a Cluster

You can update all hosts in a cluster instead of updating hosts individually. This is particularly useful during upgrades to new versions of oVirt. See https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/cluster_upgrade/README.md for more information about the Ansible role used to automate the updates.

Update one cluster at a time.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines are shut down during the update, unless you choose to skip that host instead.

Procedure
  1. In the Administration Portal, click Compute  Clusters and select the cluster. The Upgrade status column shows if an upgrade is available for any hosts in the cluster.

  2. Click Upgrade.

  3. Select the hosts to update, then click Next.

  4. Configure the options:

    • Stop Pinned VMs shuts down any virtual machines that are pinned to hosts in the cluster, and is selected by default. You can clear this check box to skip updating those hosts so that the pinned virtual machines stay running, such as when a pinned virtual machine is running important services or processes and you do not want it to shut down at an unknown time during the update.

    • Upgrade Timeout (Minutes) sets the time to wait for an individual host to be updated before the cluster upgrade fails with a timeout. The default is 60. You can increase it for large clusters where 60 minutes might not be enough, or reduce it for small clusters where the hosts update quickly.

    • Check Upgrade checks each host for available updates before running the upgrade process. It is not selected by default, but you can select it if you need to ensure that recent updates are included, such as when you have configured the Engine to check for host updates less frequently than the default.

    • Reboot After Upgrade reboots each host after it is updated, and is selected by default. You can clear this check box to speed up the process if you are sure that there are no pending updates that require a host reboot.

    • Use Maintenance Policy sets the cluster’s scheduling policy to cluster_maintenance during the update. It is selected by default, so activity is limited and virtual machines cannot start unless they are highly available. You can clear this check box if you have a custom scheduling policy that you want to keep using during the update, but this could have unknown consequences. Ensure your custom policy is compatible with cluster upgrade activity before disabling this option.

  5. Click Next.

  6. Review the summary of the hosts and virtual machines that will be affected.

  7. Click Upgrade.

You can track the progress of host updates:

  • in the Compute  Clusters view, the Upgrade Status column shows Upgrade in progress.

  • in the Compute  Hosts view

  • in the Events section of the Notification Drawer (EventsIcon).

You can track the progress of individual virtual machine migrations in the Status column of the Compute  Virtual Machines view. In large environments, you may need to filter the results to show a particular group of virtual machines.

5.7. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

5.8. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

The Engine virtual machine does not need to be rebooted.

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

5.9. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

If you previously upgraded to 4.2 without replacing SHA-1 certificates with SHA-256 certificates, you must do so now.

5.10. Replacing SHA-1 Certificates with SHA-256 Certificates

oVirt 4.4 uses SHA-256 signatures, which provide a more secure way to sign SSL certificates than SHA-1. Newly installed systems do not require any special steps to enable oVirt’s public key infrastructure (PKI) to use SHA-256 signatures.

Preventing Warning Messages from Appearing in the Browser

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Define the certificate that should be re-signed:

    # names="apache"
  4. Log in to one of the self-hosted engine nodes and enable global maintenance:

    # hosted-engine --set-maintenance --mode=global
  5. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  6. Restart the httpd service:

    # systemctl restart httpd
  7. Log in to one of the self-hosted engine nodes and disable global maintenance:

    # hosted-engine --set-maintenance --mode=none
  8. Connect to the Administration Portal to confirm that the warning no longer appears.

  9. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

Replacing All Signed Certificates with SHA-256

  1. Log in to the Engine machine as the root user.

  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Re-sign the CA certificate by backing it up and creating a new certificate in ca.pem.new:

    # cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem."$(date +"%Y%m%d%H%M%S")"
    # openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
  4. Replace the existing certificate with the new certificate:

    # mv /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/ca.pem
  5. Define the certificates that should be re-signed:

    # names="engine apache websocket-proxy jboss imageio-proxy"

    If you replaced the oVirt Engine SSL Certificate after the upgrade, run the following instead:

    # names="engine websocket-proxy jboss imageio-proxy"

    For more details see Replacing the oVirt Engine CA Certificate in the Administration Guide.

  6. Log in to one of the self-hosted engine nodes and enable global maintenance:

    # hosted-engine --set-maintenance --mode=global
  7. On the Engine, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
  8. Restart the following services:

    # systemctl restart httpd
    # systemctl restart ovirt-engine
    # systemctl restart ovirt-websocket-proxy
    # systemctl restart ovirt-imageio
  9. Log in to one of the self-hosted engine nodes and disable global maintenance:

    # hosted-engine --set-maintenance --mode=none
  10. Connect to the Administration Portal to confirm that the warning no longer appears.

  11. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).

  12. Enroll the certificates on the hosts. Repeat the following procedure for each host.

    1. In the Administration Portal, click Compute  Hosts.

    2. Select the host and click Management  Maintenance and OK.

    3. Once the host is in maintenance mode, click Installation  Enroll Certificate.

    4. Click Management  Activate.

6. Upgrading a self-Hosted engine from 4.3 to oVirt 4.4

Upgrading a self-hosted engine environment from version 4.3 to 4.4 involves the following steps:

6.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.

  • Ensure your environment meets the requirements for oVirt 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.

  • When upgrading oVirt Engine, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

6.2. Migrating virtual machines from the self-hosted engine host

Only the Engine virtual machine should remain on the host until after you have finished upgrading the host. Migrate any virtual machines other than the Engine virtual machine to another host in the same cluster.

You can use Live Migration to minimize virtual machine down-time. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide for more information.

6.3. Enabling Global Maintenance Mode

You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Engine virtual machine.

Procedure
  1. Log in to one of the self-hosted engine nodes and enable global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. Confirm that the environment is in maintenance mode before proceeding:

    # hosted-engine --vm-status

    You should see a message indicating that the cluster is in maintenance mode.

You can now update the Engine to the latest version of 4.3.

6.4. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

You can now upgrade the Engine to 4.4.

6.5. Upgrading the oVirt Engine from 4.3 to 4.4

The oVirt Engine 4.4 is only supported on Enterprise Linux 8.2 or later. You need to do a clean installation of Enterprise Linux 8.2 or oVirt Node on the self-hosted engine host, even if you are using the same physical machine that you use to run the oVirt 4.3 self-hosted engine.

The upgrade process requires restoring oVirt Engine 4.3 backup files onto the oVirt Engine 4.4 virtual machine.

Prerequisites
  • All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

  • Make note of the MAC address of the self-hosted engine if you are using DHCP and want to use the same IP address. The deploy script prompts you for this information.

  • During the deployment you need to provide a new storage domain for the Engine machine. The deployment script renames the 4.3 storage domain and retains its data to enable disaster recovery.

    In an environment with multiple highly available self-hosted engine nodes, you need to detach the storage domain hosting the version 4.3 Engine after upgrading the Engine to 4.4. Use a dedicated storage domain for the 4.4 self-hosted engine deployment.

  • If you use an external CA to sign HTTPS certificates, follow the steps in Replacing the oVirt Engine CA Certificate in the Administration Guide. The backup and restore include the 3rd-party certificate, so you should be able to log in to the Administration portal after the upgrade. Ensure the CA certificate is added to system-wide trust stores of all clients to ensure the foreign menu of virt-viewer works. See BZ#1313379 for more information.

Connected hosts and virtual machines can continue to work while the Engine is being upgraded.

Procedure
  1. Log in to the Engine virtual machine and shut down the engine service.

    # systemctl stop ovirt-engine
  2. Back up the oVirt Engine 4.3 environment.

    # engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log
  3. Copy the backup file to a storage device outside of the oVirt environment.

  4. Install oVirt Node 4.4 or Enterprise Linux 8.2 or later on the existing node currently running the Engine virtual machine to use it as the self-hosted engine deployment host. See Installing the Self-hosted Engine Deployment Host for more information.

    It is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

  5. Install the self-hosted engine deployment tool.

    # yum install ovirt-hosted-engine-setup
  6. Copy the backup file to the host.

  7. Log in to the Engine host and deploy the self-hosted engine with the backup file:

    # hosted-engine --deploy --restore-from-file=/path/backup.bck

    tmux enables the deployment script to continue if the connection to the server is interrupted, so you can reconnect and attach to the deployment and continue. Otherwise, if the connection is interrupted during deployment, the deployment fails.

    To run the deployment script using tmux, enter the tmux command before you run the deployment script:

    # tmux
    # hosted-engine --deploy --restore-from-file=backup.bck

    The deployment script automatically disables global maintenance mode and calls the HA agent to start the self-hosted engine virtual machine. The upgraded host with the 4.4 self-hosted engine reports that HA mode is active, but the other hosts report that global maintenance mode is still enabled as they are still connected to the old self-hosted engine storage.

  8. Detach the storage domain that hosts the Engine 4.3 machine. For details, see Detaching a Storage Domain from a Data Center in the Administration Guide.

  9. Log in to the Engine virtual machine and shut down the engine service.

    # systemctl stop ovirt-engine
  10. Install optional extension packages if they were installed on the oVirt Engine 4.3 machine.

    # yum install ovirt-engine-extension-aaa-ldap ovirt-engine-extension-aaa-misc ovirt-engine-extension-logger-log4j

    The configuration for these package extensions must be manually reapplied because they are not migrated as part of the backup and restore process.

The oVirt Engine 4.4 is now installed, with the cluster compatibility version set to 4.2 or 4.3, whichever was the preexisting cluster compatibility version.

You can now update the self-hosted engine nodes, and then any standard hosts. The procedure is the same for both host types.

6.6. Migrating hosts and virtual machines from oVirt 4.3 to 4.4

You can migrate hosts and virtual machines from oVirt 4.3 to 4.4 such that you minimize the downtime of virtual machines in your environment.

This process requires migrating all virtual machines from one host so as to make that host available to upgrade to oVirt 4.4. After the upgrade, you can reattach the host to the Engine.

When installing or reinstalling the host’s operating system, oVirt strongly recommends that you first detach any existing non-OS storage that is attached to the host to avoid accidental initialization of these disks, and with that, potential data loss.

Prerequisites
  • Hosts for oVirt 4.4 require Enterprise Linux 8.2 or later. A clean installation of Enterprise Linux 8.2 or later, or oVirt Node 4.4 is required, even if you are using the same physical machine that you use to run hosts for oVirt 4.3.

  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the hosts belong is set to 4.2 or 4.3. All data centers and clusters in the environment must have the cluster compatibility level set to version 4.2 or 4.3 before you start the procedure.

Procedure
  1. Pick a host to upgrade and migrate that host’s virtual machines to another host in the same cluster. You can use Live Migration to minimize virtual machine downtime. For more information, see Migrating Virtual Machines Between Hosts in the Virtual Machine Management Guide.

  2. Put the host into maintenance mode and remove the host from the Engine. For more information, see Removing a Host in the Administration Guide.

  3. Install Enterprise Linux 8.2 or later, or oVirt Node 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  4. Install the appropriate packages to enable the host for oVirt 4.4. For more information, see Installing Hosts for oVirt in one of the Installing oVirt guides.

  5. Add this host to the Engine, assigning it to the same cluster. You can now migrate virtual machines onto this host. For more information, see Adding Standard Hosts to the Engine in one of the Installing oVirt guides.

Repeat these steps to migrate virtual machines and upgrade hosts for the rest of the hosts in the same cluster, one by one, until all are running oVirt 4.4.

6.7. Upgrading oVirt Node while preserving local storage

Environments with local storage cannot migrate virtual machines to a host in another cluster because the local storage is not shared with other storage domains. To upgrade oVirt Node 4.3 hosts that have a local storage domain, reinstall the host while preserving the local storage, create a new local storage domain in the 4.4 environment, and import the previous local storage into the new domain.

Prerequisites
  • oVirt Engine 4.4 is installed and running.

  • The compatibility level of the data center and cluster to which the host belongs is set to 4.2 or 4.3.

Procedure
  1. Ensure that the local storage on the oVirt Node 4.3 host’s local storage is in maintenance mode before starting this process. Complete these steps:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the Details pane and select the storage domain in the results list.

    3. Click Maintenance.

  2. Reinstall the oVirt Node, as described in Installing oVirt Node in the Installation Guide.

    When selecting the device on which to install oVirt Node from the Installation Destination screen, do not select the device(s) storing the virtual machines. Only select the device where the operating system should be installed.

    If you are using Kickstart to install the host, ensure that you preserve the devices containing the virtual machines by adding the following to the Kickstart file, replacing `device` with the relevant device.

    # clearpart --all --drives=device

    For more information on using Kickstart, see Kickstart references in Red Hat Enterprise Linux 8 Performing an advanced RHEL installation.

  3. On the reinstalled host, create a directory, for example /data in which to recover the previous environment.

    # mkdir /data
  4. Mount the previous local storage in the new directory. In our example, /dev/sdX1 is the local storage:

    # mount /dev/sdX1 /data
  5. Set the following permissions for the new directory.

    # chown -R 36:36 /data
    # chmod -R 0755 /data
  6. oVirt recommends that you also automatically mount the local storage via /etc/fstab in case the server requires a reboot:

    # blkid | grep -i sdX1
    /dev/sdX1: UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" TYPE="ext4"
    # vi /etc/fstab
    UUID="a81a6879-3764-48d0-8b21-2898c318ef7c" /data    ext4    defaults     0       0
  7. In the Administration Portal, create a data center and select Local in the Storage Type drop-down menu.

  8. Configure a cluster on the new data center. See Creating a New Cluster in the Administration Guide for more information.

  9. Add the host to the Engine. See Adding Standard Hosts to the oVirt Manager in one of the Installing oVirt guides for more information.

  10. On the host, create a new directory that will be used to create the initial local storage domain. For example:

    # mkdir -p /localfs
    # chown 36:36 /localfs
    # chmod -R 0755 /localfs
  11. In the Administration Portal, open the Storage tab and click New Domain to create a new local storage domain.

  12. Set the name to localfs and set the path to /localfs.

  13. Once the local storage is active, click Import Domain and set the domain’s details. For example, define Data as the name, Local on Host as the storage type and /data as the path.

  14. Click OK to confirm the message that appears informing you that storage domains are already attached to the data center.

  15. Activate the new storage domain:

    1. Open the Data Centers tab.

    2. Click the Storage tab in the details pane and select the new data storage domain in the results list.

    3. Click Activate.

  16. Once the new storage domain is active, import the virtual machines and their disks:

    1. In the Storage tab, select data.

    2. Select the VM Import tab in the details pane, select the virtual machines and click Import. See Importing Virtual Machines from a Data Domain in the Virtual Machine Management Guide for more details.

  17. Once you have ensured that all virtual machines have been successfully imported and are functioning properly, you can move localfs to maintenance mode.

  18. Click the Storage tab and select localfs from the results list.

    1. Click the Data Center tab in the details pane.

    2. Click Maintenance, then click OK to move the storage domain to maintenance mode.

    3. Click Detach. The Detach Storage confirmation window opens.

    4. Click OK.

You have now upgraded the host to version 4.4, created a new local storage domain, and imported the 4.3 storage domain and its virtual machines.

6.8. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

6.9. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

The Engine virtual machine does not need to be rebooted.

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

6.10. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

Updates between minor releases

7. Updating oVirt between minor releases

To update from your current version of 4.4 to the latest version of 4.4, update the Engine, update the hosts, and then change the compatibility version for the cluster, virtual machines, and data center.

To update a standalone Engine, follow the standard procedure for minor updates:

7.1. Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated, reboot the machine to complete the update.

7.2. Updating a Self-Hosted Engine

To update a self-hosted engine from your current version to the latest version, you must place the environment in global maintenance mode and then follow the standard procedure for updating between minor versions.

Enabling Global Maintenance Mode

You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Engine virtual machine.

Procedure
  1. Log in to one of the self-hosted engine nodes and enable global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. Confirm that the environment is in maintenance mode before proceeding:

    # hosted-engine --vm-status

    You should see a message indicating that the cluster is in maintenance mode.

Updating the oVirt Engine

Prerequisites
  • The RPM package ovirt-release44.rpm is installed. This package includes the necessary repositories.

Procedure
  1. On the Engine machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    If any kernel packages were updated:

    1. Disable global maintenance mode

    2. Reboot the machine to complete the update.

Related Information

Disabling global maintenance mode

Disabling Global Maintenance Mode

Procedure
  1. Log in to the Engine virtual machine and shut it down.

  2. Log in to one of the self-hosted engine nodes and disable global maintenance mode:

    # hosted-engine --set-maintenance --mode=none

    When you exit global maintenance mode, ovirt-ha-agent starts the Engine virtual machine, and then the Engine automatically starts. It can take up to ten minutes for the Engine to start.

  3. Confirm that the environment is running:

    # hosted-engine --vm-status

    The listed information includes Engine Status. The value for Engine status should be:

    {"health": "good", "vm": "up", "detail": "Up"}

    When the virtual machine is still booting and the Engine hasn’t started yet, the Engine status is:

    {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}

    If this happens, wait a few minutes and try again.

7.3. Updating All Hosts in a Cluster

You can update all hosts in a cluster instead of updating hosts individually. This is particularly useful during upgrades to new versions of oVirt. See https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/cluster_upgrade/README.md for more information about the Ansible role used to automate the updates.

Update one cluster at a time.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines are shut down during the update, unless you choose to skip that host instead.

Procedure
  1. In the Administration Portal, click Compute  Clusters and select the cluster. The Upgrade status column shows if an upgrade is available for any hosts in the cluster.

  2. Click Upgrade.

  3. Select the hosts to update, then click Next.

  4. Configure the options:

    • Stop Pinned VMs shuts down any virtual machines that are pinned to hosts in the cluster, and is selected by default. You can clear this check box to skip updating those hosts so that the pinned virtual machines stay running, such as when a pinned virtual machine is running important services or processes and you do not want it to shut down at an unknown time during the update.

    • Upgrade Timeout (Minutes) sets the time to wait for an individual host to be updated before the cluster upgrade fails with a timeout. The default is 60. You can increase it for large clusters where 60 minutes might not be enough, or reduce it for small clusters where the hosts update quickly.

    • Check Upgrade checks each host for available updates before running the upgrade process. It is not selected by default, but you can select it if you need to ensure that recent updates are included, such as when you have configured the Engine to check for host updates less frequently than the default.

    • Reboot After Upgrade reboots each host after it is updated, and is selected by default. You can clear this check box to speed up the process if you are sure that there are no pending updates that require a host reboot.

    • Use Maintenance Policy sets the cluster’s scheduling policy to cluster_maintenance during the update. It is selected by default, so activity is limited and virtual machines cannot start unless they are highly available. You can clear this check box if you have a custom scheduling policy that you want to keep using during the update, but this could have unknown consequences. Ensure your custom policy is compatible with cluster upgrade activity before disabling this option.

  5. Click Next.

  6. Review the summary of the hosts and virtual machines that will be affected.

  7. Click Upgrade.

You can track the progress of host updates:

  • in the Compute  Clusters view, the Upgrade Status column shows Upgrade in progress.

  • in the Compute  Hosts view

  • in the Events section of the Notification Drawer (EventsIcon).

You can track the progress of individual virtual machine migrations in the Status column of the Compute  Virtual Machines view. In large environments, you may need to filter the results to show a particular group of virtual machines.

You can now update the cluster compatibility version.

7.4. Changing the Cluster Compatibility Version

oVirt clusters have a compatibility version. The cluster compatibility version indicates the features of oVirt supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites
  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations
  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. oVirt recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure
  1. In the Administration Portal, click Compute  Clusters.

  2. Select the cluster to change and click Edit.

  3. On the General tab, change the Compatibility Version to the desired value.

  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

You can now update the cluster compatibility version for virtual machines in the cluster.

7.5. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, instead of from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon (pendingchanges).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure
  1. In the Administration Portal, click Compute  Virtual Machines.

  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

You can now update the data center compatibility version.

7.6. Changing the Data Center Compatibility Version

oVirt data centers have a compatibility version. The compatibility version indicates the version of oVirt with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites
  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure
  1. In the Administration Portal, click Compute  Data Centers.

  2. Select the data center to change and click Edit.

  3. Change the Compatibility Version to the desired value.

  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.

  5. Click OK to confirm.

You can also update hosts individually:

7.7. Updating Individual Hosts

Use the host upgrade manager to update individual hosts directly from the Administration Portal.

The upgrade manager only checks hosts with a status of Up or Non-operational, but not Maintenance.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster. Update a host when its usage is relatively low.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • Do not update all hosts at the same time, as one host must remain available to perform Storage Pool Manager (SPM) tasks.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines must be shut down before updating the host.

Procedure
  1. Ensure that the correct repositories are enabled. To view a list of currently enabled repositories, run dnf repolist.

  2. In the Administration Portal, click Compute  Hosts and select the host to be updated.

  3. Click Installation  Check for Upgrade and click OK.

    Open the Notification Drawer (EventsIcon) and expand the Events section to see the result.

  4. If an update is available, click Installation  Upgrade.

  5. Click OK to update the host. Running virtual machines are migrated according to their migration policy. If migration is disabled for any virtual machines, you are prompted to shut them down.

    The details of the host are updated in Compute  Hosts and the status transitions through these stages:

    Maintenance > Installing > Reboot > Up

    If the update fails, the host’s status changes to Install Failed. From Install Failed you can click Installation  Upgrade again.

Repeat this procedure for each host in the oVirt environment.

You should update the hosts from the Administration Portal. However, you can update the hosts using dnf upgrade instead.

7.8. Manually Updating Hosts

This information is provided for advanced system administrators who need to update hosts manually, but oVirt does not support this method. The procedure described in this topic does not include important steps, including certificate renewal, assuming advanced knowledge of such information. oVirt supports updating hosts using the Administration Portal. For details, see Updating individual hosts or Updating all hosts in a cluster in the Administration Guide.

You can use the dnf command to update your hosts. Update your systems regularly, to ensure timely application of security and bug fixes.

Limitations
  • On oVirt Node, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster. Update a host when its usage is relatively low.

  • In a self-hosted engine environment, the Engine virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.

  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.

  • Do not update all hosts at the same time, as one host must remain available to perform Storage Pool Manager (SPM) tasks.

  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines must be shut down before updating the host.

Procedure
  1. Ensure the correct repositories are enabled. You can check which repositories are currently enabled by running dnf repolist.

  2. In the Administration Portal, click Compute  Hosts and select the host to be updated.

  3. Click Management  Maintenance and OK.

  4. For Enterprise Linux hosts:

    1. Identify the current version of Enterprise Linux:

      # cat /etc/redhat-release
    2. Check which version of the redhat-release package is available:

      # dnf --refresh info --available redhat-release

      This command shows any available updates. For example, when upgrading from Enterprise Linux 8.2.z to 8.3, compare the version of the package with the currently installed version:

      Available Packages
      Name         : redhat-release
      Version      : 8.3
      Release      : 1.0.el8
      …​

      The Enterprise Linux Advanced Virtualization module is usually released later than the Enterprise Linux y-stream. If no new Advanced Virtualization module is available yet, or if there is an error enabling it, stop here and cancel the upgrade. Otherwise you risk corrupting the host.

    3. If the Advanced Virtualization stream is available for Enterprise Linux 8.3 or later, reset the virt module:

      # dnf module reset virt

      If this module is already enabled in the Advanced Virtualization stream, this step is not necessary, but it has no negative impact.

      You can see the value of the stream by entering:

      # dnf module list virt
    4. Enable the virt module in the Advanced Virtualization stream with the following command:

      • For oVirt 4.4.2:

        # dnf module enable virt:8.2
      • For oVirt 4.4.3 to 4.4.5:

        # dnf module enable virt:8.3
      • For oVirt 4.4.6 or later:

        # dnf module enable virt:av

        Starting with EL 8.4, only one Advanced Virtualization stream is used, rhel:av.

  5. Update the host:

    # dnf upgrade --nobest
  6. Reboot the host to ensure all updates are correctly applied.

    Check the imgbased logs to see if any additional package updates have failed for a oVirt Node. If some packages were not successfully reinstalled after the update, check that the packages are listed in /var/imgbased/persisted-rpms. Add any missing packages then run rpm -Uvh /var/imgbased/persisted-rpms/*.

Repeat this process for each host in the oVirt environment.

Appendices

Appendix A: Updating the Local Repository for an Offline oVirt Engine Installation

If your oVirt Engine is hosted on a machine that receives packages via FTP from a local repository, you must regularly synchronize the repository to download package updates from the Content Delivery Network, then update or upgrade that machine. Updated packages address security issues, fix bugs, and add enhancements.

  1. On the system hosting the repository, synchronize the repository to download the most recent version of each available package:

    # reposync --newest-only /var/ftp/pub/ovirtrepo

    This command might download a large number of packages, and take a long time to complete.

  2. Ensure that the repository is available on the Engine machine, and then update or upgrade the machine.

Additional resources

Certain portions of this text first appeared in the Red Hat Virtualization 4.4 Upgrade Guide. Copyright © 2020 Red Hat, Inc. Licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.