Troubleshooting NFS Storage Issues
To attach NFS storage domains to an oVirt environment, the NFS exports must be configured in a specific way. This page is designed to outline the core requirements for configuring NFS exports, and assist in troubleshooting issues encountered when attempting to attach NFS storage domains to an oVirt environment for the first time.
In principle, the user vdsm, with uid 36 and gid 36, must have read and write permissions on all NFS exports. However, some daemons on the hypervisor hosts (for example, sanlock) use a different uid but need access to the directory too. Therefore, all incoming NFS requests must be mapped to the aforementioned uid and gid. Two steps are required to ensure this mapping:
- Change the ownership of the export directory, replacing directory_name with the name of the directory:
# chown 36:36 directory_name
- Add the rw options on the export in
- Set the permissions of the export directory, replacing directory_name with the name of the directory:
# chmod 0755 directory_name
- The NFS server must actually be running.
a. Ensure that the nfs and rpcbind services are running on the NFS server,
b. Ensure that
showmount -e <nfs_server_ip> shows the expected export(s).
- Ensure that selinux is not interfering with NFS access.
getsebool -a | grep virt_use_nfsshould show
virt_use_nfs --> on, if not then do
setsebool virt_use_nfs 1to allow NFS access for VMs
- One can also use
setsebool -P virt_use_nfs 1to make this setting persistent across reboots.
The easiest way to definitively test that an NFS export is ready for use by oVirt is to:
- Create the vdsm user with uid 36 on the ovirt-engine host if it does not already exist.
- Change to the vdsm user using
su - vdsm -s /bin/bash
- Attempt to mount the export on a temporary directory (for example,
/tmpmnt) using the following command form:
$ /usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 *servername:/path/of/export* /tmpmnt
- If the mount succeeds, then try to create a file in it via the
TODO: Update to a current version
The engine-setup command can optionally create an ISO domain and export it.
- Until oVirt 3.3, it always exported to the entire network.
- In 3.4 it prompted the user for an ACL, and the default was still the entire network.
- In 3.5 it still prompts the user, but the default was changed to allow access for the local machine only.
The format for the ACL is simply that of
/etc/exports - see the exports(5) manpage for details. Some simple examples:
- To allow access to 3 hosts host1, host2 and host3, input: host1(rw) host2(rw) host3(rw)
- To allow access to the entire Internet, input: *(rw)
If you use the last example, you must ensure that other means such as a firewall are in place to protect the ISO domain. When configuring this option, also consider protecting the ISO domain from untrusted guests that you might want to run on your hosts.
A new nfs-check script is now available to test whether an NFS export is ready for use by oVirt :
- nfs-check.py is a python script to validate nfs targets to use with oVirt project. Some operations include: mount the nfs target, create a file as vdsm:kvm and then remove it.
- nfs-check.py is available in the vdsm/contrib/ directory of the vdsm source
- Run it on a node via
python nfs-check.py 'server:/target'
$ git clone "https://gerrit.ovirt.org/vdsm" $ cd vdsm/contrib $ python nfs-check.py myNFSServer:/nfsTarget
Setting NFS Server
TODO: Update to current Debian
#> groupadd kvm -g 36 #> useradd vdsm -u 36 -g kvm # mkdir /storage #> chmod 0755 /storage #> chown 36:36 /storage/ #> cat /etc/exports /storage *(rw) #>/etc/init.d/nfs-kernel-server restart
Fedora 26 or higher
#> groupadd kvm -g 36 #> useradd vdsm -u 36 -g kvm # mkdir /storage # chmod 0755 /storage # chown 36:36 /storage/ # yum -y install nfs-utils # cat /etc/exports /storage *(rw) # systemctl start rpcbind.service # systemctl start nfs-server.service # Relevant only for NFS v3 # systemctl start nfs-lock.service # systemctl enable rpcbind.service # systemctl enable nfs-server.service # Relevant only for NFS v3 # systemctl enable nfs-lock.service
RHEL7 based distro
#> groupadd kvm -g 36 #> useradd vdsm -u 36 -g kvm # mkdir /storage # chmod 0755 /storage # chown 36:36 /storage/ # cat /etc/exports /storage *(rw) # systemctl enable rpcbind # systemctl enable nfs-server # systemctl start rpcbind # systemctl start nfs-server
Synology DSM 6.2.4-25556 Update 2 and later
Synology DSM 6.2.4-25556 Update 2 / DSM 7 extends Access Control Lists (ACLs) on all shared folders, and removes
To allow ovirt nodes to connect via NFS the ACL permissions and /etc/exports file need to be updated manually (take backups before editing!).
Create the shared folder required as usual and allow your oVirt host IP’s in the NFS permissions section
Login to your Synology NAS via SSH (search the web for ‘synology enable ssh access’) as root or sudo to root once logged in
Add a line for the kvm group (id 36) to /etc/group
Add a line for the vdsm user (id 36) to /etc/passwd
- Update the Synology ACLs to allow access to kvm:vdsm
EXPORT_DIR=/volumeX/... ALLOW_USER='user:vdsm:allow:rwxpdDaARWcCo:fd--' ALLOW_GROUP='group:kvm:allow:rwxpdDaARWcCo:fd--' synoacltool -get "$EXPORT_DIR" | grep -q "$ALLOW_USER" || synoacltool -add "$EXPORT_DIR" $ALLOW_USER > /dev/null synoacltool -get "$EXPORT_DIR" | grep -q "$ALLOW_GROUP" || synoacltool -add "$EXPORT_DIR" $ALLOW_GROUP > /dev/null synoacltool -get "$EXPORT_DIR"
Edit /etc/exports and replace the
anongidvalues with 36, e.g.
/volume1/ovirt 220.127.116.11(<Synology defined options>,anonuid=1025,anongid=100) 18.104.22.168(<Synology defined options>anonuid=1025,anongid=100)
/volume1/ovirt 22.214.171.124(<Synology defined options>,anonuid=36,anongid=36) 126.96.36.199(<Synology defined options>,anonuid=36,anongid=36)
Warning any edits (i.e. adding a new host) to the NFS share in the Synology UI will reset the
anongid back to Synology defatuls.
Workarounds for known issues
TODO: update for the newer supported releases
Normally the NFS server of any distro should work out of the box. Using older NFS servers or following different tuning advices throughout the internet may lead to a misconfiguration that gives lockups/freezes/stalls. Rule of thumb is to always ensure that the tcp window size parameters of your server are larger than the wsize and rsize mount option of your hypervisor hosts. E.g. using Fedora 19 as a hypervisor node these parameters are set to 1 MB.
# df ... 10.10.30.253:/var/nas3/ovirt on /rhev/data-center/mnt/10.10.30.253:_var_nas3_ovirt type nfs (...,rsize=1048576,wsize=1048576,...) ...
In this case it is a good idea to set the tcp window parameters on the NFS server to at least 4 MB in
# cat /etc/sysctl.conf net.ipv4.tcp_mem=4096 65536 4194304 net.ipv4.tcp_rmem=4096 65536 4194304 net.ipv4.tcp_wmem=4096 65536 4194304 net.core.rmem_max=8388608 net.core.wmem_max=8388608
To activate these settings for the running server reload them with
# sysctl -p