From e2766d563efc709a46a517972539c7c9e4faab8a Mon Sep 17 00:00:00 2001 From: Wong Hoi Sing Edison Date: Sat, 17 Apr 2021 01:37:59 +0800 Subject: [PATCH] Add `virtiofs` Support to vagrant-libvirt From : > Virtio-fs is a shared file system that lets virtual machines access a directory tree on the host. Unlike existing approaches, it is designed to offer local file system semantics and performance. From : > Existing solutions to this problem, such as virtio-9p, are based on existing network protocols that are not optimized for virtualization use cases. As a result they do not perform as well as local file systems and do not provide the semantics that some applications rely on. This PR add `virtiofs` support to vagrant-libvirt, which simply clone-and-hack from our existing 9p implementation. It also tidy up and synchronize naming for 9p implementation. Tested with: - Host: Ubuntu 20.10 + Linux 5.10.30 + QEMU 5.0.0 + Libvirt 6.6.0 + Vagrant 2.2.15 - Guest: Ubuntu 20.04 + Linux 5.4.0 Signed-off-by: Wong Hoi Sing Edison --- README.md | 134 +++++++++++++++--- .../cap/{mount_p9.rb => mount_9p.rb} | 4 +- lib/vagrant-libvirt/cap/mount_virtiofs.rb | 37 +++++ .../{synced_folder.rb => synced_folder_9p.rb} | 9 +- .../cap/synced_folder_virtiofs.rb | 109 ++++++++++++++ lib/vagrant-libvirt/plugin.rb | 18 ++- 6 files changed, 280 insertions(+), 31 deletions(-) rename lib/vagrant-libvirt/cap/{mount_p9.rb => mount_9p.rb} (94%) create mode 100644 lib/vagrant-libvirt/cap/mount_virtiofs.rb rename lib/vagrant-libvirt/cap/{synced_folder.rb => synced_folder_9p.rb} (95%) create mode 100644 lib/vagrant-libvirt/cap/synced_folder_virtiofs.rb diff --git a/README.md b/README.md index 4f9ef1f..286a6c7 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ can help a lot :-) * SSH into domains. * Setup hostname and network interfaces. * Provision domains with any built-in Vagrant provisioner. -* Synced folder support via `rsync`, `nfs` or `9p`. +* Synced folder support via `rsync`, `nfs`, `9p` or `virtiofs`. * Snapshots via [sahara](https://github.com/jedi4ever/sahara). * Package caching via [vagrant-cachier](http://fgrehm.viewdocs.io/vagrant-cachier/). @@ -1416,38 +1416,134 @@ Default is `eth0`. ## Synced Folders -Vagrant automatically syncs the project folder on the host to `/vagrant` in the guest. You can also configure -additional synced folders. +Vagrant automatically syncs the project folder on the host to `/vagrant` in +the guest. You can also configure additional synced folders. -`vagrant-libvirt` supports bidirectional synced folders via [NFS](https://en.wikipedia.org/wiki/Network_File_System) or [VirtFS](http://www.linux-kvm.org/page/VirtFS) ([9p or Plan 9](https://en.wikipedia.org/wiki/9P_(protocol))) and -unidirectional via rsync. The default is NFS. Difference between NFS and 9p is explained [here](https://unix.stackexchange.com/questions/240281/virtfs-plan-9-vs-nfs-as-tool-for-share-folder-for-virtual-machine). +**SECURITY NOTE:** for remote Libvirt, nfs synced folders requires a bridged +public network interface and you must connect to Libvirt via ssh. -You can change the synced folder type for `/vagrant` by explicity configuring -it an setting the type, e.g. +**NFS** -```shell -config.vm.synced_folder './', '/vagrant', type: 'rsync' +`vagrant-libvirt` supports +[NFS](https://www.vagrantup.com/docs/synced-folders/nfs) as default with +bidirectional synced folders. + +Example with NFS: + +``` ruby +Vagrant.configure("2") do |config| + config.vm.synced_folder "./", "/vagrant" +end ``` -or +**RSync** -```shell -config.vm.synced_folder './', '/vagrant', type: '9p', disabled: false, accessmode: "squash", owner: "1000" +`vagrant-libvirt` supports +[rsync](https://www.vagrantup.com/docs/synced-folders/rsync) with +unidirectional synced folders. + +Example with rsync: + +``` ruby +Vagrant.configure("2") do |config| + config.vm.synced_folder "./", "/vagrant", type: "rsync" +end ``` -or +**9P** -```shell -config.vm.synced_folder './', '/vagrant', type: '9p', disabled: false, accessmode: "mapped", mount: false -``` +`vagrant-libvirt` supports [VirtFS](http://www.linux-kvm.org/page/VirtFS) ([9p +or Plan 9](https://en.wikipedia.org/wiki/9P_\(protocol\))) with bidirectional +synced folders. + +Difference between NFS and 9p is explained +[here](https://unix.stackexchange.com/questions/240281/virtfs-plan-9-vs-nfs-as-tool-for-share-folder-for-virtual-machine). For 9p shares, a `mount: false` option allows to define synced folders without mounting them at boot. -Further documentation on using 9p can be found in [kernel docs](https://www.kernel.org/doc/Documentation/filesystems/9p.txt) and in [QEMU wiki](https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly). Please do note that 9p depends on support in the guest and not all distros come with the 9p module by default. +Example for `accessmode: "squash"` with 9p: -**SECURITY NOTE:** for remote Libvirt, nfs synced folders requires a bridged -public network interface and you must connect to Libvirt via ssh. +``` ruby +Vagrant.configure("2") do |config| + config.vm.synced_folder "./", "/vagrant", type: "9p", disabled: false, accessmode: "squash", owner: "1000" +end +``` + +Example for `accessmode: "mapped"` with 9p: + +``` ruby +Vagrant.configure("2") do |config| + config.vm.synced_folder "./", "/vagrant", type: "9p", disabled: false, accessmode: "mapped", mount: false +end +``` + +Further documentation on using 9p can be found in [kernel +docs](https://www.kernel.org/doc/Documentation/filesystems/9p.txt) and in +[QEMU +wiki](https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly). + +Please do note that 9p depends on support in the guest and not all distros +come with the 9p module by default. + +**Virtio-fs** + +`vagrant-libvirt` supports [Virtio-fs](https://virtio-fs.gitlab.io/) with +bidirectional synced folders. + +For virtiofs shares, a `mount: false` option allows to define synced folders +without mounting them at boot. + +So far, passthrough is the only supported access mode and it requires running +the virtiofsd daemon as root. + +QEMU needs to allocate the backing memory for all the guest RAM as shared +memory, e.g. [Use file-backed +memory](https://libvirt.org/kbase/virtiofs.html#host-setup) by enable +`memory_backing_dir` option in `/etc/libvirt/qemu.conf`: + +``` shell +memory_backing_dir = "/dev/shm" +``` + +Example for Libvirt \>= 6.2.0 (e.g. Ubuntu 20.10 with Linux 5.8.0 + QEMU 5.0 + +Libvirt 6.6.0, i.e. NUMA nodes required) with virtiofs: + +``` ruby +Vagrant.configure("2") do |config| + config.vm.provider :libvirt do |libvirt| + libvirt.cpus = 2 + libvirt.numa_nodes = [{ :cpus => "0-1", :memory => 8192, :memAccess => "shared" }] + libvirt.memorybacking :access, :mode => "shared" + end + config.vm.synced_folder "./", "/vagrant", type: "virtiofs" +end +``` + +Example for Libvirt \>= 6.9.0 (e.g. Ubuntu 21.04 with Linux 5.11.0 + QEMU 5.2 + +Libvirt 7.0.0, or Ubuntu 20.04 + [PPA +enabled](https://launchpad.net/~savoury1/+archive/ubuntu/virtualisation)) with +virtiofs: + +``` ruby +Vagrant.configure("2") do |config| + config.vm.provider :libvirt do |libvirt| + libvirt.cpus = 2 + libvirt.memory = 8192 + libvirt.memorybacking :access, :mode => "shared" + end + config.vm.synced_folder "./", "/vagrant", type: "virtiofs" +end +``` + +Further documentation on using virtiofs can be found in [official +HowTo](https://virtio-fs.gitlab.io/index.html#howto) and in [Libvirt +KB](https://libvirt.org/kbase/virtiofs.html). + +Please do note that virtiofs depends on: + + - Host: Linux \>= 5.4, QEMU \>= 4.2 and Libvirt \>= 6.2 (e.g. Ubuntu 20.10) + - Guest: Linux \>= 5.4 (e.g. Ubuntu 20.04) ## QEMU Session Support diff --git a/lib/vagrant-libvirt/cap/mount_p9.rb b/lib/vagrant-libvirt/cap/mount_9p.rb similarity index 94% rename from lib/vagrant-libvirt/cap/mount_p9.rb rename to lib/vagrant-libvirt/cap/mount_9p.rb index 66b141b..b35eda7 100644 --- a/lib/vagrant-libvirt/cap/mount_p9.rb +++ b/lib/vagrant-libvirt/cap/mount_9p.rb @@ -4,10 +4,10 @@ require 'vagrant/util/retryable' module VagrantPlugins module ProviderLibvirt module Cap - class MountP9 + class Mount9P extend Vagrant::Util::Retryable - def self.mount_p9_shared_folder(machine, folders) + def self.mount_9p_shared_folder(machine, folders) folders.each do |_name, opts| # Expand the guest path so we can handle things like "~/vagrant" expanded_guest_path = machine.guest.capability( diff --git a/lib/vagrant-libvirt/cap/mount_virtiofs.rb b/lib/vagrant-libvirt/cap/mount_virtiofs.rb new file mode 100644 index 0000000..645af28 --- /dev/null +++ b/lib/vagrant-libvirt/cap/mount_virtiofs.rb @@ -0,0 +1,37 @@ +require 'digest/md5' +require 'vagrant/util/retryable' + +module VagrantPlugins + module ProviderLibvirt + module Cap + class MountVirtioFS + extend Vagrant::Util::Retryable + + def self.mount_virtiofs_shared_folder(machine, folders) + folders.each do |_name, opts| + # Expand the guest path so we can handle things like "~/vagrant" + expanded_guest_path = machine.guest.capability( + :shell_expand_guest_path, opts[:guestpath] + ) + + # Do the actual creating and mounting + machine.communicate.sudo("mkdir -p #{expanded_guest_path}") + + # Mount + mount_tag = Digest::MD5.new.update(opts[:hostpath]).to_s[0, 31] + + mount_opts = "-o #{opts[:mount_opts]}" if opts[:mount_opts] + + mount_command = "mount -t virtiofs #{mount_opts} '#{mount_tag}' #{expanded_guest_path}" + retryable(on: Vagrant::Errors::LinuxMountFailed, + tries: 5, + sleep: 3) do + machine.communicate.sudo(mount_command, + error_class: Vagrant::Errors::LinuxMountFailed) + end + end + end + end + end + end +end diff --git a/lib/vagrant-libvirt/cap/synced_folder.rb b/lib/vagrant-libvirt/cap/synced_folder_9p.rb similarity index 95% rename from lib/vagrant-libvirt/cap/synced_folder.rb rename to lib/vagrant-libvirt/cap/synced_folder_9p.rb index ce496da..84f752b 100644 --- a/lib/vagrant-libvirt/cap/synced_folder.rb +++ b/lib/vagrant-libvirt/cap/synced_folder_9p.rb @@ -6,10 +6,9 @@ require 'digest/md5' require 'vagrant/util/subprocess' require 'vagrant/errors' require 'vagrant-libvirt/errors' -# require_relative "helper" module VagrantPlugins - module SyncedFolder9p + module SyncedFolder9P class SyncedFolder < Vagrant.plugin('2', :synced_folder) include Vagrant::Util include VagrantPlugins::ProviderLibvirt::Util::ErbTemplate @@ -69,10 +68,10 @@ module VagrantPlugins end end - # TODO: once up, mount folders + # once up, mount folders def enable(machine, folders, _opts) # Go through each folder and mount - machine.ui.info('mounting p9 share in guest') + machine.ui.info('mounting 9p share in guest') # Only mount folders that have a guest path specified. mount_folders = {} folders.each do |id, opts| @@ -83,7 +82,7 @@ module VagrantPlugins end # Mount the actual folder machine.guest.capability( - :mount_p9_shared_folder, mount_folders + :mount_9p_shared_folder, mount_folders ) end diff --git a/lib/vagrant-libvirt/cap/synced_folder_virtiofs.rb b/lib/vagrant-libvirt/cap/synced_folder_virtiofs.rb new file mode 100644 index 0000000..d2c914b --- /dev/null +++ b/lib/vagrant-libvirt/cap/synced_folder_virtiofs.rb @@ -0,0 +1,109 @@ +require 'log4r' +require 'ostruct' +require 'nokogiri' +require 'digest/md5' + +require 'vagrant/util/subprocess' +require 'vagrant/errors' +require 'vagrant-libvirt/errors' + +module VagrantPlugins + module SyncedFolderVirtioFS + class SyncedFolder < Vagrant.plugin('2', :synced_folder) + include Vagrant::Util + include VagrantPlugins::ProviderLibvirt::Util::ErbTemplate + + def initialize(*args) + super + @logger = Log4r::Logger.new('vagrant_libvirt::synced_folders::virtiofs') + end + + def usable?(machine, _raise_error = false) + # bail now if not using Libvirt since checking version would throw error + return false unless machine.provider_name == :libvirt + + # virtiofs support introduced since 6.2.0 + # version number format is major * 1,000,000 + minor * 1,000 + release + libvirt_version = machine.provider.driver.connection.client.libversion + libvirt_version >= 6_002_000 + end + + def prepare(machine, folders, _opts) + raise Vagrant::Errors::Error('No Libvirt connection') if machine.provider.driver.connection.nil? + @conn = machine.provider.driver.connection.client + + begin + # loop through folders + folders.each do |id, folder_opts| + folder_opts.merge!(target: id, + mount: true, + readonly: nil) { |_k, ov, _nv| ov } + + mount_tag = Digest::MD5.new.update(folder_opts[:hostpath]).to_s[0, 31] + folder_opts[:mount_tag] = mount_tag + + machine.ui.info "================\nMachine id: #{machine.id}\nShould be mounting folders\n #{id}, opts: #{folder_opts}" + + #xml = to_xml('filesystem', folder_opts) + xml = Nokogiri::XML::Builder.new do |xml| + xml.filesystem(type: 'mount', accessmode: 'passthrough') do + xml.driver(type: 'virtiofs') + xml.source(dir: folder_opts[:hostpath]) + xml.target(dir: mount_tag) + xml.readonly unless folder_opts[:readonly].nil? + end + end.to_xml( + save_with: Nokogiri::XML::Node::SaveOptions::NO_DECLARATION | + Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS | + Nokogiri::XML::Node::SaveOptions::FORMAT + ) + # puts "<<<<< XML:\n #{xml}\n >>>>>" + @conn.lookup_domain_by_uuid(machine.id).attach_device(xml, 0) + end + rescue => e + machine.ui.error("could not attach device because: #{e}") + raise VagrantPlugins::ProviderLibvirt::Errors::AttachDeviceError, + error_message: e.message + end + end + + # once up, mount folders + def enable(machine, folders, _opts) + # Go through each folder and mount + machine.ui.info('mounting virtiofs share in guest') + # Only mount folders that have a guest path specified. + mount_folders = {} + folders.each do |id, opts| + next unless opts[:mount] && opts[:guestpath] && !opts[:guestpath].empty? + mount_folders[id] = opts.dup + end + # Mount the actual folder + machine.guest.capability( + :mount_virtiofs_shared_folder, mount_folders + ) + end + + def cleanup(machine, _opts) + if machine.provider.driver.connection.nil? + raise Vagrant::Errors::Error('No Libvirt connection') + end + @conn = machine.provider.driver.connection.client + begin + if machine.id && machine.id != '' + dom = @conn.lookup_domain_by_uuid(machine.id) + Nokogiri::XML(dom.xml_desc).xpath( + '/domain/devices/filesystem' + ).each do |xml| + dom.detach_device(xml.to_s) + machine.ui.info 'Cleaned up shared folders' + end + end + rescue => e + machine.ui.error("could not detach device because: #{e}") + raise VagrantPlugins::ProviderLibvirt::Errors::DetachDeviceError, + error_message: e.message + end + end + end + end +end diff --git a/lib/vagrant-libvirt/plugin.rb b/lib/vagrant-libvirt/plugin.rb index f4b3e38..8dc50a8 100644 --- a/lib/vagrant-libvirt/plugin.rb +++ b/lib/vagrant-libvirt/plugin.rb @@ -30,9 +30,13 @@ module VagrantPlugins hook.after Vagrant::Action::Builtin::BoxRemove, Action.remove_libvirt_image end - guest_capability('linux', 'mount_p9_shared_folder') do - require_relative 'cap/mount_p9' - Cap::MountP9 + guest_capability('linux', 'mount_9p_shared_folder') do + require_relative 'cap/mount_9p' + Cap::Mount9P + end + guest_capability('linux', 'mount_virtiofs_shared_folder') do + require_relative 'cap/mount_virtiofs' + Cap::MountVirtioFS end provider_capability(:libvirt, :nic_mac_addresses) do @@ -48,8 +52,12 @@ module VagrantPlugins # lower priority than nfs or rsync # https://github.com/vagrant-libvirt/vagrant-libvirt/pull/170 synced_folder('9p', 4) do - require_relative 'cap/synced_folder' - VagrantPlugins::SyncedFolder9p::SyncedFolder + require_relative 'cap/synced_folder_9p' + VagrantPlugins::SyncedFolder9P::SyncedFolder + end + synced_folder('virtiofs', 5) do + require_relative 'cap/synced_folder_virtiofs' + VagrantPlugins::SyncedFolderVirtioFS::SyncedFolder end # This initializes the internationalization strings.