bhyve: Support /domain/bootloader configuration for non-FreeBSD guests.

We still default to bhyveloader(1) if no explicit bootloader
configuration is supplied in the domain.

If the /domain/bootloader looks like grub-bhyve and the user doesn't
supply /domain/bootloader_args, we make an intelligent guess and try
chainloading the first partition on the disk (or a CD if one exists,
under the assumption that for a VM a CD is likely an install source).

Caveat: Assumes the HDD boots from the msdos1 partition. I think this is
a pretty reasonable assumption for a VM. (DrvBhyve with Bhyveload
already assumes that the first disk should be booted.)

I've tested both HDD and CD boot and they seem to work.

docs/drvbhyve.html.in

@@ -37,8 +37,7 @@ bhyve+ssh://root@example.com/system (remote access, SSH tunnelled)
 <h3>Example config</h3>
 <p>
 The bhyve driver in libvirt is in its early stage and under active development. So it supports
-only limited number of features bhyve provides. All the supported features could be found
-in this sample domain XML.
+only limited number of features bhyve provides.
 </p>
 
 <p>
@@ -48,10 +47,21 @@ disk device were supported per-domain. However,
 up to 31 PCI devices.
 </p>
 
+<p>
+Note: the Bhyve driver in libvirt will boot whichever device is first. If you
+want to install from CD, put the CD device first. If not, put the root HDD
+first.
+</p>
+
+<p>
+Note: Only the SATA bus is supported. Only <code>cdrom</code>- and
+<code>disk</code>-type disks are supported.
+</p>
+
 <pre>
 &lt;domain type='bhyve'&gt;
-  &lt;name&gt;bhyve&lt;/name&gt;
-  &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
+    &lt;name&gt;bhyve&lt;/name&gt;
+    &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
     &lt;memory&gt;219136&lt;/memory&gt;
     &lt;currentMemory&gt;219136&lt;/currentMemory&gt;
     &lt;vcpu&gt;1&lt;/vcpu&gt;
@@ -76,6 +86,7 @@ up to 31 PCI devices.
         &lt;driver name='file' type='raw'/&gt;
         &lt;source file='/path/to/cdrom.iso'/&gt;
         &lt;target dev='hdc' bus='sata'/&gt;
+        &lt;readonly/&gt;
       &lt;/disk&gt;
       &lt;interface type='bridge'&gt;
         &lt;model type='virtio'/&gt;
@@ -85,6 +96,53 @@ up to 31 PCI devices.
 &lt;/domain&gt;
 </pre>
 
+<p>(The &lt;disk&gt; sections may be swapped in order to install from
+<em>cdrom.iso</em>.)</p>
+
+<h3>Example config (Linux guest)</h3>
+
+<p>
+Note the addition of &lt;bootloader&gt;.
+</p>
+
+<pre>
+&lt;domain type='bhyve'&gt;
+    &lt;name&gt;linux_guest&lt;/name&gt;
+    &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
+    &lt;memory&gt;131072&lt;/memory&gt;
+    &lt;currentMemory&gt;131072&lt;/currentMemory&gt;
+    &lt;vcpu&gt;1&lt;/vcpu&gt;
+    &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
+    &lt;os&gt;
+       &lt;type&gt;hvm&lt;/type&gt;
+    &lt;/os&gt;
+    &lt;features&gt;
+      &lt;apic/&gt;
+      &lt;acpi/&gt;
+    &lt;/features&gt;
+    &lt;clock offset='utc'/&gt;
+    &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
+    &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
+    &lt;on_crash&gt;destroy&lt;/on_crash&gt;
+    &lt;devices&gt;
+      &lt;disk type='file' device='disk'&gt;
+        &lt;driver name='file' type='raw'/&gt;
+        &lt;source file='/path/to/guest_hdd.img'/&gt;
+        &lt;target dev='hda' bus='sata'/&gt;
+      &lt;/disk&gt;
+      &lt;disk type='file' device='cdrom'&gt;
+        &lt;driver name='file' type='raw'/&gt;
+        &lt;source file='/path/to/cdrom.iso'/&gt;
+        &lt;target dev='hdc' bus='sata'/&gt;
+        &lt;readonly/&gt;
+      &lt;/disk&gt;
+      &lt;interface type='bridge'&gt;
+        &lt;model type='virtio'/&gt;
+        &lt;source bridge="virbr0"/&gt;
+      &lt;/interface&gt;
+    &lt;/devices&gt;
+&lt;/domain&gt;
+</pre>
 
 <h2><a name="usage">Guest usage / management</a></h2>
 
@@ -119,6 +177,20 @@ to let a guest boot or start a guest using:</p>
 
 <pre>start --console domname</pre>
 
+<p><b>NB:</b> An bootloader configured to require user interaction will prevent
+the domain from starting (and thus <code>virsh console</code> or <code>start
+--console</code> from functioning) until the user interacts with it manually on
+the VM host. Because users typically do not have access to the VM host,
+interactive bootloaders are unsupported by libvirt. <em>However,</em> if you happen to
+run into this scenario and also happen to have access to the Bhyve host
+machine, you may select a boot option and allow the domain to finish starting
+by using an alternative terminal client on the VM host to connect to the
+domain-configured null modem device. One example (assuming
+<code>/dev/nmdm0B</code> is configured as the slave end of the domain serial
+device) is:</p>
+
+<pre>cu -l /dev/nmdm0B</pre>
+
 <h3><a name="xmltonative">Converting from domain XML to Bhyve args</a></h3>
 
 <p>
@@ -157,5 +229,25 @@ An example of domain XML device entry for that will look like:</p>
 <p>Please refer to the <a href="storage.html">Storage documentation</a> for more details on storage
 management.</p>
 
+<h3><a name="grubbhyve">Using grub2-bhyve or Alternative Bootloaders</a></h3>
+
+<p>It's possible to boot non-FreeBSD guests by specifying an explicit
+bootloader, e.g. <code>grub-bhyve(1)</code>. Arguments to the bootloader may be
+specified as well. If the bootloader is <code>grub-bhyve</code> and arguments
+are omitted, libvirt will try and boot the first disk in the domain (either
+<code>cdrom</code>- or <code>disk</code>-type devices). If the disk type is
+<code>disk</code>, it will attempt to boot from the first partition in the disk
+image.</p>
+
+<pre>
+  ...
+    &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
+    &lt;bootloader_args&gt;...&lt;/bootloader_args&gt;
+  ...
+</pre>
+
+<p>Caveat: <code>bootloader_args</code> does not support any quoting.
+Filenames, etc, must not have spaces or they will be tokenized incorrectly.</p>
+
   </body>
 </html>

docs/formatdomain.html.in

@@ -217,7 +217,9 @@
       a BIOS, and instead the host is responsible to kicking off the
       operating system boot. This may use a pseudo-bootloader in the
       host to provide an interface to choose a kernel for the guest.
-      An example is <code>pygrub</code> with Xen.
+      An example is <code>pygrub</code> with Xen. The Bhyve hypervisor
+      also uses a host bootloader, either <code>bhyveload</code> or
+      <code>grub-bhyve</code>.
     </p>
 
 <pre>