From 252958ee16a9eefc2b55ce1be7af34ba8c84452e Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Wed, 21 Aug 2019 20:42:41 -0500 Subject: [PATCH] backup: Document new XML for backups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prepare for new backup APIs by describing the XML that will represent a backup. The XML resembles snapshots and checkpoints in being able to select actions for a set of disks, but has other differences. It can support both push model (the hypervisor does the backup directly into the destination file) and pull model (the hypervisor exposes an access port for a third party to grab what is necessary). Add testsuite coverage for some minimal uses of the XML. The element within tries to model the same elements as a under , but sharing the RNG grammar proved to be hairy. That is in part because while use to describe a host resource in use by the guest, a backup job is using a host resource that is not visible to the guest: a push backup action is instead describing a (which ultimately could be a remote network resource, but for simplicity the RNG just validates a local file for now), and a pull backup action is instead describing a temporary local file (which probably should not be a remote resource). A future refactoring may thus introduce some way to parameterize RNG to accept ... so that the name of the subelement can be for domain, or or as needed for backups. Future patches may improve this area of code. Signed-off-by: Eric Blake Signed-off-by: Peter Krempa Reviewed-by: Daniel P. Berrangé Reviewed-by: Ján Tomko --- docs/docs.html.in | 3 +- docs/format.html.in | 1 + docs/formatbackup.html.in | 175 ++++++++++++++ docs/formatcheckpoint.html.in | 12 +- docs/index.html.in | 3 +- docs/schemas/domainbackup.rng | 223 ++++++++++++++++++ libvirt.spec.in | 1 + mingw-libvirt.spec.in | 2 + tests/Makefile.am | 1 + .../backup-pull-seclabel.xml | 18 ++ tests/domainbackupxml2xmlin/backup-pull.xml | 10 + .../backup-push-seclabel.xml | 17 ++ tests/domainbackupxml2xmlin/backup-push.xml | 10 + tests/domainbackupxml2xmlin/empty.xml | 1 + tests/virschematest.c | 1 + 15 files changed, 470 insertions(+), 8 deletions(-) create mode 100644 docs/formatbackup.html.in create mode 100644 docs/schemas/domainbackup.rng create mode 100644 tests/domainbackupxml2xmlin/backup-pull-seclabel.xml create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml create mode 100644 tests/domainbackupxml2xmlin/backup-push-seclabel.xml create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml create mode 100644 tests/domainbackupxml2xmlin/empty.xml diff --git a/docs/docs.html.in b/docs/docs.html.in index f441769617..eddf3de6d5 100644 --- a/docs/docs.html.in +++ b/docs/docs.html.in @@ -85,7 +85,8 @@ node devices, secrets, snapshots, - checkpoints + checkpoints, + backup jobs
URI format
The URI formats used for connecting to libvirt
diff --git a/docs/format.html.in b/docs/format.html.in index 3be2237663..d013528fe0 100644 --- a/docs/format.html.in +++ b/docs/format.html.in @@ -27,6 +27,7 @@
  • Secrets
  • Snapshots
  • Checkpoints
    • +
  • Backup jobs

    • Command line validation

    diff --git a/docs/formatbackup.html.in b/docs/formatbackup.html.in new file mode 100644 index 0000000000..1c690901c7 --- /dev/null +++ b/docs/formatbackup.html.in @@ -0,0 +1,175 @@ + + + + +

    Backup XML format

    + +
      + +

      Backup XML

      + +

      + Creating a backup, whether full or incremental, is done + via virDomainBackupBegin(), which takes an XML + description of the actions to perform, as well as an optional + second XML document describing a + checkpoint to create at the same point in time. See + also a comparison between + the various state capture APIs. +

      +

      + There are two general modes for backups: a push mode (where the + hypervisor writes out the data to the destination file, which + may be local or remote), and a pull mode (where the hypervisor + creates an NBD server that a third-party client can then read as + needed, and which requires the use of temporary storage, + typically local, until the backup is complete). +

      +

      + The instructions for beginning a backup job are provided as + attributes and elements of the + top-level domainbackup element. This element + includes an optional attribute mode which can be + either "push" or "pull" (default + push). virDomainBackupGetXMLDesc() can be used to + see the actual values selected for elements omitted during + creation (for example, learning which port the NBD server is + using in the pull model or what file names libvirt generated + when none were supplied). The following child elements and attributes + are supported: +

      +
      +
      incremental
      +
      An optional element giving the name of an existing + checkpoint of the domain, which will be used to make this + backup an incremental one. In the push model, only changes + since the named checkpoint are written to the destination. In + the pull model, the NBD server uses the + NBD_OPT_SET_META_CONTEXT extension to advertise to the client + which portions of the export contain changes since the named + checkpoint. If omitted, a full backup is performed. +
      +
      server
      +
      Present only for a pull mode backup. Contains the same + attributes as + the protocol + element of a disk attached via NBD in the domain (such as + transport, socket, name, port, or tls), necessary to set up an + NBD server that exposes the content of each disk at the time + the backup is started. +
      +
      disks
      +
      An optional listing of instructions for disks participating + in the backup (if omitted, all disks participate and libvirt + attempts to generate filenames by appending the current + timestamp as a suffix). If the entire element was omitted on + input, then all disks participate in the backup, otherwise, + only the disks explicitly listed which do not also + use backup='no' will participate. On output, this + is the state of each of the domain's disk in relation to the + backup operation. +
      +
      disk
      +
      This sub-element describes the backup properties of a + specific disk, with the following attributes and child + elements: +
      +
      name
      +
      A mandatory attribute which must match + the <target dev='name'/> + of one of + the disk + devices specified for the domain at the time of + the checkpoint.
      +
      backup
      +
      Setting this attribute to yes(default) specifies + that the disk should take part in the backup and using + no excludes the disk from the backup.
      +
      type
      +
      A mandatory attribute to describe the type of the + disk, except when backup='no' is + used. Valid values include file, + block, or network. + Similar to a disk declaration for a domain, the choice of type + controls what additional sub-elements are needed to describe + the destination (such as protocol for a + network destination).
      +
      target
      +
      Valid only for push mode backups, this is the + primary sub-element that describes the file name of + the backup destination, similar to + the source sub-element of a domain + disk. An optional sub-element driver can + also be used, with an attribute type to + specify a destination format different from + qcow2.
      +
      scratch
      +
      Valid only for pull mode backups, this is the + primary sub-element that describes the file name of + the local scratch file to be used in facilitating the + backup, and is similar to the source + sub-element of a domain disk. Currently only file + and block scratch storage is supported. The + file scratch file is created and deleted by + libvirt in the given location. A block scratch + device must exist prior to starting the backup and is formatted. + The block device must have enough space for the corresponding + disk data including format overhead. + + If VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL flag is + used the file for a scratch of file type must + exist with the correct format and size to hold the copy and is + used without modification. The file is not deleted after the + backup but the contents of the file don't make sense outside + of the backup. The same applies for the block device which + must be formatted appropriately.
      +
      +
      +
      +
      +
      + +

      Examples

      + +

      Use virDomainBackupBegin() to perform a full + backup using push mode. The example lets libvirt pick the + destination and format for 'vda', fully specifies that we want a + raw backup of 'vdb', and omits 'vdc' from the operation. +

      + 
      +<domainbackup>
+  <disks>
+    <disk name='vda' backup='yes'/>
+    <disk name='vdb' type='file'>
+      <target file='/path/to/vdb.backup'/>
+      <driver type='raw'/>
+    </disk>
+    <disk name='vdc' backup='no'/>
+  </disks>
+</domainbackup>
+
      + +

      If the previous full backup also passed a parameter describing + checkpoint XML that resulted + in a checkpoint named 1525889631, we can make + another call to virDomainBackupBegin() to perform + an incremental backup of just the data changed since that + checkpoint, this time using the following XML to start a pull + model export of the 'vda' and 'vdb' disks, where a third-party + NBD client connecting to '/path/to/server' completes the backup + (omitting 'vdc' from the explicit list has the same effect as + the backup='no' from the previous example): +

      + 
      +<domainbackup mode="pull">
+  <incremental>1525889631</incremental>
+  <server transport="unix" socket="/path/to/server"/>
+  <disks>
+    <disk name='vda' backup='yes' type='file'>
+      <scratch file='/path/to/file1.scratch'/>
+    </disk>
+  </disks>
+</domainbackup>
+
      + + diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in index 044bbfe4b0..ee56194523 100644 --- a/docs/formatcheckpoint.html.in +++ b/docs/formatcheckpoint.html.in @@ -28,12 +28,12 @@ first checkpoint and the second backup operation), it is possible to do an offline reconstruction of the state of the disk at the time of the second backup without having to copy as - much data as a second full backup would require. Future API - additions will make it possible to create checkpoints in - conjunction with a backup - via virDomainBackupBegin() or with an external - snapshot via virDomainSnapshotCreateXML2; but for - now, libvirt exposes enough support to create disk checkpoints + much data as a second full backup would require. Most disk + checkpoints are created in conjunction with a backup + via virDomainBackupBegin(), although a future API + addition of virDomainSnapshotCreateXML2() will also + make this possible when creating external snapshots; however, + libvirt also exposes enough support to create disk checkpoints independently from a backup operation via virDomainCheckpointCreateXML() since 5.6.0. Likewise, the creation of checkpoints when diff --git a/docs/index.html.in b/docs/index.html.in index 7d0ab650e3..26e8406917 100644 --- a/docs/index.html.in +++ b/docs/index.html.in @@ -59,7 +59,8 @@ node devices, secrets, snapshots, - checkpoints + checkpoints, + backup jobs
      Wiki
      Read further community contributed content
      diff --git a/docs/schemas/domainbackup.rng b/docs/schemas/domainbackup.rng new file mode 100644 index 0000000000..7286acb18c --- /dev/null +++ b/docs/schemas/domainbackup.rng @@ -0,0 +1,223 @@ + + + + + + + + + + + + + + + + + + + + + + push + + + + + + + pull + + + + + + + + tcp + + + + + + + + + + + + + + + + + + unix + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + qcow2 + + + + + + + + + + yes + + + + + + + + + + + + + + + + + + + no + + + + + + file + + + + + + + + + + + + + + + + + + + block + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + no + + + + + + + file + + + + + + + + + + + + + + + + + + + + block + + + + + + + + + + + + + + + + + + + + + diff --git a/libvirt.spec.in b/libvirt.spec.in index c431f0dfb5..8bbbe93cb0 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1886,6 +1886,7 @@ exit 0 %{_datadir}/libvirt/schemas/capability.rng %{_datadir}/libvirt/schemas/cputypes.rng %{_datadir}/libvirt/schemas/domain.rng +%{_datadir}/libvirt/schemas/domainbackup.rng %{_datadir}/libvirt/schemas/domaincaps.rng %{_datadir}/libvirt/schemas/domaincheckpoint.rng %{_datadir}/libvirt/schemas/domaincommon.rng diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in index 033c8f9658..de8d3316b4 100644 --- a/mingw-libvirt.spec.in +++ b/mingw-libvirt.spec.in @@ -238,6 +238,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh %{mingw32_datadir}/libvirt/schemas/capability.rng %{mingw32_datadir}/libvirt/schemas/cputypes.rng %{mingw32_datadir}/libvirt/schemas/domain.rng +%{mingw32_datadir}/libvirt/schemas/domainbackup.rng %{mingw32_datadir}/libvirt/schemas/domaincaps.rng %{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng %{mingw32_datadir}/libvirt/schemas/domaincommon.rng @@ -329,6 +330,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh %{mingw64_datadir}/libvirt/schemas/capability.rng %{mingw64_datadir}/libvirt/schemas/cputypes.rng %{mingw64_datadir}/libvirt/schemas/domain.rng +%{mingw64_datadir}/libvirt/schemas/domainbackup.rng %{mingw64_datadir}/libvirt/schemas/domaincaps.rng %{mingw64_datadir}/libvirt/schemas/domaincheckpoint.rng %{mingw64_datadir}/libvirt/schemas/domaincommon.rng diff --git a/tests/Makefile.am b/tests/Makefile.am index 9716d9d2be..9e825163e5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -91,6 +91,7 @@ EXTRA_DIST = \ commanddata \ cputestdata \ domaincapsdata \ + domainbackupxml2xmlin \ domainconfdata \ domainschemadata \ fchostdata \ diff --git a/tests/domainbackupxml2xmlin/backup-pull-seclabel.xml b/tests/domainbackupxml2xmlin/backup-pull-seclabel.xml new file mode 100644 index 0000000000..a00d8758bb --- /dev/null +++ b/tests/domainbackupxml2xmlin/backup-pull-seclabel.xml @@ -0,0 +1,18 @@ + + 1525889631 + + + + + + + + + + + + + + + + diff --git a/tests/domainbackupxml2xmlin/backup-pull.xml b/tests/domainbackupxml2xmlin/backup-pull.xml new file mode 100644 index 0000000000..c0bea4771d --- /dev/null +++ b/tests/domainbackupxml2xmlin/backup-pull.xml @@ -0,0 +1,10 @@ + + 1525889631 + + + + + + + + diff --git a/tests/domainbackupxml2xmlin/backup-push-seclabel.xml b/tests/domainbackupxml2xmlin/backup-push-seclabel.xml new file mode 100644 index 0000000000..dbaf7f8e7c --- /dev/null +++ b/tests/domainbackupxml2xmlin/backup-push-seclabel.xml @@ -0,0 +1,17 @@ + + 1525889631 + + + + + + + + + + + + + + + diff --git a/tests/domainbackupxml2xmlin/backup-push.xml b/tests/domainbackupxml2xmlin/backup-push.xml new file mode 100644 index 0000000000..0bfec9b270 --- /dev/null +++ b/tests/domainbackupxml2xmlin/backup-push.xml @@ -0,0 +1,10 @@ + + 1525889631 + + + + + + + + diff --git a/tests/domainbackupxml2xmlin/empty.xml b/tests/domainbackupxml2xmlin/empty.xml new file mode 100644 index 0000000000..7ed511f97b --- /dev/null +++ b/tests/domainbackupxml2xmlin/empty.xml @@ -0,0 +1 @@ + diff --git a/tests/virschematest.c b/tests/virschematest.c index df50ef1717..5ae2d207d1 100644 --- a/tests/virschematest.c +++ b/tests/virschematest.c @@ -205,6 +205,7 @@ mymain(void) "genericxml2xmloutdata", "xlconfigdata", "libxlxml2domconfigdata", "qemuhotplugtestdomains"); DO_TEST_DIR("domaincaps.rng", "domaincapsdata"); + DO_TEST_DIR("domainbackup.rng", "domainbackupxml2xmlin"); DO_TEST_DIR("domaincheckpoint.rng", "qemudomaincheckpointxml2xmlin", "qemudomaincheckpointxml2xmlout"); DO_TEST_DIR("domainsnapshot.rng", "qemudomainsnapshotxml2xmlin",