Commit Graph
100 Commits
Author SHA1 Message Date
Peter Krempa 03a2235ab7 tests: qemuxml2xml: Modernize 'disk-cache' test
Mirror what's done in the xml2argv test and use recent capabilities.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2020-01-27 14:55:03 +01:00
Peter Krempa 66cba5c065 tests: qemuxml2argv: Use 64 bit qemu binary and 1.5 machine type in 'disk-cache' case
The data is tested against the latest qemu binaries so we should use the
proper architecture. Also the test is used against data from qemu 1.5.3
and thus we should use a machine type that qemu supported.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2020-01-27 14:55:03 +01:00
Peter Krempa b168fa88b8 qemu: snapshot: Prevent too-nested domain XML when doing inactive snapshot
Similarly to 510d154a0b we need to prevent
doing too deeply nested backing chains and reject them with a sane error
message.

Add a loop to go through the snapshots prior to attempting actually
creating them to prevent some possible inconsistent scenarios.

We don't need to do it when reusing backing chains as we'll be
re-detecting the backing chain in that case anyways.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-27 14:02:01 +01:00
Peter Krempa 8e9e73a984 qemu: snapshot: Always rewrite backingStore data when reusing existing images
Don't adopt the backing store data when reusing images provided by the
user. This will force a backing chain re-probe as users might have
passed in something unexpected in the overlay where our view of the
backing chain would not correspond.

This is done only for inactive snapshots as there we have way less
verification.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-27 14:02:01 +01:00
Peter Krempa 29d43bf96a qemu: monitor: Improve error message when QEMU reply is too large
Don't use ERANGE as it doesn't make much sense in the error message.
Also point out that the reply from qemu was too large which is not
obvious from the original error:

 error: No complete monitor response found in 10485760 bytes: Numerical result out of range

The new message will read:

 error: internal error: QEMU monitor reply exceeds buffer size (10485760 bytes)

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2020-01-24 13:47:45 +01:00
Peter Krempa f4e7c792d5 qemu: block: Don't skip creation of 'luks' formatted images
libvirt treats 'luks' images as raw+encryption. The logic in
qemuBlockStorageSourceCreateFormat skipped the creation if the requested
image was raw but didn't take into account the encryption.

This manifested itself e.g. when attempting to do a virsh blockcopy with
the following XML:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/tmp/enccpy'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
        </encryption>
      </source>
    </disk>

Where qemu would report the following error:

 unable to execute QEMU command 'blockdev-add': Volume is not in LUKS format

rather than actually formatting the image first.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:46:46 +01:00
Peter Krempa 7134f26b73 tests: hash: Test case for adding duplicate hash entry
Test that adding a duplicate entry is rejected properly. This also
allows to see the error message of the duplicate key addition in verbose
mode.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2020-01-24 13:45:38 +01:00
Peter Krempa ae60e05817 util: hash: Improve debugability of "Duplicate key" error message
If we get a user reporting this error message being shown it's pretty
useless in terms of actually debugging it since we don't know which hash
and which key are actually subject to the error.

This patch adds a new hash table callback which formats the
user-readable version of the hash key and reports it in the new message
which will look like:

"Duplicate hash table key 'blah'"

That way we will at least have an anchor point where to start the
search.

There are two special implementations of keys which are numeric so we
add specific printer functions for them.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2020-01-24 13:45:38 +01:00
Peter Krempa 0c3792a155 qemu: backup: Implement support for backup disk bitmap name configuration
Use the user-configured name of the bitmap when merging the appropriate
bitmaps for an incremental backup so that the user can see it as
configured. Additionally expose the default bitmap name if nothing is
configured.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
2020-01-24 13:40:53 +01:00
Peter Krempa bce4ac55f8 qemu: backup: Implement support for backup disk export name configuration
Pass the exportname as configured when exporting the image via NBD and
fill it with the default if it's not configured.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:48 +01:00
Peter Krempa 12ccd8d4db conf: backup: Allow configuration of names exported via NBD
If users wish to use different name for exported disks or bitmaps
the new fields allow to do so. Additionally they also document the
current settings.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:48 +01:00
Peter Krempa 69908db0f6 qemu: Fix value of 'device' argument for block-commit
When using blockdev configurations the 'device' argument of
'blockdev-commit' must correspond to the topmost node in the block node
graph. Libvirt didn't do this properly in case when 'copy_on_read'
option was enabled on the disk.

Use qemuDomainDiskGetTopNodename to fix it when calling block-commit.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:36 +01:00
Peter Krempa e3137539a9 qemu: Fix value of 'device' argument for blockdev-mirror
When using blockdev configurations the 'device' argument of
'blockdev-mirror' must correspond to the topmost node in the block node
graph. Libvirt didn't do this properly in case when 'copy_on_read'
option was enabled on the disk.

Use qemuDomainDiskGetTopNodename to fix it for the blockdev-mirror calls
in qemuDomainBlockCopy and the non-shared-storage migration.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:36 +01:00
Peter Krempa 0b0f389335 qemu: domain: Extract code to determine topmost nodename to qemuDomainDiskGetTopNodename
There are more places which require getting the topmost nodename to be
passed to qemu. Separate it out into a new function.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:36 +01:00
Peter Krempa 623366d130 qemu: blockcopy: Actually unplug unused images when mirror job fails to start
If a mirror job fails to start in -blockdev mode we'd not unplug the
backing files we added first because the code on the error path checked
the wrong value. 'rc' is used as status of the code which added the
images, but the state of the 'block(dev)-mirror' call is stored in 'ret'
at that point.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-01-24 13:40:36 +01:00
Peter Krempa 2775aada16 tests: virstorage: Add tests for NBD URI style syntax over UNIX
Add few test cases for nbd+unix style URIs with few corner cases.

The NBD URI syntax is documented at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2020-01-17 12:34:10 +01:00
Peter Krempa 5f2fa393f7 util: storagefile: Properly set transport type when parsing NBD strings
When parsing legacy NBD backing file strings such as
'nbd:unix:/tmp/sock:exportname=/' we'd fail to set the transport to
VIR_STORAGE_NET_HOST_TRANS_UNIX. This started to be a problem once we
actually started to generate config of the backing store on the command
line with -blockdev as the JSON code would try to format it as TCP and
fail with:

 internal error: argument key 'host' must not have null value

Set the type properly and add a test.

This bug was found by the libguestfs test suite in:

https://bugzilla.redhat.com/show_bug.cgi?id=1791614

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reported-by: Ming Xie <mxie@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
2020-01-17 12:34:10 +01:00
Peter Krempa 3f2d167d9c conf: Always format storage source auth and encryption under <source> for backing files
Historically there are two places where we format authentication and
encryption for a disk. The logich which formats it for backing files was
flawed though and didn't format it at all. This worked if the image
became a backing file through the means of a snapshot but not directly.

Force formatting of the source and encryption for any non-disk case to
fix the issue.

This caused problems in many places as we use the formatter to copy the
definition. Effectively any copy lost the secret definition.

https://bugzilla.redhat.com/show_bug.cgi?id=1789310
https://bugzilla.redhat.com/show_bug.cgi?id=1788898

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2020-01-13 12:53:58 +01:00
Peter Krempa 23b52d9420 tests: qemuxml2xml: Enable luks-disks-source-qcow2 case
The test data was used only in xml->argv testing but it will have some
interresting fallout soon.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2020-01-13 12:53:58 +01:00
Peter Krempa ebebf63b9c tests: qemuxml2argv: Run luks-disks-source-qcow2 case with latest caps
Try also the modern incarnation of the test.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2020-01-13 12:53:58 +01:00
Peter Krempa d06391d611 tests: qemuxml2argv: Add disk image with encrypted backing file
Add another disk to luks-disks-source-qcow2 case to cover a backing
chain with encrypted members.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2020-01-13 12:53:58 +01:00
Peter Krempa 0604b9e627 news: Mention problems with backing image format probing
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-10 12:49:03 +01:00
Peter Krempa 30867fb098 util: storage: Link to knowledge base when reporting missing image backing format
Mention the knowledge base article which has tips how to fix the backing
chain to work with current libvirt.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-10 12:49:03 +01:00
Peter Krempa c314222a01 qemu: backup: Move capability check after inactive check
Inactive VM doesn't have qemuCaps set thus we'd never properly report
that VM backups are supported only for running VMs.

Move the capability check after the active check.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2020-01-08 07:10:46 +01:00
Peter Krempa a4877192a1 qemu: backup: roll-back checkpoint metadata if the checkpoint wasn't taken
We insert the checkpoint metadata into the list of checkpoints prior to
actually creating the on-disk bits. If the 'transaction' or any other
steps done between inserting the checkpoint and creating the on-disk
data fail we'd end up with an unusable checkpoint that would vanish
after libvirtd restart.

Prevent this by rolling back the metadata if we didn't actually take and
record the checkpoint.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-07 15:20:14 +01:00
Peter Krempa 54dd75ec8d qemu: checkpoint: Extract and export rollback of checkpoint metadata storing
If we are certain that the checkpoint creation failed we remove the
metadata from the list. To allow reusing this in the backup code add a
new helper and export it.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-07 15:19:58 +01:00
Peter Krempa 0809e40b5f man: virsh: Mention how to disable snapshot of disk in snapshot-create-as
Document that --diskspec DISK,snapshot=no disables snapshot for the
given disk.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2020-01-07 15:17:58 +01:00
Peter Krempa 76592c596a util: time: Fix comment for virTimeFieldsNow
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
2020-01-06 10:33:35 +01:00
Peter Krempa 5b53d9dd91 schemas: backup: Remove pointless <choice> for 'name' of backup disk
One of the first versions thought of using disk path as the second
option but this was dropped as being a legacy interface. Remove the
leftover pointless <choice> wrapper for the disk name as there's just
one option now.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:36 +01:00
Peter Krempa 5632ed8bad qemu: process: Terminate backup job on VM destroy
Commit d75f865fb9 caused a job-deadlock if
a VM is running the backup job and being destroyed as it removed the
cleanup of the async job type and there was nothing to clean up the
backup job.

Add an explicit cleanup of the backup job when destroying a VM.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:36 +01:00
Peter Krempa bc8b159cb1 qemu: backup: Properly propagate async job type when cancelling the job
When cancelling the blockjobs as part of failed backup job startup
recover we didn't pass in the correct async job type. Luckily the block
job handler and cancellation code paths use no block job at all
currently so those were correct.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:36 +01:00
Peter Krempa 3a98fe9db3 qemu: blockjob: Remove infrastructure for remembering to delete image
Now that we delete the images elsewhere it's not required. Additionally
it's safe to do as we never released an upstream version which required
this being in place.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:36 +01:00
Peter Krempa 40485059ab qemu: backup: Move deletion of backup images to job termination
While qemu is running both locations are identical in semantics, but the
move will allow us to fix the scenario when the VM is destroyed or
crashes where we'd leak the images.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:35 +01:00
Peter Krempa d6b994bafd qemu: backup: Configure backup store image with backing file
In contrast to snapshots the backup job does not complain when the
backup job's store file has backing pre-configured. It's actually
required so that the NBD server exposes all the data properly.

Remove our fake termination and use the existing disk source as backing.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:35 +01:00
Peter Krempa 728b993c8a qemu: Reset the node-name allocator in qemuDomainObjPrivateDataClear
qemuDomainObjPrivateDataClear clears state which become invalid after VM
stopped running and the node name allocator belongs there.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:35 +01:00
Peter Krempa bae81b8e76 qemu: block: Use proper asyncJob when waiting for completion of blockdev-create
The waiting loop used QEMU_ASYNC_JOB_NONE rather than 'asyncJob' passed
from the caller.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-01-06 10:15:35 +01:00
Peter Krempa 3e719fe949 test: qemucaps: Refresh x86_64 caps probe data for the qemu-4.2 release
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
2019-12-18 09:49:31 +01:00
Peter Krempa 5949ac0f59 kbase: Add document outlining backing chain XML config and troubleshooting
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2019-12-18 09:36:49 +01:00
Peter Krempa 3615e8b39b util: storage: Don't treat files with missing backing store format as 'raw'
Assuming that the backing image format is raw is wrong when doing image
detection:

1) In -drive mode qemu will still probe the image format of the backing
   image. This means it will try to open a backing file of the image
   which will fail if a more advanced security model is in use.

2) In blockdev mode the image will be opened as raw actually which is
   wrong since it might be qcow. Not opening the backing images will
   also end up in the guest seeing corrupted data.

Rather than attempt to solve various corner cases when us assuming the
storage file being raw and actually being right forbid startup when the
guest image doesn't have the format specified in the metadata.

https://bugzilla.redhat.com/show_bug.cgi?id=1588373

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2019-12-18 09:36:48 +01:00
Peter Krempa a649369480 tests: storage: Remove unused test modes
EXP_WARN and ALLOW_PROBE flags for the testStorageChain cases are no
longer used so we can remove them.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2019-12-18 09:36:48 +01:00
Peter Krempa 7e582fe995 tests: storage: Use strict version of virStorageFileGetMetadata
Pass in 'true' as '@report_broken' of virStorageFileGetMetadata to make
it fail in the tests. The most important code paths (when starting the
VM) expect this function to fail rather than silently return partial
data. Switch the test to exercise this more important code path.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2019-12-18 09:36:48 +01:00
Peter Krempa a9fdcb0b78 tests: securityselinuxlabel: Add QEMU_CAPS_VNC to fake qemuCaps
In commit 45270337f0 forgot to make sure that tests pass.
Add the missing capability to fix the test.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
2019-12-17 10:03:35 +01:00
Peter Krempa 9851a0f390 tests: qemublock: Add tests for cross-snapshot incremental backups
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 450888d96b qemu: backup: Merge bitmaps accross the backing chain
To allow backups work across external snapshots we need to improve the
algorithm which calculates which bitmaps to merge.

The algorithm must look for appropriately named bitmaps in the image and
possibly descend into a backing image if the current image does not have
the bitmap.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa b7f2cd4652 qemu: block: Introduce qemuBlockNamedNodeDataGetBitmapByName
This function looks up a named bitmap for a virStorageSource in the data
returned from query-named-block-nodes.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 7dfd5d8d81 tests: qemublock: Add testing of bitmap merging for incremental backups
Add test code which will crawl a fake internal list of checkpoints and
generate the list of bitmaps for merging to gather the final bitmap for
the backup.

The initial tests cover the basic case of all bitmaps being present in
the top layer of the backing chain.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 59999670f2 qemu: backup: Export qemuBackupDiskPrepareOneBitmapsChain for tests
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 775228dccf qemu: backup: Propagate bitmap metadata into qemuBackupDiskPrepareOneBitmapsChain
The function will require the bitmap topology for the full
implementation. To facilitate testing, add the propagation of the
necessary data beforehand so that the test code can stay unchanged
during the changes.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 3323e85bf6 qemu: backup: Extract calculations of bitmaps to merge for incremental backup
Separate the for now incomplete code that collects the bitmaps to be
merged for an incremental backup into a separate function. This will
allow adding testing prior to the improvement of the algorithm to
include snapshots.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa d0e829e232 qemu: backup: Return 'def' instead of 'obj' from qemuBackupBeginCollectIncrementalCheckpoints
The object itself has no extra value and it would make testing the code
harder. Refactor it to remove just the definition pointer.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 0b27b655b1 tests: qemublock: Add test case for detecting bitmaps as we create snapshots
Add test data gathered from a run of qemu after creating bitmaps and
snapshots together in various combinations.

The following sequence of commands was used to achieve the
configuration:

virsh checkpoint-create-as VM --name a
virsh snapshot-create-as VM --disk-only
virsh snapshot-create-as VM --disk-only
virsh checkpoint-create-as VM --name b
virsh checkpoint-create-as VM --name c
virsh snapshot-create-as VM --disk-only
virsh checkpoint-create-as VM --name d
virsh snapshot-create-as VM --disk-only
virsh checkpoint-create-as VM --name current

Note that VM was restarted after these operations to allow renumbering
of the bitmaps in a more human-readable way.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 3283445212 qemu: snapshot: Propagate active bitmaps through external snapshots
Re-create any active persistent bitmap in the snapshot overlay image so
that tracking for a checkpoint is persisted. While this basically
duplicates data in the allocation map it's currently the only possible
way as qemu can't mirror the allocation map into a dirty bitmap if we'd
ever want to do a backup.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa f1bc1f0fe5 qemu: monitor: Add 'granularity' parameter for block-dirty-bitmap-add
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 1946485207 qemu: snapshot: Fold formatting of snapshot transaction into prepare func
qemuDomainSnapshotDiskPrepareOne is already called for each disk which
is member of the snapshot so we don't need to iterate through the
snapshot list again to generate members of the 'transaction' command for
each snapshot.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 4258eba6ed qemu: Check for explicit failure of qemuBlockSnapshotAddBlockdev
Check that the value is less than 0.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 1326fd1ce6 tests: qemublocktest: Add a synthetic test case for bitmap detection
The real data gathered for the 'basic' test case don't exercise some
fields. Add a copy with a few values modified manually.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 9aac9d5bda tests: qemublock: Add test for bitmap detection
Test the extraction of data about changed block tracking bitmaps. The
first test case adds a simple scenario of multiple bitmaps in one layer.

The test data will be also later reused for testing the code that
determines which bitmaps to merge for an incremental backup.

The sequence of bitmaps was created by the libvirt checkpoint API with
the following sequence of commands:

virsh checkpoint-create-as VM --name a
virsh checkpoint-create-as VM --name b
virsh checkpoint-create-as VM --name c
virsh checkpoint-create-as VM --name d
virsh checkpoint-create-as VM --name current

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 7a185d2f6e qemu: monitor: Extract internals of qemuMonitorJSONBlockGetNamedNodeData
For testing purposes it will be beneficial to be able to parse the data
from JSON directly rather than trying to simulate the monitor. Extract
the worker bits and export them.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa fdb22e4e43 qemu: monitor: Extract data about dirty-bimaps in qemuMonitorBlockGetNamedNodeData
We will need to inspect the presence and attributes for dirty bitmaps.
Extract them when processing reply of query-named-block-nodes.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-13 13:22:55 +01:00
Peter Krempa 1a2934d61c qemu: Add support for VIR_DOMAIN_CAPS_FEATURE_BACKUP
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa 528191c754 conf: domaincaps: Add 'backup' feature flag
This flag will allow figuring out whether the hypervisor supports the
incremental backup and checkpoint features.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa 9400302477 qemu: blockjob: Implement concluded blockjob handler for backup blockjobs
After the individual sub-blockjobs of a backup libvirt job finish we
must detect it and notify the parent job, so that it can be properly
terminated.

Since we update job information to determine success of a blockjob we
can directly report back also statistics of the blockjob.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa e0daa03a35 qemu: driver: Allow cancellation of the backup job
Use the helper which cancels all blockjobs to perform the backup job
cancellation in qemuDomainAbortJob.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa 5ea6cec9ef qemu: backup: Implement stats gathering while the job is running
We can use the output of 'query-jobs' to figure out some useful
information about a backup job. That is progress in case of a push job
and scratch file use in case of a pull job.

Add a worker which will total up the data and call it from
qemuDomainGetJobStatsInternal.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa a1521f84a5 qemu: Implement backup job APIs and qemu handling
This allows to start and manage the backup job.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa e8ae2ddbb1 doc: Document quirk of getting block job info for a 'backup' blockjob
The stats reported for a blockjob which is member of a domain pull
backup refer to the utilization of the scratch file rather than the
progress of the backup as the progress of the backup depends on the
client. Note this quirk in the docs.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa 86d2866a90 conf: backup: Add fields for tracking stats of completed sub-jobs
We need a place to store stats of completed sub-jobs so that we can
later report accurate stats.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:58 +01:00
Peter Krempa 1b412fb7bf tests: qemustatusxml2xml: Add test for 'pull' type backup job
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa 2c59f0083e qemu: blockjob: Track internal data for 'backup' blockjob
A backup blockjob needs to be able to notify the parent backup job as
well as track all data to be able to clean up the bitmap and blockdev
used for the backup.

Add the data structure, job allocation function and status XML formatter
and parser.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa 9ac4b7db8d qemu: domain: Track backup job data in the status XML
Store the data of a backup job along with the index counter for new
backup jobs in the status XML. Currently we will support only one
backup job and thus there's no necessity to add arrays of jobs.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa e87120307a qemu: monitor: Add support for blockdev-backup via 'transaction'
Implement the transaction actions generator for blockdev-backup.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa 08b810053a Add 'backup' block job type
A backup job may consist of many backup sub-blockjobs. Add the new
blockjob type and add all type converter strings.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa 3656bb0a13 qemu: domain: Introduce QEMU_ASYNC_JOB_BACKUP async job type
We will want to use the async job infrastructure along with all the APIs
and event for the backup job so add the backup job as a new async job
type.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa fed2cb487b qemu: Add infrastructure for statistics of a backup job
Introduce QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP and the convertors and other
plumbing to be able to report statistics for the backup job.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:57 +01:00
Peter Krempa d0805c27f5 tests: genericxml2xml: Add testing of backup XML files
Now that the parser and formatter are in place we can exercise it on
the test files.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:56 +01:00
Peter Krempa 9e408454a8 API: Add domain job operation for backups
Introduce VIR_DOMAIN_JOB_OPERATION_BACKUP into virDomainJobOperation.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:56 +01:00
Peter Krempa c3e18c13ca virsh: Implement VIR_DOMAIN_JOB_DISK_TEMP_(USED|TOTAL) in cmdDomjobinfo
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:56 +01:00
Peter Krempa 2a2f26d321 API: Introduce field for reporting temporary disk space usage of a domain job
A pull mode backup job uses temporary disk images to hold the changed
parts of the disk while the client is copying the changes. Since usage
of the temporary space can be monitored but doesn't really fit any of
the existing stats fields introduce new fields for reporting this data.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:56 +01:00
Peter Krempa 1e320c9d83 qemu: domain: Export qemuDomainGetImageIds
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-10 12:41:56 +01:00
Peter Krempa 6f6a1763a1 qemu: snapshot: Mark file becoming backingStore as read-only
For any backing file we set 'read-only' to true, but didn't do this when
modifying the recorded backing store when creating external snapshots.

This meant that qemu would attempt to open the backing-file read-write.
This would fail for example when selinux is used as qemu doesn't have
write permission for the backing file.

https://bugzilla.redhat.com/show_bug.cgi?id=1781079

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-09 14:07:24 +01:00
Peter Krempa 35614b6022 xen: Remove unused 'cfg'
Refactoring of the XML parser left few instances of the
libxlDriverConfig object unused. Remove them.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
2019-12-09 14:06:15 +01:00
Peter Krempa f5259ba402 qemu: migration: Properly setup mirror for blockdev configurations
With blockdev we need to refer to the nodename of the disk source image
as the source argument for the blockdev-mirror operation while still
keeping the old job name. With blockdev we must also persist the job in
qemu.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 3e35156bd1 qemu: migration: Mention disk target rather than the drive name in debug msg
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 69abc80f5e qemu: migration: Split out setup of the migration target
Separate out allocation of the virStorageSource corresponding to the
target NBD export of the migration.

As part of the splitout we allocate the export name explicitly as that
one must not change regardless whether blockdev is used or not to
provide compatibility.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 0d0b490a32 qemu: blockjob: Allow NULL 'mirror' for block copy jobs due to migration
The non-shared-storage migration tracks the storage source used
explicitly in the migration data so we must allow for processing of the
block job which has NULL mirror as the mirror will not be populated.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa e1910a1f3b qemu: migration: Simplify cleanup in qemuMigrationSrcNBDCopyCancelOne
Now that the cleanup section does not exist remove the label.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 37e801340a qemu: migration: Access job name from job struct
qemuMigrationSrcNBDCopyCancelOne uses the block job data structure but
generated it's own job name rather than taking it from the block job
data.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 5339f57a0b qemu: migration: Properly export backend for NBD storage migration
With -blockdev we must use the nodename as the export but we must keep
the name of the export as it was before to ensure compatiblity.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa 83137f5d37 qemu: migration: Simplify handling of 'diskAlias' when adding NBD exports
Declare the variable inside the loop with automatic clearing.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa d179acf4ad qemu: driver: Use appropriate job name when setting blockjob speed
qemuDomainBlockJobSetSpeed was not converted to get the job name from
the block job data. This means that after enabling blockdev the API call
would fail as we wouldn't use the appropriate name.

https://bugzilla.redhat.com/show_bug.cgi?id=1780497

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:24 +01:00
Peter Krempa e3faa84819 qemu: driver: Use qemuDomainDiskByName instead of virDomainDiskByName
Where appropriate replace the open coded call with the qemu wrapper
which already reports the error.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:23 +01:00
Peter Krempa ca67d02fbe qemu: domain: Mention searched disk in error of qemuDomainDiskByName
Mention the argument used if the disk can't be located.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2019-12-09 07:11:23 +01:00
Peter Krempa 6865dc295e qemu: Convert aborting of a domain job to a switch statement
Until now we only really aborted migration via qemuDomainAbortJob. This
will change with the upcoming addition of the backup job. Additionally
there were a bunch of if statements checking various aspects of the
current job.

To make it more obvious convert qemuDomainAbortJob to use a switch
statement and move the individual conditions to the appropriate job
type.

Every job type has now it's own case despite multiple job types just
plainly cancelling the job for clarity and future extension.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2019-12-05 13:53:15 +01:00
Peter Krempa 4ef3401eb8 qemu: driver: Split out cancellation of migration from qemuDomainAbortJob
Following patch will refactor qemuDomainAbortJob to use a per-job-type
switch where we will need to abort a migration job in various branches.

Save some code duplication by introducing a helper.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2019-12-05 13:53:15 +01:00
Peter Krempa cd1e6fd478 qemu: process: Re-process qemu capability lockout in qemuProcessPrepareQEMUCaps
We clear some capabilities here so the lockouts need to be
re-evaluated.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com>
2019-12-03 15:26:54 +01:00
Peter Krempa be1a201533 qemu: capabilities: Lock out incremental backup capability without blockdev
Blockdev is required to do incremental backups properly. Add a helper
function for locking out capabilities and export it to allow re-doing
the processing if a different code path modifies capabilities.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com>
2019-12-03 15:26:54 +01:00
Peter Krempa a64265f440 qemu: caps: Don't check capability before clearing it
Checking whether a qemu capability set right before clearing it without
any other logic doesn't make sense.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com>
2019-12-03 15:26:54 +01:00
Peter Krempa 78c2a8b934 qemu: process: Move handling of qemu capability overrides
Do all post-processing of capabilities in qemuProcessPrepareQEMUCaps.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com>
2019-12-03 15:26:54 +01:00
Peter Krempa 97c9ece79b qemu: process: Move clearing of QEMU_CAPS_CHARDEV_FD_PASS to qemuProcessPrepareQEMUCaps
Move the post-processing of the QEMU_CAPS_CHARDEV_FD_PASS flag to the
new function.

The clearing of the capability is based on the presence of
VIR_QEMU_PROCESS_START_STANDALONE so we must also pass in the process
start flags.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com>
2019-12-03 15:26:54 +01:00
Peter Krempa 3a075524d9 qemu: process: Move clearing of the BLOCKDEV capability to qemuProcessPrepareQEMUCaps
Start aggregating all capability post-processing code in one place.

The comment was modified while moving it as it was mentioning floppies
which are no longer clearing the blockdev capability.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2019-12-03 15:26:54 +01:00