fix(xapi): support ISO 8601 datetime format

Fixes #6082

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,34 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'status: triaging :triangular_flag_on_post:, type: bug :bug:'
+assignees: ''
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+
+- Node: [e.g. 16.12.1]
+- xo-server: [e.g. 5.82.3]
+- xo-web: [e.g. 5.87.0]
+- hypervisor: [e.g. XCP-ng 8.2.0]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

@vates/async-each/.npmignore

@@ -0,0 +1 @@
+../../scripts/npmignore

@vates/async-each/README.md

@@ -0,0 +1,68 @@
+<!-- DO NOT EDIT MANUALLY, THIS FILE HAS BEEN GENERATED -->
+
+# @vates/async-each
+
+[![Package Version](https://badgen.net/npm/v/@vates/async-each)](https://npmjs.org/package/@vates/async-each) ![License](https://badgen.net/npm/license/@vates/async-each) [![PackagePhobia](https://badgen.net/bundlephobia/minzip/@vates/async-each)](https://bundlephobia.com/result?p=@vates/async-each) [![Node compatibility](https://badgen.net/npm/node/@vates/async-each)](https://npmjs.org/package/@vates/async-each)
+
+> Run async fn for each item in (async) iterable
+
+## Install
+
+Installation of the [npm package](https://npmjs.org/package/@vates/async-each):
+
+```
+> npm install --save @vates/async-each
+```
+
+## Usage
+
+### `asyncEach(iterable, iteratee, [opts])`
+
+Executes `iteratee` in order for each value yielded by `iterable`.
+
+Returns a promise wich rejects as soon as a call to `iteratee` throws or a promise returned by it rejects, and which resolves when all promises returned by `iteratee` have resolved.
+
+`iterable` must be an iterable or async iterable.
+
+`iteratee` is called with the same `this` value as `asyncEach`, and with the following arguments:
+
+- `value`: the value yielded by `iterable`
+- `index`: the 0-based index for this value
+- `iterable`: the iterable itself
+
+`opts` is an object that can contains the following options:
+
+- `concurrency`: a number which indicates the maximum number of parallel call to `iteratee`, defaults to `1`
+- `signal`: an abort signal to stop the iteration
+- `stopOnError`: wether to stop iteration of first error, or wait for all calls to finish and throw an `AggregateError`, defaults to `true`
+
+```js
+import { asyncEach } from '@vates/async-each'
+
+const contents = []
+await asyncEach(
+  ['foo.txt', 'bar.txt', 'baz.txt'],
+  async function (filename, i) {
+    contents[i] = await readFile(filename)
+  },
+  {
+    // reads two files at a time
+    concurrency: 2,
+  }
+)
+```
+
+## Contributions
+
+Contributions are _very_ welcomed, either on the documentation or on
+the code.
+
+You may:
+
+- report any [issue](https://github.com/vatesfr/xen-orchestra/issues)
+  you've encountered;
+- fork and create a pull request.
+
+## License
+
+[ISC](https://spdx.org/licenses/ISC) © [Vates SAS](https://vates.fr)

@vates/async-each/USAGE.md

@@ -0,0 +1,35 @@
+### `asyncEach(iterable, iteratee, [opts])`
+
+Executes `iteratee` in order for each value yielded by `iterable`.
+
+Returns a promise wich rejects as soon as a call to `iteratee` throws or a promise returned by it rejects, and which resolves when all promises returned by `iteratee` have resolved.
+
+`iterable` must be an iterable or async iterable.
+
+`iteratee` is called with the same `this` value as `asyncEach`, and with the following arguments:
+
+- `value`: the value yielded by `iterable`
+- `index`: the 0-based index for this value
+- `iterable`: the iterable itself
+
+`opts` is an object that can contains the following options:
+
+- `concurrency`: a number which indicates the maximum number of parallel call to `iteratee`, defaults to `1`
+- `signal`: an abort signal to stop the iteration
+- `stopOnError`: wether to stop iteration of first error, or wait for all calls to finish and throw an `AggregateError`, defaults to `true`
+
+```js
+import { asyncEach } from '@vates/async-each'
+
+const contents = []
+await asyncEach(
+  ['foo.txt', 'bar.txt', 'baz.txt'],
+  async function (filename, i) {
+    contents[i] = await readFile(filename)
+  },
+  {
+    // reads two files at a time
+    concurrency: 2,
+  }
+)
+```

@vates/async-each/index.js

@@ -0,0 +1,99 @@
+'use strict'
+
+const noop = Function.prototype
+
+class AggregateError extends Error {
+  constructor(errors, message) {
+    super(message)
+    this.errors = errors
+  }
+}
+
+exports.asyncEach = function asyncEach(iterable, iteratee, { concurrency = 1, signal, stopOnError = true } = {}) {
+  return new Promise((resolve, reject) => {
+    const it = (iterable[Symbol.iterator] || iterable[Symbol.asyncIterator]).call(iterable)
+    const errors = []
+    let running = 0
+    let index = 0
+
+    let onAbort
+    if (signal !== undefined) {
+      onAbort = () => {
+        onRejectedWrapper(new Error('asyncEach aborted'))
+      }
+      signal.addEventListener('abort', onAbort)
+    }
+
+    const clean = () => {
+      onFulfilled = onRejected = noop
+      if (onAbort !== undefined) {
+        signal.removeEventListener('abort', onAbort)
+      }
+    }
+
+    resolve = (resolve =>
+      function resolveAndClean(value) {
+        resolve(value)
+        clean()
+      })(resolve)
+    reject = (reject =>
+      function rejectAndClean(reason) {
+        reject(reason)
+        clean()
+      })(reject)
+
+    let onFulfilled = value => {
+      --running
+      next()
+    }
+    const onFulfilledWrapper = value => onFulfilled(value)
+
+    let onRejected = stopOnError
+      ? reject
+      : error => {
+          --running
+          errors.push(error)
+          next()
+        }
+    const onRejectedWrapper = reason => onRejected(reason)
+
+    let nextIsRunning = false
+    let next = async () => {
+      if (nextIsRunning) {
+        return
+      }
+      nextIsRunning = true
+      if (running < concurrency) {
+        const cursor = await it.next()
+        if (cursor.done) {
+          next = () => {
+            if (running === 0) {
+              if (errors.length === 0) {
+                resolve()
+              } else {
+                reject(new AggregateError(errors))
+              }
+            }
+          }
+        } else {
+          ++running
+          try {
+            const result = iteratee.call(this, cursor.value, index++, iterable)
+            let then
+            if (result != null && typeof result === 'object' && typeof (then = result.then) === 'function') {
+              then.call(result, onFulfilledWrapper, onRejectedWrapper)
+            } else {
+              onFulfilled(result)
+            }
+          } catch (error) {
+            onRejected(error)
+          }
+        }
+        nextIsRunning = false
+        return next()
+      }
+      nextIsRunning = false
+    }
+    next()
+  })
+}

@vates/async-each/index.spec.js

@@ -0,0 +1,99 @@
+'use strict'
+
+/* eslint-env jest */
+
+const { asyncEach } = require('./')
+
+const randomDelay = (max = 10) =>
+  new Promise(resolve => {
+    setTimeout(resolve, Math.floor(Math.random() * max + 1))
+  })
+
+const rejectionOf = p =>
+  new Promise((resolve, reject) => {
+    p.then(reject, resolve)
+  })
+
+describe('asyncEach', () => {
+  const thisArg = 'qux'
+  const values = ['foo', 'bar', 'baz']
+
+  Object.entries({
+    'sync iterable': () => values,
+    'async iterable': async function* () {
+      for (const value of values) {
+        await randomDelay()
+        yield value
+      }
+    },
+  }).forEach(([what, getIterable]) =>
+    describe('with ' + what, () => {
+      let iterable
+      beforeEach(() => {
+        iterable = getIterable()
+      })
+
+      it('works', async () => {
+        const iteratee = jest.fn(async () => {})
+
+        await asyncEach.call(thisArg, iterable, iteratee)
+
+        expect(iteratee.mock.instances).toEqual(Array.from(values, () => thisArg))
+        expect(iteratee.mock.calls).toEqual(Array.from(values, (value, index) => [value, index, iterable]))
+      })
+      ;[1, 2, 4].forEach(concurrency => {
+        it('respects a concurrency of ' + concurrency, async () => {
+          let running = 0
+
+          await asyncEach(
+            values,
+            async () => {
+              ++running
+              expect(running).toBeLessThanOrEqual(concurrency)
+              await randomDelay()
+              --running
+            },
+            { concurrency }
+          )
+        })
+      })
+
+      it('stops on first error when stopOnError is true', async () => {
+        const error = new Error()
+        const iteratee = jest.fn((_, i) => {
+          if (i === 1) {
+            throw error
+          }
+        })
+
+        expect(await rejectionOf(asyncEach(iterable, iteratee, { stopOnError: true }))).toBe(error)
+        expect(iteratee).toHaveBeenCalledTimes(2)
+      })
+
+      it('rejects AggregateError when stopOnError is false', async () => {
+        const errors = []
+        const iteratee = jest.fn(() => {
+          const error = new Error()
+          errors.push(error)
+          throw error
+        })
+
+        const error = await rejectionOf(asyncEach(iterable, iteratee, { stopOnError: false }))
+        expect(error.errors).toEqual(errors)
+        expect(iteratee.mock.calls).toEqual(Array.from(values, (value, index) => [value, index, iterable]))
+      })
+
+      it('can be interrupted with an AbortSignal', async () => {
+        const ac = new AbortController()
+        const iteratee = jest.fn((_, i) => {
+          if (i === 1) {
+            ac.abort()
+          }
+        })
+
+        await expect(asyncEach(iterable, iteratee, { signal: ac.signal })).rejects.toThrow('asyncEach aborted')
+        expect(iteratee).toHaveBeenCalledTimes(2)
+      })
+    })
+  )
+})

@vates/async-each/package.json

@@ -0,0 +1,34 @@
+{
+  "private": false,
+  "name": "@vates/async-each",
+  "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@vates/async-each",
+  "description": "Run async fn for each item in (async) iterable",
+  "keywords": [
+    "array",
+    "async",
+    "collection",
+    "each",
+    "for",
+    "foreach",
+    "iterable",
+    "iterator"
+  ],
+  "bugs": "https://github.com/vatesfr/xen-orchestra/issues",
+  "repository": {
+    "directory": "@vates/async-each",
+    "type": "git",
+    "url": "https://github.com/vatesfr/xen-orchestra.git"
+  },
+  "author": {
+    "name": "Vates SAS",
+    "url": "https://vates.fr"
+  },
+  "license": "ISC",
+  "version": "0.1.0",
+  "engines": {
+    "node": ">=8.10"
+  },
+  "scripts": {
+    "postversion": "npm publish --access public"
+  }
+}

@vates/compose/README.md

@@ -65,6 +65,23 @@ const f = compose(
 )
 ```
 
+Functions can receive extra parameters:
+
+```js
+const isIn = (value, min, max) => min <= value && value <= max
+
+// Only compatible when `fns` is passed as an array!
+const f = compose([
+  [add, 2],
+  [isIn, 3, 10],
+])
+
+console.log(f(1))
+// → true
+```
+
+> Note: if the first function is defined with extra parameters, it will only receive the first value passed to the composed function, instead of all the parameters.
+
 ## Contributions
 
 Contributions are _very_ welcomed, either on the documentation or on

@vates/compose/USAGE.md

@@ -46,3 +46,20 @@ const f = compose(
   [add2, mul3]
 )
 ```
+
+Functions can receive extra parameters:
+
+```js
+const isIn = (value, min, max) => min <= value && value <= max
+
+// Only compatible when `fns` is passed as an array!
+const f = compose([
+  [add, 2],
+  [isIn, 3, 10],
+])
+
+console.log(f(1))
+// → true
+```
+
+> Note: if the first function is defined with extra parameters, it will only receive the first value passed to the composed function, instead of all the parameters.

@vates/compose/index.js

@@ -4,11 +4,13 @@ const defaultOpts = { async: false, right: false }
 
 exports.compose = function compose(opts, fns) {
   if (Array.isArray(opts)) {
-    fns = opts
+    fns = opts.slice() // don't mutate passed array
     opts = defaultOpts
   } else if (typeof opts === 'object') {
     opts = Object.assign({}, defaultOpts, opts)
-    if (!Array.isArray(fns)) {
+    if (Array.isArray(fns)) {
+      fns = fns.slice() // don't mutate passed array
+    } else {
       fns = Array.prototype.slice.call(arguments, 1)
     }
   } else {
@@ -20,6 +22,24 @@ exports.compose = function compose(opts, fns) {
   if (n === 0) {
     throw new TypeError('at least one function must be passed')
   }
+
+  for (let i = 0; i < n; ++i) {
+    const entry = fns[i]
+    if (Array.isArray(entry)) {
+      const fn = entry[0]
+      const args = entry.slice()
+      args[0] = undefined
+      fns[i] = function composeWithArgs(value) {
+        args[0] = value
+        try {
+          return fn.apply(this, args)
+        } finally {
+          args[0] = undefined
+        }
+      }
+    }
+  }
+
   if (n === 1) {
     return fns[0]
   }

@vates/compose/package.json

@@ -14,7 +14,7 @@
     "url": "https://vates.fr"
   },
   "license": "ISC",
-  "version": "2.0.0",
+  "version": "2.1.0",
   "engines": {
     "node": ">=7.6"
   },

@vates/decorate-with/README.md

@@ -16,7 +16,9 @@ Installation of the [npm package](https://npmjs.org/package/@vates/decorate-with
 
 ## Usage
 
-For instance, allows using Lodash's functions as decorators:
+### `decorateWith(fn, ...args)`
+
+Creates a new ([legacy](https://babeljs.io/docs/en/babel-plugin-syntax-decorators#legacy)) method decorator from a function decorator, for instance, allows using Lodash's functions as decorators:
 
 ```js
 import { decorateWith } from '@vates/decorate-with'
@@ -29,6 +31,64 @@ class Foo {
 }
 ```
 
+### `decorateMethodsWith(class, map)`
+
+Decorates a number of methods directly, without using the decorator syntax:
+
+```js
+import { decorateMethodsWith } from '@vates/decorate-with'
+
+class Foo {
+  bar() {
+    // body
+  }
+
+  baz() {
+    // body
+  }
+}
+
+decorateMethodsWith(Foo, {
+  // without arguments
+  bar: lodash.curry,
+
+  // with arguments
+  baz: [lodash.debounce, 150],
+})
+```
+
+The decorated class is returned, so you can export it directly.
+
+To apply multiple transforms to a method, you can either call `decorateMethodsWith` multiple times or use [`@vates/compose`](https://www.npmjs.com/package/@vates/compose):
+
+```js
+decorateMethodsWith(Foo, {
+  bar: compose([
+    [lodash.debounce, 150]
+    lodash.curry,
+  ])
+})
+```
+
+### `perInstance(fn, ...args)`
+
+Helper to decorate the method by instance instead of for the whole class.
+
+This is often necessary for caching or deduplicating calls.
+
+```js
+import { perInstance } from '@vates/decorateWith'
+
+class Foo {
+  @decorateWith(perInstance, lodash.memoize)
+  bar() {
+    // body
+  }
+}
+```
+
+Because it's a normal function, it can also be used with `decorateMethodsWith`, with `compose` or even by itself.
+
 ## Contributions
 
 Contributions are _very_ welcomed, either on the documentation or on

@vates/decorate-with/USAGE.md

@@ -1,4 +1,6 @@
-For instance, allows using Lodash's functions as decorators:
+### `decorateWith(fn, ...args)`
+
+Creates a new ([legacy](https://babeljs.io/docs/en/babel-plugin-syntax-decorators#legacy)) method decorator from a function decorator, for instance, allows using Lodash's functions as decorators:
 
 ```js
 import { decorateWith } from '@vates/decorate-with'
@@ -10,3 +12,61 @@ class Foo {
   }
 }
 ```
+
+### `decorateMethodsWith(class, map)`
+
+Decorates a number of methods directly, without using the decorator syntax:
+
+```js
+import { decorateMethodsWith } from '@vates/decorate-with'
+
+class Foo {
+  bar() {
+    // body
+  }
+
+  baz() {
+    // body
+  }
+}
+
+decorateMethodsWith(Foo, {
+  // without arguments
+  bar: lodash.curry,
+
+  // with arguments
+  baz: [lodash.debounce, 150],
+})
+```
+
+The decorated class is returned, so you can export it directly.
+
+To apply multiple transforms to a method, you can either call `decorateMethodsWith` multiple times or use [`@vates/compose`](https://www.npmjs.com/package/@vates/compose):
+
+```js
+decorateMethodsWith(Foo, {
+  bar: compose([
+    [lodash.debounce, 150]
+    lodash.curry,
+  ])
+})
+```
+
+### `perInstance(fn, ...args)`
+
+Helper to decorate the method by instance instead of for the whole class.
+
+This is often necessary for caching or deduplicating calls.
+
+```js
+import { perInstance } from '@vates/decorateWith'
+
+class Foo {
+  @decorateWith(perInstance, lodash.memoize)
+  bar() {
+    // body
+  }
+}
+```
+
+Because it's a normal function, it can also be used with `decorateMethodsWith`, with `compose` or even by itself.

@vates/decorate-with/index.js

@@ -1,4 +1,33 @@
-exports.decorateWith = (fn, ...args) => (target, name, descriptor) => ({
-  ...descriptor,
-  value: fn(descriptor.value, ...args),
-})
+exports.decorateWith = function decorateWith(fn, ...args) {
+  return (target, name, descriptor) => ({
+    ...descriptor,
+    value: fn(descriptor.value, ...args),
+  })
+}
+
+const { getOwnPropertyDescriptor, defineProperty } = Object
+
+exports.decorateMethodsWith = function decorateMethodsWith(klass, map) {
+  const { prototype } = klass
+  for (const name of Object.keys(map)) {
+    const descriptor = getOwnPropertyDescriptor(prototype, name)
+    const { value } = descriptor
+
+    const decorator = map[name]
+    descriptor.value = typeof decorator === 'function' ? decorator(value) : decorator[0](value, ...decorator.slice(1))
+    defineProperty(prototype, name, descriptor)
+  }
+  return klass
+}
+
+exports.perInstance = function perInstance(fn, decorator, ...args) {
+  const map = new WeakMap()
+  return function () {
+    let decorated = map.get(this)
+    if (decorated === undefined) {
+      decorated = decorator(fn, ...args)
+      map.set(this, decorated)
+    }
+    return decorated.apply(this, arguments)
+  }
+}

@vates/decorate-with/package.json

@@ -20,7 +20,7 @@
     "url": "https://vates.fr"
   },
   "license": "ISC",
-  "version": "0.0.1",
+  "version": "1.0.0",
   "engines": {
     "node": ">=8.10"
   },

@vates/disposable/package.json

@@ -24,7 +24,7 @@
   "dependencies": {
     "@vates/multi-key-map": "^0.1.0",
     "@xen-orchestra/async-map": "^0.1.2",
-    "@xen-orchestra/log": "^0.2.0",
+    "@xen-orchestra/log": "^0.3.0",
     "ensure-array": "^1.0.0"
   }
 }

@xen-orchestra/async-map/index.js

@@ -18,17 +18,6 @@ const wrapCall = (fn, arg, thisArg) => {
  * @returns {Promise<Item[]>}
  */
 exports.asyncMap = function asyncMap(iterable, mapFn, thisArg = iterable) {
-  let onError
-  if (onError !== undefined) {
-    const original = mapFn
-    mapFn = async function () {
-      try {
-        return await original.apply(this, arguments)
-      } catch (error) {
-        return onError.call(this, error, ...arguments)
-      }
-    }
-  }
   return Promise.all(Array.from(iterable, mapFn, thisArg))
 }
 

@xen-orchestra/audit-core/package.json

@@ -9,7 +9,7 @@
   },
   "version": "0.2.0",
   "engines": {
-    "node": ">=8.10"
+    "node": ">=10"
   },
   "main": "dist/",
   "scripts": {
@@ -30,9 +30,8 @@
     "rimraf": "^3.0.0"
   },
   "dependencies": {
-    "@vates/decorate-with": "^0.0.1",
-    "@xen-orchestra/log": "^0.2.0",
-    "core-js": "^3.6.4",
+    "@vates/decorate-with": "^1.0.0",
+    "@xen-orchestra/log": "^0.3.0",
     "golike-defer": "^0.5.1",
     "object-hash": "^2.0.1"
   },

@xen-orchestra/audit-core/src/index.js

@@ -1,6 +1,3 @@
-// see https://github.com/babel/babel/issues/8450
-import 'core-js/features/symbol/async-iterator'
-
 import assert from 'assert'
 import hash from 'object-hash'
 import { createLogger } from '@xen-orchestra/log'

@xen-orchestra/audit-core/src/specification.ts

@@ -17,10 +17,10 @@ interface Record {
 }
 
 export class AuditCore {
-  constructor(storage: Storage) { }
-  public add(subject: any, event: string, data: any): Promise<Record> { }
-  public checkIntegrity(oldest: string, newest: string): Promise<number> { }
-  public getFrom(newest?: string): AsyncIterator { }
-  public deleteFrom(newest: string): Promise<void> { }
-  public deleteRangeAndRewrite(newest: string, oldest: string): Promise<void> { }
+  constructor(storage: Storage) {}
+  public add(subject: any, event: string, data: any): Promise<Record> {}
+  public checkIntegrity(oldest: string, newest: string): Promise<number> {}
+  public getFrom(newest?: string): AsyncIterator {}
+  public deleteFrom(newest: string): Promise<void> {}
+  public deleteRangeAndRewrite(newest: string, oldest: string): Promise<void> {}
 }

@xen-orchestra/babel-config/index.js

@@ -46,7 +46,7 @@ module.exports = function (pkg, configs = {}) {
 
   return {
     comments: !__PROD__,
-    ignore: __PROD__ ? [/\.spec\.js$/] : undefined,
+    ignore: __PROD__ ? [/\btests?\//, /\.spec\.js$/] : undefined,
     plugins: Object.keys(plugins)
       .map(plugin => [plugin, plugins[plugin]])
       .sort(([a], [b]) => {

@xen-orchestra/backups-cli/commands/clean-vms.js

@@ -10,12 +10,13 @@ const { resolve } = require('path')
 const adapter = new RemoteAdapter(require('@xen-orchestra/fs').getHandler({ url: 'file://' }))
 
 module.exports = async function main(args) {
-  const { _, remove, merge } = getopts(args, {
+  const { _, fix, remove, merge } = getopts(args, {
     alias: {
+      fix: 'f',
       remove: 'r',
       merge: 'm',
     },
-    boolean: ['merge', 'remove'],
+    boolean: ['fix', 'merge', 'remove'],
     default: {
       merge: false,
       remove: false,
@@ -25,7 +26,7 @@ module.exports = async function main(args) {
   await asyncMap(_, async vmDir => {
     vmDir = resolve(vmDir)
     try {
-      await adapter.cleanVm(vmDir, { remove, merge, onLog: log => console.warn(log) })
+      await adapter.cleanVm(vmDir, { fixMetadata: fix, remove, merge, onLog: (...args) => console.warn(...args) })
     } catch (error) {
       console.error('adapter.cleanVm', vmDir, error)
     }

@xen-orchestra/backups-cli/index.js

@@ -5,11 +5,12 @@ require('./_composeCommands')({
     get main() {
       return require('./commands/clean-vms')
     },
-    usage: `[--merge] [--remove] xo-vm-backups/*
+    usage: `[--fix] [--merge] [--remove] xo-vm-backups/*
 
       Detects and repair issues with VM backups.
 
       Options:
+        -f, --fix       Fix metadata issues (like size)
         -m, --merge     Merge (or continue merging) VHD files that are unused
         -r, --remove    Remove unused, incomplete, orphan, or corrupted files
 `,

@xen-orchestra/backups-cli/package.json

@@ -7,12 +7,12 @@
   "bugs": "https://github.com/vatesfr/xen-orchestra/issues",
   "dependencies": {
     "@xen-orchestra/async-map": "^0.1.2",
-    "@xen-orchestra/backups": "^0.11.0",
-    "@xen-orchestra/fs": "^0.17.0",
+    "@xen-orchestra/backups": "^0.18.3",
+    "@xen-orchestra/fs": "^0.19.3",
     "filenamify": "^4.1.0",
     "getopts": "^2.2.5",
     "lodash": "^4.17.15",
-    "promise-toolbox": "^0.19.2"
+    "promise-toolbox": "^0.20.0"
   },
   "engines": {
     "node": ">=7.10.1"
@@ -27,7 +27,7 @@
   "scripts": {
     "postversion": "npm publish --access public"
   },
-  "version": "0.6.0",
+  "version": "0.6.1",
   "license": "AGPL-3.0-or-later",
   "author": {
     "name": "Vates SAS",

@xen-orchestra/backups/RemoteAdapter.js

@@ -3,19 +3,20 @@ const Disposable = require('promise-toolbox/Disposable.js')
 const fromCallback = require('promise-toolbox/fromCallback.js')
 const fromEvent = require('promise-toolbox/fromEvent.js')
 const pDefer = require('promise-toolbox/defer.js')
-const pump = require('pump')
-const { basename, dirname, join, normalize, resolve } = require('path')
+const groupBy = require('lodash/groupBy.js')
+const { dirname, join, normalize, resolve } = require('path')
 const { createLogger } = require('@xen-orchestra/log')
-const { createSyntheticStream, mergeVhd, default: Vhd } = require('vhd-lib')
+const { Constants, createVhdDirectoryFromStream, openVhd, VhdAbstract, VhdDirectory, VhdSynthetic } = require('vhd-lib')
 const { deduped } = require('@vates/disposable/deduped.js')
 const { execFile } = require('child_process')
 const { readdir, stat } = require('fs-extra')
+const { v4: uuidv4 } = require('uuid')
 const { ZipFile } = require('yazl')
 
 const { BACKUP_DIR } = require('./_getVmBackupDir.js')
 const { cleanVm } = require('./_cleanVm.js')
 const { getTmpDir } = require('./_getTmpDir.js')
-const { isMetadataFile, isVhdFile } = require('./_backupType.js')
+const { isMetadataFile } = require('./_backupType.js')
 const { isValidXva } = require('./_isValidXva.js')
 const { listPartitions, LVM_PARTITION_TYPE } = require('./_listPartitions.js')
 const { lvs, pvs } = require('./_lvm.js')
@@ -67,58 +68,17 @@ const debounceResourceFactory = factory =>
   }
 
 class RemoteAdapter {
-  constructor(handler, { debounceResource = res => res, dirMode } = {}) {
+  constructor(handler, { debounceResource = res => res, dirMode, vhdDirectoryCompression } = {}) {
     this._debounceResource = debounceResource
     this._dirMode = dirMode
     this._handler = handler
+    this._vhdDirectoryCompression = vhdDirectoryCompression
   }
 
   get handler() {
     return this._handler
   }
 
-  async _deleteVhd(path) {
-    const handler = this._handler
-    const vhds = await asyncMapSettled(
-      await handler.list(dirname(path), {
-        filter: isVhdFile,
-        prependDir: true,
-      }),
-      async path => {
-        try {
-          const vhd = new Vhd(handler, path)
-          await vhd.readHeaderAndFooter()
-          return {
-            footer: vhd.footer,
-            header: vhd.header,
-            path,
-          }
-        } catch (error) {
-          // Do not fail on corrupted VHDs (usually uncleaned temporary files),
-          // they are probably inconsequent to the backup process and should not
-          // fail it.
-          warn(`BackupNg#_deleteVhd ${path}`, { error })
-        }
-      }
-    )
-    const base = basename(path)
-    const child = vhds.find(_ => _ !== undefined && _.header.parentUnicodeName === base)
-    if (child === undefined) {
-      await handler.unlink(path)
-      return 0
-    }
-
-    try {
-      const childPath = child.path
-      const mergedDataSize = await mergeVhd(handler, path, handler, childPath)
-      await handler.rename(path, childPath)
-      return mergedDataSize
-    } catch (error) {
-      handler.unlink(path).catch(warn)
-      throw error
-    }
-  }
-
   async _findPartition(devicePath, partitionId) {
     const partitions = await listPartitions(devicePath)
     const partition = partitions.find(_ => _.id === partitionId)
@@ -232,6 +192,22 @@ class RemoteAdapter {
     return files
   }
 
+  // check if we will be allowed to merge a a vhd created in this adapter
+  // with the vhd at path `path`
+  async isMergeableParent(packedParentUid, path) {
+    return await Disposable.use(openVhd(this.handler, path), vhd => {
+      // this baseUuid is not linked with this vhd
+      if (!vhd.footer.uuid.equals(packedParentUid)) {
+        return false
+      }
+
+      const isVhdDirectory = vhd instanceof VhdDirectory
+      return isVhdDirectory
+        ? this.#useVhdDirectory() && this.#getCompressionType() === vhd.compressionType
+        : !this.#useVhdDirectory()
+    })
+  }
+
   fetchPartitionFiles(diskId, partitionId, paths) {
     const { promise, reject, resolve } = pDefer()
     Disposable.use(
@@ -253,16 +229,9 @@ class RemoteAdapter {
 
   async deleteDeltaVmBackups(backups) {
     const handler = this._handler
-    let mergedDataSize = 0
-    await asyncMapSettled(backups, ({ _filename, vhds }) =>
-      Promise.all([
-        handler.unlink(_filename),
-        asyncMap(Object.values(vhds), async _ => {
-          mergedDataSize += await this._deleteVhd(resolveRelativeFromFile(_filename, _))
-        }),
-      ])
-    )
-    return mergedDataSize
+
+    // unused VHDs will be detected by `cleanVm`
+    await asyncMapSettled(backups, ({ _filename }) => VhdAbstract.unlink(handler, _filename))
   }
 
   async deleteMetadataBackup(backupId) {
@@ -292,17 +261,34 @@ class RemoteAdapter {
     )
   }
 
-  async deleteVmBackup(filename) {
-    const metadata = JSON.parse(String(await this._handler.readFile(filename)))
-    metadata._filename = filename
+  deleteVmBackup(file) {
+    return this.deleteVmBackups([file])
+  }
 
-    if (metadata.mode === 'delta') {
-      await this.deleteDeltaVmBackups([metadata])
-    } else if (metadata.mode === 'full') {
-      await this.deleteFullVmBackups([metadata])
-    } else {
-      throw new Error(`no deleter for backup mode ${metadata.mode}`)
+  async deleteVmBackups(files) {
+    const { delta, full, ...others } = groupBy(await asyncMap(files, file => this.readVmBackupMetadata(file)), 'mode')
+
+    const unsupportedModes = Object.keys(others)
+    if (unsupportedModes.length !== 0) {
+      throw new Error('no deleter for backup modes: ' + unsupportedModes.join(', '))
     }
+
+    await Promise.all([
+      delta !== undefined && this.deleteDeltaVmBackups(delta),
+      full !== undefined && this.deleteFullVmBackups(full),
+    ])
+  }
+
+  #getCompressionType() {
+    return this._vhdDirectoryCompression
+  }
+
+  #useVhdDirectory() {
+    return this.handler.type === 's3'
+  }
+
+  #useAlias() {
+    return this.#useVhdDirectory()
   }
 
   getDisk = Disposable.factory(this.getDisk)
@@ -361,6 +347,14 @@ class RemoteAdapter {
     return yield this._getPartition(devicePath, await this._findPartition(devicePath, partitionId))
   }
 
+  // if we use alias on this remote, we have to name the file alias.vhd
+  getVhdFileName(baseName) {
+    if (this.#useAlias()) {
+      return `${baseName}.alias.vhd`
+    }
+    return `${baseName}.vhd`
+  }
+
   async listAllVmBackups() {
     const handler = this._handler
 
@@ -505,6 +499,25 @@ class RemoteAdapter {
     return backups.sort(compareTimestamp)
   }
 
+  async writeVhd(path, input, { checksum = true, validator = noop } = {}) {
+    const handler = this._handler
+
+    if (this.#useVhdDirectory()) {
+      const dataPath = `${dirname(path)}/data/${uuidv4()}.vhd`
+      await createVhdDirectoryFromStream(handler, dataPath, input, {
+        concurrency: 16,
+        compression: this.#getCompressionType(),
+        async validator() {
+          await input.task
+          return validator.apply(this, arguments)
+        },
+      })
+      await VhdAbstract.createAlias(handler, path, dataPath)
+    } else {
+      await this.outputStream(path, input, { checksum, validator })
+    }
+  }
+
   async outputStream(path, input, { checksum = true, validator = noop } = {}) {
     await this._handler.outputStream(path, input, {
       checksum,
@@ -516,6 +529,52 @@ class RemoteAdapter {
     })
   }
 
+  async _createSyntheticStream(handler, paths) {
+    let disposableVhds = []
+
+    // if it's a path : open all hierarchy of parent
+    if (typeof paths === 'string') {
+      let vhd,
+        vhdPath = paths
+      do {
+        const disposable = await openVhd(handler, vhdPath)
+        vhd = disposable.value
+        disposableVhds.push(disposable)
+        vhdPath = resolveRelativeFromFile(vhdPath, vhd.header.parentUnicodeName)
+      } while (vhd.footer.diskType !== Constants.DISK_TYPES.DYNAMIC)
+    } else {
+      // only open the list of path given
+      disposableVhds = paths.map(path => openVhd(handler, path))
+    }
+
+    // I don't want the vhds to be disposed on return
+    // but only when the stream is done ( or failed )
+    const disposables = await Disposable.all(disposableVhds)
+    const vhds = disposables.value
+
+    let disposed = false
+    const disposeOnce = async () => {
+      if (!disposed) {
+        disposed = true
+
+        try {
+          await disposables.dispose()
+        } catch (error) {
+          warn('_createSyntheticStream: failed to dispose VHDs', { error })
+        }
+      }
+    }
+
+    const synthetic = new VhdSynthetic(vhds)
+    await synthetic.readHeaderAndFooter()
+    await synthetic.readBlockAllocationTable()
+    const stream = await synthetic.stream()
+    stream.on('end', disposeOnce)
+    stream.on('close', disposeOnce)
+    stream.on('error', disposeOnce)
+    return stream
+  }
+
   async readDeltaVmBackup(metadata) {
     const handler = this._handler
     const { vbds, vdis, vhds, vifs, vm } = metadata
@@ -523,7 +582,7 @@ class RemoteAdapter {
 
     const streams = {}
     await asyncMapSettled(Object.keys(vdis), async id => {
-      streams[`${id}.vhd`] = await createSyntheticStream(handler, join(dir, vhds[id]))
+      streams[`${id}.vhd`] = await this._createSyntheticStream(handler, join(dir, vhds[id]))
     })
 
     return {
@@ -543,40 +602,6 @@ class RemoteAdapter {
   async readVmBackupMetadata(path) {
     return Object.defineProperty(JSON.parse(await this._handler.readFile(path)), '_filename', { value: path })
   }
-
-  async writeFullVmBackup({ jobId, mode, scheduleId, timestamp, vm, vmSnapshot, xva }, sizeContainer, stream) {
-    const basename = formatFilenameDate(timestamp)
-
-    const dataBasename = basename + '.xva'
-    const dataFilename = backupDir + '/' + dataBasename
-
-    const metadataFilename = `${backupDir}/${basename}.json`
-    const metadata = {
-      jobId: job.id,
-      mode: job.mode,
-      scheduleId,
-      timestamp,
-      version: '2.0.0',
-      vm,
-      vmSnapshot: this._backup.exportedVm,
-      xva: './' + dataBasename,
-    }
-
-    const { deleteFirst } = settings
-    if (deleteFirst) {
-      await deleteOldBackups()
-    }
-
-    await adapter.outputStream(stream, dataFilename, {
-      validator: tmpPath => {
-        if (handler._getFilePath !== undefined) {
-          return isValidXva(handler._getFilePath('/' + tmpPath))
-        }
-      },
-    })
-    metadata.size = sizeContainer.size
-    await handler.outputFile(metadataFilename, JSON.stringify(metadata))
-  }
 }
 
 Object.assign(RemoteAdapter.prototype, {

@xen-orchestra/backups/Task.js

@@ -1,11 +1,8 @@
 const CancelToken = require('promise-toolbox/CancelToken.js')
 const Zone = require('node-zone')
 
-const logAfterEnd = function (log) {
-  const error = new Error('task has already ended:' + this.id)
-  error.result = log.result
-  error.log = log
-  throw error
+const logAfterEnd = () => {
+  throw new Error('task has already ended')
 }
 
 const noop = Function.prototype
@@ -47,19 +44,11 @@ class Task {
     }
   }
 
-  get id() {
-    return this.#id
-  }
-
   #cancelToken
   #id = Math.random().toString(36).slice(2)
   #onLog
   #zone
 
-  get id() {
-    return this.#id
-  }
-
   constructor({ name, data, onLog }) {
     let parentCancelToken, parentId
     if (onLog === undefined) {
@@ -111,8 +100,6 @@ class Task {
   run(fn, last = false) {
     return this.#zone.run(() => {
       try {
-        this.#cancelToken.throwIfRequested()
-
         const result = fn()
         let then
         if (result != null && typeof (then = result.then) === 'function') {

@xen-orchestra/backups/_VmBackup.js

@@ -1,10 +1,10 @@
 const assert = require('assert')
-// const asyncFn = require('promise-toolbox/asyncFn')
 const findLast = require('lodash/findLast.js')
+const groupBy = require('lodash/groupBy.js')
 const ignoreErrors = require('promise-toolbox/ignoreErrors.js')
 const keyBy = require('lodash/keyBy.js')
 const mapValues = require('lodash/mapValues.js')
-const { asyncMap, asyncMapSettled } = require('@xen-orchestra/async-map')
+const { asyncMap } = require('@xen-orchestra/async-map')
 const { createLogger } = require('@xen-orchestra/log')
 const { defer } = require('golike-defer')
 const { formatDateTime } = require('@xen-orchestra/xapi')
@@ -36,6 +36,11 @@ const forkDeltaExport = deltaExport =>
 
 exports.VmBackup = class VmBackup {
   constructor({ config, getSnapshotNameLabel, job, remoteAdapters, remotes, schedule, settings, srs, vm }) {
+    if (vm.other_config['xo:backup:job'] === job.id) {
+      // otherwise replicated VMs would be matched and replicated again and again
+      throw new Error('cannot backup a VM created by this very job')
+    }
+
     this.config = config
     this.job = job
     this.remoteAdapters = remoteAdapters
@@ -104,9 +109,21 @@ exports.VmBackup = class VmBackup {
   // calls fn for each function, warns of any errors, and throws only if there are no writers left
   async _callWriters(fn, warnMessage, parallel = true) {
     const writers = this._writers
-    if (writers.size === 0) {
+    const n = writers.size
+    if (n === 0) {
       return
     }
+    if (n === 1) {
+      const [writer] = writers
+      try {
+        await fn(writer)
+      } catch (error) {
+        writers.delete(writer)
+        throw error
+      }
+      return
+    }
+
     await (parallel ? asyncMap : asyncEach)(writers, async function (writer) {
       try {
         await fn(writer)
@@ -144,7 +161,6 @@ exports.VmBackup = class VmBackup {
 
     const doSnapshot =
       this._isDelta || (!settings.offlineBackup && vm.power_state === 'Running') || settings.snapshotRetention !== 0
-    console.log({ doSnapshot })
     if (doSnapshot) {
       await Task.run({ name: 'snapshot' }, async () => {
         if (!settings.bypassVdiChainsCheck) {
@@ -183,7 +199,6 @@ exports.VmBackup = class VmBackup {
     await this._callWriters(writer => writer.prepare({ isFull }), 'writer.prepare()')
 
     const deltaExport = await exportDeltaVm(exportedVm, baseVm, {
-      cancelToken: Task.cancelToken,
       fullVdisRequired,
     })
     const sizeContainers = mapValues(deltaExport.streams, stream => watchStreamSize(stream))
@@ -229,7 +244,6 @@ exports.VmBackup = class VmBackup {
   async _copyFull() {
     const { compression } = this.job
     const stream = await this._xapi.VM_export(this.exportedVm.$ref, {
-      cancelToken: Task.cancelToken,
       compress: Boolean(compression) && (compression === 'native' ? 'gzip' : 'zstd'),
       useSnapshot: false,
     })
@@ -276,17 +290,28 @@ exports.VmBackup = class VmBackup {
   }
 
   async _removeUnusedSnapshots() {
-    // TODO: handle all schedules (no longer existing schedules default to 0 retention)
-
-    const { scheduleId } = this
-    const scheduleSnapshots = this._jobSnapshots.filter(_ => _.other_config['xo:backup:schedule'] === scheduleId)
-
+    const jobSettings = this.job.settings
     const baseVmRef = this._baseVm?.$ref
+    const { config } = this
+    const baseSettings = {
+      ...config.defaultSettings,
+      ...config.metadata.defaultSettings,
+      ...jobSettings[''],
+    }
+
+    const snapshotsPerSchedule = groupBy(this._jobSnapshots, _ => _.other_config['xo:backup:schedule'])
     const xapi = this._xapi
-    await asyncMap(getOldEntries(this._settings.snapshotRetention, scheduleSnapshots), ({ $ref }) => {
-      if ($ref !== baseVmRef) {
-        return xapi.VM_destroy($ref)
+    await asyncMap(Object.entries(snapshotsPerSchedule), ([scheduleId, snapshots]) => {
+      const settings = {
+        ...baseSettings,
+        ...jobSettings[scheduleId],
+        ...jobSettings[this.vm.uuid],
       }
+      return asyncMap(getOldEntries(settings.snapshotRetention, snapshots), ({ $ref }) => {
+        if ($ref !== baseVmRef) {
+          return xapi.VM_destroy($ref)
+        }
+      })
     })
   }
 
@@ -295,12 +320,14 @@ exports.VmBackup = class VmBackup {
 
     let baseVm = findLast(this._jobSnapshots, _ => 'xo:backup:exported' in _.other_config)
     if (baseVm === undefined) {
+      debug('no base VM found')
       return
     }
 
     const fullInterval = this._settings.fullInterval
     const deltaChainLength = +(baseVm.other_config['xo:backup:deltaChainLength'] ?? 0) + 1
     if (!(fullInterval === 0 || fullInterval > deltaChainLength)) {
+      debug('not using base VM becaust fullInterval reached')
       return
     }
 
@@ -311,10 +338,17 @@ exports.VmBackup = class VmBackup {
 
     const baseUuidToSrcVdi = new Map()
     await asyncMap(await baseVm.$getDisks(), async baseRef => {
-      const snapshotOf = await xapi.getField('VDI', baseRef, 'snapshot_of')
+      const [baseUuid, snapshotOf] = await Promise.all([
+        xapi.getField('VDI', baseRef, 'uuid'),
+        xapi.getField('VDI', baseRef, 'snapshot_of'),
+      ])
       const srcVdi = srcVdis[snapshotOf]
       if (srcVdi !== undefined) {
-        baseUuidToSrcVdi.set(await xapi.getField('VDI', baseRef, 'uuid'), srcVdi)
+        baseUuidToSrcVdi.set(baseUuid, srcVdi)
+      } else {
+        debug('ignore snapshot VDI because no longer present on VM', {
+          vdi: baseUuid,
+        })
       }
     })
 
@@ -325,31 +359,33 @@ exports.VmBackup = class VmBackup {
       false
     )
 
+    if (presentBaseVdis.size === 0) {
+      debug('no base VM found')
+      return
+    }
+
     const fullVdisRequired = new Set()
     baseUuidToSrcVdi.forEach((srcVdi, baseUuid) => {
-      if (!presentBaseVdis.has(baseUuid)) {
+      if (presentBaseVdis.has(baseUuid)) {
+        debug('found base VDI', {
+          base: baseUuid,
+          vdi: srcVdi.uuid,
+        })
+      } else {
+        debug('missing base VDI', {
+          base: baseUuid,
+          vdi: srcVdi.uuid,
+        })
         fullVdisRequired.add(srcVdi.uuid)
       }
     })
 
     this._baseVm = baseVm
     this._fullVdisRequired = fullVdisRequired
-
-    Task.info('base data', {
-      vm: baseVm.uuid,
-      fullVdisRequired: Array.from(fullVdisRequired),
-    })
   }
 
   run = defer(this.run)
   async run($defer) {
-    this.exportedVm = this.vm
-    this.timestamp = Date.now()
-
-    const doSnapshot = this._isDelta || vm.power_state === 'Running' || settings.snapshotRetention !== 0
-    if (!this._isDelta) {
-    }
-
     const settings = this._settings
     assert(
       !settings.offlineBackup || settings.snapshotRetention === 0,
@@ -396,6 +432,3 @@ exports.VmBackup = class VmBackup {
     }
   }
 }
-
-// const { prototype } = exports.VmBackup
-// prototype.run = asyncFn.cancelable(prototype.run)

@xen-orchestra/backups/_backupWorker.js

@@ -70,6 +70,7 @@ class BackupWorker {
       yield new RemoteAdapter(handler, {
         debounceResource: this.debounceResource,
         dirMode: this.#config.dirMode,
+        vhdDirectoryCompression: this.#config.vhdDirectoryCompression,
       })
     } finally {
       await handler.forget()

@xen-orchestra/backups/_cleanVm.integ.spec.js

@@ -0,0 +1,407 @@
+/* eslint-env jest */
+
+const rimraf = require('rimraf')
+const tmp = require('tmp')
+const fs = require('fs-extra')
+const { getHandler } = require('@xen-orchestra/fs')
+const { pFromCallback } = require('promise-toolbox')
+const crypto = require('crypto')
+const { RemoteAdapter } = require('./RemoteAdapter')
+const { VHDFOOTER, VHDHEADER } = require('./tests.fixtures.js')
+const { VhdFile, Constants, VhdDirectory, VhdAbstract } = require('vhd-lib')
+
+let tempDir, adapter, handler, jobId, vdiId, basePath
+
+jest.setTimeout(60000)
+
+beforeEach(async () => {
+  tempDir = await pFromCallback(cb => tmp.dir(cb))
+  handler = getHandler({ url: `file://${tempDir}` })
+  await handler.sync()
+  adapter = new RemoteAdapter(handler)
+  jobId = uniqueId()
+  vdiId = uniqueId()
+  basePath = `vdis/${jobId}/${vdiId}`
+  await fs.mkdirp(`${tempDir}/${basePath}`)
+})
+
+afterEach(async () => {
+  await pFromCallback(cb => rimraf(tempDir, cb))
+  await handler.forget()
+})
+
+const uniqueId = () => crypto.randomBytes(16).toString('hex')
+
+async function generateVhd(path, opts = {}) {
+  let vhd
+
+  const dataPath = opts.useAlias ? path + '.data' : path
+  if (opts.mode === 'directory') {
+    await handler.mkdir(dataPath)
+    vhd = new VhdDirectory(handler, dataPath)
+  } else {
+    const fd = await handler.openFile(dataPath, 'wx')
+    vhd = new VhdFile(handler, fd)
+  }
+
+  vhd.header = { ...VHDHEADER, ...opts.header }
+  vhd.footer = { ...VHDFOOTER, ...opts.footer }
+  vhd.footer.uuid = Buffer.from(crypto.randomBytes(16))
+
+  if (vhd.header.parentUnicodeName) {
+    vhd.footer.diskType = Constants.DISK_TYPES.DIFFERENCING
+  } else {
+    vhd.footer.diskType = Constants.DISK_TYPES.DYNAMIC
+  }
+
+  if (opts.useAlias === true) {
+    await VhdAbstract.createAlias(handler, path + '.alias.vhd', dataPath)
+  }
+
+  await vhd.writeBlockAllocationTable()
+  await vhd.writeHeader()
+  await vhd.writeFooter()
+  return vhd
+}
+
+test('It remove broken vhd', async () => {
+  // todo also tests a directory and an alias
+
+  await handler.writeFile(`${basePath}/notReallyAVhd.vhd`, 'I AM NOT A VHD')
+  expect((await handler.list(basePath)).length).toEqual(1)
+  let loggued = ''
+  const onLog = message => {
+    loggued += message
+  }
+  await adapter.cleanVm('/', { remove: false, onLog })
+  expect(loggued).toEqual(`error while checking the VHD with path /${basePath}/notReallyAVhd.vhd`)
+  // not removed
+  expect((await handler.list(basePath)).length).toEqual(1)
+  // really remove it
+  await adapter.cleanVm('/', { remove: true, onLog })
+  expect((await handler.list(basePath)).length).toEqual(0)
+})
+
+test('it remove vhd with missing or multiple ancestors', async () => {
+  // one with a broken parent
+  await generateVhd(`${basePath}/abandonned.vhd`, {
+    header: {
+      parentUnicodeName: 'gone.vhd',
+      parentUid: Buffer.from(crypto.randomBytes(16)),
+    },
+  })
+
+  // one orphan, which is a full vhd, no parent
+  const orphan = await generateVhd(`${basePath}/orphan.vhd`)
+  // a child to the orphan
+  await generateVhd(`${basePath}/child.vhd`, {
+    header: {
+      parentUnicodeName: 'orphan.vhd',
+      parentUid: orphan.footer.uuid,
+    },
+  })
+
+  // clean
+  let loggued = ''
+  const onLog = message => {
+    loggued += message + '\n'
+  }
+  await adapter.cleanVm('/', { remove: true, onLog })
+
+  const deletedOrphanVhd = loggued.match(/deleting orphan VHD/g) || []
+  expect(deletedOrphanVhd.length).toEqual(1) // only one vhd should have been deleted
+  const deletedAbandonnedVhd = loggued.match(/abandonned.vhd is missing/g) || []
+  expect(deletedAbandonnedVhd.length).toEqual(1) // and it must be abandonned.vhd
+
+  // we don't test the filew on disk, since they will all be marker as unused and deleted without a metadata.json file
+})
+
+test('it remove backup meta data referencing a missing vhd in delta backup', async () => {
+  // create a metadata file marking child and orphan as ok
+  await handler.writeFile(
+    `metadata.json`,
+    JSON.stringify({
+      mode: 'delta',
+      vhds: [
+        `${basePath}/orphan.vhd`,
+        `${basePath}/child.vhd`,
+        // abandonned.json is not here
+      ],
+    })
+  )
+
+  await generateVhd(`${basePath}/abandonned.vhd`)
+
+  // one orphan, which is a full vhd, no parent
+  const orphan = await generateVhd(`${basePath}/orphan.vhd`)
+
+  // a child to the orphan
+  await generateVhd(`${basePath}/child.vhd`, {
+    header: {
+      parentUnicodeName: 'orphan.vhd',
+      parentUid: orphan.footer.uuid,
+    },
+  })
+
+  let loggued = ''
+  const onLog = message => {
+    loggued += message + '\n'
+  }
+  await adapter.cleanVm('/', { remove: true, onLog })
+  let matched = loggued.match(/deleting unused VHD /g) || []
+  expect(matched.length).toEqual(1) // only one vhd should have been deleted
+  matched = loggued.match(/abandonned.vhd is unused/g) || []
+  expect(matched.length).toEqual(1) // and it must be abandonned.vhd
+
+  // a missing vhd cause clean to remove all vhds
+  await handler.writeFile(
+    `metadata.json`,
+    JSON.stringify({
+      mode: 'delta',
+      vhds: [
+        `${basePath}/deleted.vhd`, // in metadata but not in vhds
+        `${basePath}/orphan.vhd`,
+        `${basePath}/child.vhd`,
+        // abandonned.json is not here
+      ],
+    }),
+    { flags: 'w' }
+  )
+  loggued = ''
+  await adapter.cleanVm('/', { remove: true, onLog })
+  matched = loggued.match(/deleting unused VHD /g) || []
+  expect(matched.length).toEqual(2) // all vhds (orphan and  child  ) should have been deleted
+})
+
+test('it merges delta of non destroyed chain', async () => {
+  await handler.writeFile(
+    `metadata.json`,
+    JSON.stringify({
+      mode: 'delta',
+      size: 12000, // a size too small
+      vhds: [
+        `${basePath}/grandchild.vhd`, // grand child should not be merged
+        `${basePath}/child.vhd`,
+        // orphan is not here, he should be merged in child
+      ],
+    })
+  )
+
+  // one orphan, which is a full vhd, no parent
+  const orphan = await generateVhd(`${basePath}/orphan.vhd`)
+  // a child to the orphan
+  const child = await generateVhd(`${basePath}/child.vhd`, {
+    header: {
+      parentUnicodeName: 'orphan.vhd',
+      parentUid: orphan.footer.uuid,
+    },
+  })
+  // a grand child
+  await generateVhd(`${basePath}/grandchild.vhd`, {
+    header: {
+      parentUnicodeName: 'child.vhd',
+      parentUid: child.footer.uuid,
+    },
+  })
+
+  let loggued = []
+  const onLog = message => {
+    loggued.push(message)
+  }
+  await adapter.cleanVm('/', { remove: true, onLog })
+  expect(loggued[0]).toEqual(`the parent /${basePath}/orphan.vhd of the child /${basePath}/child.vhd is unused`)
+  expect(loggued[1]).toEqual(`incorrect size in metadata: 12000 instead of 209920`)
+
+  loggued = []
+  await adapter.cleanVm('/', { remove: true, merge: true, onLog })
+  const [unused, merging] = loggued
+  expect(unused).toEqual(`the parent /${basePath}/orphan.vhd of the child /${basePath}/child.vhd is unused`)
+  expect(merging).toEqual(`merging /${basePath}/child.vhd into /${basePath}/orphan.vhd`)
+
+  const metadata = JSON.parse(await handler.readFile(`metadata.json`))
+  // size should be the size of children + grand children after the merge
+  expect(metadata.size).toEqual(209920)
+
+  // merging is already tested in vhd-lib, don't retest it here (and theses vhd are as empty as my stomach at 12h12)
+  // only check deletion
+  const remainingVhds = await handler.list(basePath)
+  expect(remainingVhds.length).toEqual(2)
+  expect(remainingVhds.includes('child.vhd')).toEqual(true)
+  expect(remainingVhds.includes('grandchild.vhd')).toEqual(true)
+})
+
+test('it finish unterminated merge ', async () => {
+  await handler.writeFile(
+    `metadata.json`,
+    JSON.stringify({
+      mode: 'delta',
+      size: undefined,
+      vhds: [
+        `${basePath}/orphan.vhd`, // grand child should not be merged
+        `${basePath}/child.vhd`,
+        // orphan is not here, he should be merged in child
+      ],
+    })
+  )
+
+  // one orphan, which is a full vhd, no parent
+  const orphan = await generateVhd(`${basePath}/orphan.vhd`)
+  // a child to the orphan
+  const child = await generateVhd(`${basePath}/child.vhd`, {
+    header: {
+      parentUnicodeName: 'orphan.vhd',
+      parentUid: orphan.footer.uuid,
+    },
+  })
+  // a merge in progress file
+  await handler.writeFile(
+    `${basePath}/.orphan.vhd.merge.json`,
+    JSON.stringify({
+      parent: {
+        header: orphan.header.checksum,
+      },
+      child: {
+        header: child.header.checksum,
+      },
+    })
+  )
+
+  // a unfinished merging
+  await adapter.cleanVm('/', { remove: true, merge: true })
+  // merging is already tested in vhd-lib, don't retest it here (and theses vhd are as empty as my stomach at 12h12)
+
+  // only check deletion
+  const remainingVhds = await handler.list(basePath)
+  expect(remainingVhds.length).toEqual(1)
+  expect(remainingVhds.includes('child.vhd')).toEqual(true)
+})
+
+// each of the vhd can be a file, a directory, an alias to a file or an alias to a directory
+// the message an resulting files should be identical to the output with vhd files which is tested independantly
+
+describe('tests mulitple combination ', () => {
+  for (const useAlias of [true, false]) {
+    for (const vhdMode of ['file', 'directory']) {
+      test(`alias : ${useAlias}, mode: ${vhdMode}`, async () => {
+        // a broken VHD
+        const brokenVhdDataPath = basePath + useAlias ? 'broken.data' : 'broken.vhd'
+        if (vhdMode === 'directory') {
+          await handler.mkdir(brokenVhdDataPath)
+        } else {
+          await handler.writeFile(brokenVhdDataPath, 'notreallyavhd')
+        }
+        if (useAlias) {
+          await VhdAbstract.createAlias(handler, 'broken.alias.vhd', brokenVhdDataPath)
+        }
+
+        // a vhd non referenced in metada
+        await generateVhd(`${basePath}/nonreference.vhd`, { useAlias, mode: vhdMode })
+        // an abandonded delta vhd without its parent
+        await generateVhd(`${basePath}/abandonned.vhd`, {
+          useAlias,
+          mode: vhdMode,
+          header: {
+            parentUnicodeName: 'gone.vhd',
+            parentUid: crypto.randomBytes(16),
+          },
+        })
+        // an ancestor of a vhd present in metadata
+        const ancestor = await generateVhd(`${basePath}/ancestor.vhd`, {
+          useAlias,
+          mode: vhdMode,
+        })
+        const child = await generateVhd(`${basePath}/child.vhd`, {
+          useAlias,
+          mode: vhdMode,
+          header: {
+            parentUnicodeName: 'ancestor.vhd' + (useAlias ? '.alias.vhd' : ''),
+            parentUid: ancestor.footer.uuid,
+          },
+        })
+        // a grand child  vhd in metadata
+        await generateVhd(`${basePath}/grandchild.vhd`, {
+          useAlias,
+          mode: vhdMode,
+          header: {
+            parentUnicodeName: 'child.vhd' + (useAlias ? '.alias.vhd' : ''),
+            parentUid: child.footer.uuid,
+          },
+        })
+
+        // an older parent that was merging in clean
+        const cleanAncestor = await generateVhd(`${basePath}/cleanAncestor.vhd`, {
+          useAlias,
+          mode: vhdMode,
+        })
+        // a clean  vhd in metadata
+        const clean = await generateVhd(`${basePath}/clean.vhd`, {
+          useAlias,
+          mode: vhdMode,
+          header: {
+            parentUnicodeName: 'cleanAncestor.vhd' + (useAlias ? '.alias.vhd' : ''),
+            parentUid: cleanAncestor.footer.uuid,
+          },
+        })
+
+        await handler.writeFile(
+          `${basePath}/.cleanAncestor.vhd${useAlias ? '.alias.vhd' : ''}.merge.json`,
+          JSON.stringify({
+            parent: {
+              header: cleanAncestor.header.checksum,
+            },
+            child: {
+              header: clean.header.checksum,
+            },
+          })
+        )
+
+        // the metadata file
+        await handler.writeFile(
+          `metadata.json`,
+          JSON.stringify({
+            mode: 'delta',
+            vhds: [
+              `${basePath}/grandchild.vhd` + (useAlias ? '.alias.vhd' : ''), // grand child should not be merged
+              `${basePath}/child.vhd` + (useAlias ? '.alias.vhd' : ''),
+              `${basePath}/clean.vhd` + (useAlias ? '.alias.vhd' : ''),
+            ],
+          })
+        )
+        await adapter.cleanVm('/', { remove: true, merge: true })
+
+        const metadata = JSON.parse(await handler.readFile(`metadata.json`))
+        // size should be the size of children + grand children + clean after the merge
+        expect(metadata.size).toEqual(vhdMode === 'file' ? 314880 : undefined)
+
+        // broken vhd, non referenced, abandonned should be deleted ( alias and data)
+        // ancestor and child should be merged
+        // grand child and clean vhd should not have changed
+        const survivors = await handler.list(basePath)
+        // console.log(survivors)
+        if (useAlias) {
+          // the goal of the alias : do not move a full folder
+          expect(survivors).toContain('ancestor.vhd.data')
+          expect(survivors).toContain('grandchild.vhd.data')
+          expect(survivors).toContain('cleanAncestor.vhd.data')
+          expect(survivors).toContain('clean.vhd.alias.vhd')
+          expect(survivors).toContain('child.vhd.alias.vhd')
+          expect(survivors).toContain('grandchild.vhd.alias.vhd')
+          expect(survivors.length).toEqual(6)
+        } else {
+          expect(survivors).toContain('clean.vhd')
+          expect(survivors).toContain('child.vhd')
+          expect(survivors).toContain('grandchild.vhd')
+          expect(survivors.length).toEqual(3)
+        }
+      })
+    }
+  }
+})
+
+test('it cleans orphan merge states ', async () => {
+  await handler.writeFile(`${basePath}/.orphan.vhd.merge.json`, '')
+
+  await adapter.cleanVm('/', { remove: true })
+
+  expect(await handler.list(basePath)).toEqual([])
+})

@xen-orchestra/backups/_cleanVm.js

@@ -1,16 +1,38 @@
 const assert = require('assert')
+const sum = require('lodash/sum')
 const { asyncMap } = require('@xen-orchestra/async-map')
-const { default: Vhd, mergeVhd } = require('vhd-lib')
+const { Constants, mergeVhd, openVhd, VhdAbstract, VhdFile } = require('vhd-lib')
 const { dirname, resolve } = require('path')
-const { DISK_TYPE_DIFFERENCING } = require('vhd-lib/dist/_constants.js')
+const { DISK_TYPES } = Constants
 const { isMetadataFile, isVhdFile, isXvaFile, isXvaSumFile } = require('./_backupType.js')
 const { limitConcurrency } = require('limit-concurrency-decorator')
 
+const { Task } = require('./Task.js')
+const { Disposable } = require('promise-toolbox')
+
+// checking the size of a vhd directory is costly
+// 1 Http Query per 1000 blocks
+// we only check size of all the vhd are VhdFiles
+function shouldComputeVhdsSize(vhds) {
+  return vhds.every(vhd => vhd instanceof VhdFile)
+}
+
+const computeVhdsSize = (handler, vhdPaths) =>
+  Disposable.use(
+    vhdPaths.map(vhdPath => openVhd(handler, vhdPath)),
+    async vhds => {
+      if (shouldComputeVhdsSize(vhds)) {
+        const sizes = await asyncMap(vhds, vhd => vhd.getSize())
+        return sum(sizes)
+      }
+    }
+  )
+
 // chain is an array of VHDs from child to parent
 //
 // the whole chain will be merged into parent, parent will be renamed to child
 // and all the others will deleted
-const mergeVhdChain = limitConcurrency(1)(async function mergeVhdChain(chain, { handler, onLog, remove, merge }) {
+async function mergeVhdChain(chain, { handler, onLog, remove, merge }) {
   assert(chain.length >= 2)
 
   let child = chain[0]
@@ -43,7 +65,7 @@ const mergeVhdChain = limitConcurrency(1)(async function mergeVhdChain(chain, {
       }
     }, 10e3)
 
-    await mergeVhd(
+    const mergedSize = await mergeVhd(
       handler,
       parent,
       handler,
@@ -62,24 +84,26 @@ const mergeVhdChain = limitConcurrency(1)(async function mergeVhdChain(chain, {
     clearInterval(handle)
 
     await Promise.all([
-      handler.rename(parent, child),
+      VhdAbstract.rename(handler, parent, child),
       asyncMap(children.slice(0, -1), child => {
         onLog(`the VHD ${child} is unused`)
         if (remove) {
           onLog(`deleting unused VHD ${child}`)
-          return handler.unlink(child)
+          return VhdAbstract.unlink(handler, child)
         }
       }),
     ])
+
+    return mergedSize
   }
-})
+}
 
 const noop = Function.prototype
 
-const INTERRUPTED_VHDS_REG = /^(?:(.+)\/)?\.(.+)\.merge.json$/
+const INTERRUPTED_VHDS_REG = /^\.(.+)\.merge.json$/
 const listVhds = async (handler, vmDir) => {
-  const vhds = []
-  const interruptedVhds = new Set()
+  const vhds = new Set()
+  const interruptedVhds = new Map()
 
   await asyncMap(
     await handler.list(`${vmDir}/vdis`, {
@@ -94,16 +118,14 @@ const listVhds = async (handler, vmDir) => {
         async vdiDir => {
           const list = await handler.list(vdiDir, {
             filter: file => isVhdFile(file) || INTERRUPTED_VHDS_REG.test(file),
-            prependDir: true,
           })
 
           list.forEach(file => {
             const res = INTERRUPTED_VHDS_REG.exec(file)
             if (res === null) {
-              vhds.push(file)
+              vhds.add(`${vdiDir}/${file}`)
             } else {
-              const [, dir, file] = res
-              interruptedVhds.add(`${dir}/${file}`)
+              interruptedVhds.set(`${vdiDir}/${res[1]}`, `${vdiDir}/${file}`)
             }
           })
         }
@@ -113,65 +135,91 @@ const listVhds = async (handler, vmDir) => {
   return { vhds, interruptedVhds }
 }
 
-exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop }) {
+const defaultMergeLimiter = limitConcurrency(1)
+
+exports.cleanVm = async function cleanVm(
+  vmDir,
+  { fixMetadata, remove, merge, mergeLimiter = defaultMergeLimiter, onLog = noop }
+) {
+  const limitedMergeVhdChain = mergeLimiter(mergeVhdChain)
+
   const handler = this._handler
 
-  const vhds = new Set()
+  const vhdsToJSons = new Set()
   const vhdParents = { __proto__: null }
   const vhdChildren = { __proto__: null }
 
-  const vhdsList = await listVhds(handler, vmDir)
+  const { vhds, interruptedVhds } = await listVhds(handler, vmDir)
 
   // remove broken VHDs
-  await asyncMap(vhdsList.vhds, async path => {
+  await asyncMap(vhds, async path => {
     try {
-      const vhd = new Vhd(handler, path)
-      await vhd.readHeaderAndFooter(!vhdsList.interruptedVhds.has(path))
-      vhds.add(path)
-      if (vhd.footer.diskType === DISK_TYPE_DIFFERENCING) {
-        const parent = resolve('/', dirname(path), vhd.header.parentUnicodeName)
-        vhdParents[path] = parent
-        if (parent in vhdChildren) {
-          const error = new Error('this script does not support multiple VHD children')
-          error.parent = parent
-          error.child1 = vhdChildren[parent]
-          error.child2 = path
-          throw error // should we throw?
+      await Disposable.use(openVhd(handler, path, { checkSecondFooter: !interruptedVhds.has(path) }), vhd => {
+        if (vhd.footer.diskType === DISK_TYPES.DIFFERENCING) {
+          const parent = resolve('/', dirname(path), vhd.header.parentUnicodeName)
+          vhdParents[path] = parent
+          if (parent in vhdChildren) {
+            const error = new Error('this script does not support multiple VHD children')
+            error.parent = parent
+            error.child1 = vhdChildren[parent]
+            error.child2 = path
+            throw error // should we throw?
+          }
+          vhdChildren[parent] = path
         }
-        vhdChildren[parent] = path
-      }
+      })
     } catch (error) {
+      vhds.delete(path)
       onLog(`error while checking the VHD with path ${path}`, { error })
       if (error?.code === 'ERR_ASSERTION' && remove) {
         onLog(`deleting broken ${path}`)
-        await handler.unlink(path)
+        return VhdAbstract.unlink(handler, path)
       }
     }
   })
 
+  // remove interrupted merge states for missing VHDs
+  for (const interruptedVhd of interruptedVhds.keys()) {
+    if (!vhds.has(interruptedVhd)) {
+      const statePath = interruptedVhds.get(interruptedVhd)
+      interruptedVhds.delete(interruptedVhd)
+
+      onLog('orphan merge state', {
+        mergeStatePath: statePath,
+        missingVhdPath: interruptedVhd,
+      })
+      if (remove) {
+        onLog(`deleting orphan merge state ${statePath}`)
+        await handler.unlink(statePath)
+      }
+    }
+  }
+
+  // @todo : add check for data folder of alias not referenced in a valid alias
+
   // remove VHDs with missing ancestors
   {
     const deletions = []
 
     // return true if the VHD has been deleted or is missing
-    const deleteIfOrphan = vhd => {
-      const parent = vhdParents[vhd]
+    const deleteIfOrphan = vhdPath => {
+      const parent = vhdParents[vhdPath]
       if (parent === undefined) {
         return
       }
 
       // no longer needs to be checked
-      delete vhdParents[vhd]
+      delete vhdParents[vhdPath]
 
       deleteIfOrphan(parent)
 
       if (!vhds.has(parent)) {
-        vhds.delete(vhd)
+        vhds.delete(vhdPath)
 
-        onLog(`the parent ${parent} of the VHD ${vhd} is missing`)
+        onLog(`the parent ${parent} of the VHD ${vhdPath} is missing`)
         if (remove) {
-          onLog(`deleting orphan VHD ${vhd}`)
-          deletions.push(handler.unlink(vhd))
+          onLog(`deleting orphan VHD ${vhdPath}`)
+          deletions.push(VhdAbstract.unlink(handler, vhdPath))
         }
       }
     }
@@ -187,7 +235,7 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
     await Promise.all(deletions)
   }
 
-  const jsons = []
+  const jsons = new Set()
   const xvas = new Set()
   const xvaSums = []
   const entries = await handler.list(vmDir, {
@@ -195,7 +243,7 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
   })
   entries.forEach(path => {
     if (isMetadataFile(path)) {
-      jsons.push(path)
+      jsons.add(path)
     } else if (isXvaFile(path)) {
       xvas.add(path)
     } else if (isXvaSumFile(path)) {
@@ -217,17 +265,25 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
   // compile the list of unused XVAs and VHDs, and remove backup metadata which
   // reference a missing XVA/VHD
   await asyncMap(jsons, async json => {
-    const metadata = JSON.parse(await handler.readFile(json))
+    let metadata
+    try {
+      metadata = JSON.parse(await handler.readFile(json))
+    } catch (error) {
+      onLog(`failed to read metadata file ${json}`, { error })
+      jsons.delete(json)
+      return
+    }
+
     const { mode } = metadata
     if (mode === 'full') {
       const linkedXva = resolve('/', vmDir, metadata.xva)
-
       if (xvas.has(linkedXva)) {
         unusedXvas.delete(linkedXva)
       } else {
         onLog(`the XVA linked to the metadata ${json} is missing`)
         if (remove) {
           onLog(`deleting incomplete backup ${json}`)
+          jsons.delete(json)
           await handler.unlink(json)
         }
       }
@@ -237,14 +293,20 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
         return Object.keys(vhds).map(key => resolve('/', vmDir, vhds[key]))
       })()
 
+      const missingVhds = linkedVhds.filter(_ => !vhds.has(_))
+
       // FIXME: find better approach by keeping as much of the backup as
       // possible (existing disks) even if one disk is missing
-      if (linkedVhds.every(_ => vhds.has(_))) {
+      if (missingVhds.length === 0) {
         linkedVhds.forEach(_ => unusedVhds.delete(_))
+        linkedVhds.forEach(path => {
+          vhdsToJSons[path] = json
+        })
       } else {
-        onLog(`Some VHDs linked to the metadata ${json} are missing`)
+        onLog(`Some VHDs linked to the metadata ${json} are missing`, { missingVhds })
         if (remove) {
           onLog(`deleting incomplete backup ${json}`)
+          jsons.delete(json)
           await handler.unlink(json)
         }
       }
@@ -253,6 +315,7 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
 
   // TODO: parallelize by vm/job/vdi
   const unusedVhdsDeletion = []
+  const toMerge = []
   {
     // VHD chains (as list from child to ancestor) to merge indexed by last
     // ancestor
@@ -286,7 +349,7 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
       onLog(`the VHD ${vhd} is unused`)
       if (remove) {
         onLog(`deleting unused VHD ${vhd}`)
-        unusedVhdsDeletion.push(handler.unlink(vhd))
+        unusedVhdsDeletion.push(VhdAbstract.unlink(handler, vhd))
       }
     }
 
@@ -295,22 +358,31 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
     })
 
     // merge interrupted VHDs
-    if (merge) {
-      vhdsList.interruptedVhds.forEach(parent => {
-        vhdChainsToMerge[parent] = [vhdChildren[parent], parent]
-      })
+    for (const parent of interruptedVhds.keys()) {
+      vhdChainsToMerge[parent] = [vhdChildren[parent], parent]
     }
 
-    Object.keys(vhdChainsToMerge).forEach(key => {
-      const chain = vhdChainsToMerge[key]
+    Object.values(vhdChainsToMerge).forEach(chain => {
       if (chain !== undefined) {
-        unusedVhdsDeletion.push(mergeVhdChain(chain, { handler, onLog, remove, merge }))
+        toMerge.push(chain)
+      }
+    })
+  }
+
+  const metadataWithMergedVhd = {}
+  const doMerge = async () => {
+    await asyncMap(toMerge, async chain => {
+      const merged = await limitedMergeVhdChain(chain, { handler, onLog, remove, merge })
+      if (merged !== undefined) {
+        const metadataPath = vhdsToJSons[chain[0]] // all the chain should have the same metada file
+        metadataWithMergedVhd[metadataPath] = true
       }
     })
   }
 
   await Promise.all([
     ...unusedVhdsDeletion,
+    toMerge.length !== 0 && (merge ? Task.run({ name: 'merge' }, doMerge) : doMerge()),
     asyncMap(unusedXvas, path => {
       onLog(`the XVA ${path} is unused`)
       if (remove) {
@@ -329,4 +401,55 @@ exports.cleanVm = async function cleanVm(vmDir, { remove, merge, onLog = noop })
       }
     }),
   ])
+
+  // update size for delta metadata with merged VHD
+  // check for the other that the size is the same as the real file size
+
+  await asyncMap(jsons, async metadataPath => {
+    const metadata = JSON.parse(await handler.readFile(metadataPath))
+
+    let fileSystemSize
+    const merged = metadataWithMergedVhd[metadataPath] !== undefined
+
+    const { mode, size, vhds, xva } = metadata
+
+    try {
+      if (mode === 'full') {
+        // a full backup : check size
+        const linkedXva = resolve('/', vmDir, xva)
+        fileSystemSize = await handler.getSize(linkedXva)
+      } else if (mode === 'delta') {
+        const linkedVhds = Object.keys(vhds).map(key => resolve('/', vmDir, vhds[key]))
+        fileSystemSize = await computeVhdsSize(handler, linkedVhds)
+
+        // the size is not computed in some cases (e.g. VhdDirectory)
+        if (fileSystemSize === undefined) {
+          return
+        }
+
+        // don't warn if the size has changed after a merge
+        if (!merged && fileSystemSize !== size) {
+          onLog(`incorrect size in metadata: ${size ?? 'none'} instead of ${fileSystemSize}`)
+        }
+      }
+    } catch (error) {
+      onLog(`failed to get size of ${metadataPath}`, { error })
+      return
+    }
+
+    // systematically update size after a merge
+    if ((merged || fixMetadata) && size !== fileSystemSize) {
+      metadata.size = fileSystemSize
+      try {
+        await handler.writeFile(metadataPath, JSON.stringify(metadata), { flags: 'w' })
+      } catch (error) {
+        onLog(`failed to update size in backup metadata ${metadataPath} after merge`, { error })
+      }
+    }
+  })
+
+  return {
+    // boolean whether some VHDs were merged (or should be merged)
+    merge: toMerge.length !== 0,
+  }
 }

@xen-orchestra/backups/_deltaVm.js

@@ -202,6 +202,7 @@ exports.importDeltaVm = defer(async function importDeltaVm(
       blocked_operations: {
         ...vmRecord.blocked_operations,
         start: 'Importing…',
+        start_on: 'Importing…',
       },
       ha_always_run: false,
       is_a_template: false,
@@ -305,9 +306,6 @@ exports.importDeltaVm = defer(async function importDeltaVm(
       }
     }),
 
-    // Wait for VDI export tasks (if any) termination.
-    Promise.all(Object.values(streams).map(stream => stream.task)),
-
     // Create VIFs.
     asyncMap(Object.values(deltaVm.vifs), vif => {
       let network = vif.$network$uuid && xapi.getObjectByUuid(vif.$network$uuid, undefined)

@xen-orchestra/backups/_isValidXva.js

@@ -1,11 +1,24 @@
 const assert = require('assert')
 
-const isGzipFile = async (handler, fd) => {
+const COMPRESSED_MAGIC_NUMBERS = [
   // https://tools.ietf.org/html/rfc1952.html#page-5
-  const magicNumber = Buffer.allocUnsafe(2)
+  Buffer.from('1F8B', 'hex'),
 
-  assert.strictEqual((await handler.read(fd, magicNumber, 0)).bytesRead, magicNumber.length)
-  return magicNumber[0] === 31 && magicNumber[1] === 139
+  // https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames
+  Buffer.from('28B52FFD', 'hex'),
+]
+const MAGIC_NUMBER_MAX_LENGTH = Math.max(...COMPRESSED_MAGIC_NUMBERS.map(_ => _.length))
+
+const isCompressedFile = async (handler, fd) => {
+  const header = Buffer.allocUnsafe(MAGIC_NUMBER_MAX_LENGTH)
+  assert.strictEqual((await handler.read(fd, header, 0)).bytesRead, header.length)
+
+  for (const magicNumber of COMPRESSED_MAGIC_NUMBERS) {
+    if (magicNumber.compare(header, 0, magicNumber.length) === 0) {
+      return true
+    }
+  }
+  return false
 }
 
 // TODO: better check?
@@ -43,8 +56,8 @@ async function isValidXva(path) {
         return false
       }
 
-      return (await isGzipFile(handler, fd))
-        ? true // gzip files cannot be validated at this time
+      return (await isCompressedFile(handler, fd))
+        ? true // compressed files cannot be validated at this time
         : await isValidTar(handler, size, fd)
     } finally {
       handler.closeFile(fd).catch(noop)

@xen-orchestra/backups/_lvm.js

@@ -7,23 +7,25 @@ const { execFile } = require('child_process')
 const parse = createParser({
   keyTransform: key => key.slice(5).toLowerCase(),
 })
-const makeFunction = command => async (fields, ...args) => {
-  const info = await fromCallback(execFile, command, [
-    '--noheading',
-    '--nosuffix',
-    '--nameprefixes',
-    '--unbuffered',
-    '--units',
-    'b',
-    '-o',
-    String(fields),
-    ...args,
-  ])
-  return info
-    .trim()
-    .split(/\r?\n/)
-    .map(Array.isArray(fields) ? parse : line => parse(line)[fields])
-}
+const makeFunction =
+  command =>
+  async (fields, ...args) => {
+    const info = await fromCallback(execFile, command, [
+      '--noheading',
+      '--nosuffix',
+      '--nameprefixes',
+      '--unbuffered',
+      '--units',
+      'b',
+      '-o',
+      String(fields),
+      ...args,
+    ])
+    return info
+      .trim()
+      .split(/\r?\n/)
+      .map(Array.isArray(fields) ? parse : line => parse(line)[fields])
+  }
 
 exports.lvs = makeFunction('lvs')
 exports.pvs = makeFunction('pvs')

@xen-orchestra/backups/merge-worker/cli.js

@@ -0,0 +1,69 @@
+#!/usr/bin/env node
+
+const { catchGlobalErrors } = require('@xen-orchestra/log/configure.js')
+const { createLogger } = require('@xen-orchestra/log')
+const { getSyncedHandler } = require('@xen-orchestra/fs')
+const { join } = require('path')
+const Disposable = require('promise-toolbox/Disposable')
+const min = require('lodash/min')
+
+const { getVmBackupDir } = require('../_getVmBackupDir.js')
+const { RemoteAdapter } = require('../RemoteAdapter.js')
+
+const { CLEAN_VM_QUEUE } = require('./index.js')
+
+// -------------------------------------------------------------------
+
+catchGlobalErrors(createLogger('xo:backups:mergeWorker'))
+
+const { fatal, info, warn } = createLogger('xo:backups:mergeWorker')
+
+// -------------------------------------------------------------------
+
+const main = Disposable.wrap(async function* main(args) {
+  const handler = yield getSyncedHandler({ url: 'file://' + process.cwd() })
+
+  yield handler.lock(CLEAN_VM_QUEUE)
+
+  const adapter = new RemoteAdapter(handler)
+
+  const listRetry = async () => {
+    const timeoutResolver = resolve => setTimeout(resolve, 10e3)
+    for (let i = 0; i < 10; ++i) {
+      const entries = await handler.list(CLEAN_VM_QUEUE)
+      if (entries.length !== 0) {
+        return entries
+      }
+      await new Promise(timeoutResolver)
+    }
+  }
+
+  let taskFiles
+  while ((taskFiles = await listRetry()) !== undefined) {
+    const taskFileBasename = min(taskFiles)
+    const taskFile = join(CLEAN_VM_QUEUE, '_' + taskFileBasename)
+
+    // move this task to the end
+    await handler.rename(join(CLEAN_VM_QUEUE, taskFileBasename), taskFile)
+    try {
+      const vmDir = getVmBackupDir(String(await handler.readFile(taskFile)))
+      await adapter.cleanVm(vmDir, { merge: true, onLog: info, remove: true })
+
+      handler.unlink(taskFile).catch(error => warn('deleting task failure', { error }))
+    } catch (error) {
+      warn('failure handling task', { error })
+    }
+  }
+})
+
+info('starting')
+main(process.argv.slice(2)).then(
+  () => {
+    info('bye :-)')
+  },
+  error => {
+    fatal(error)
+
+    process.exit(1)
+  }
+)

@xen-orchestra/backups/merge-worker/index.js

@@ -0,0 +1,25 @@
+const { join, resolve } = require('path')
+const { spawn } = require('child_process')
+const { check } = require('proper-lockfile')
+
+const CLEAN_VM_QUEUE = (exports.CLEAN_VM_QUEUE = '/xo-vm-backups/.queue/clean-vm/')
+
+const CLI_PATH = resolve(__dirname, 'cli.js')
+exports.run = async function runMergeWorker(remotePath) {
+  try {
+    // TODO: find a way to pass the acquire the lock and then pass it down the worker
+    if (await check(join(remotePath, CLEAN_VM_QUEUE))) {
+      // already locked, don't start another worker
+      return
+    }
+
+    spawn(CLI_PATH, {
+      cwd: remotePath,
+      detached: true,
+      stdio: 'inherit',
+    }).unref()
+  } catch (error) {
+    // we usually don't want to throw if the merge worker failed to start
+    return error
+  }
+}

@xen-orchestra/backups/package.json

@@ -8,7 +8,7 @@
     "type": "git",
     "url": "https://github.com/vatesfr/xen-orchestra.git"
   },
-  "version": "0.11.0",
+  "version": "0.18.3",
   "engines": {
     "node": ">=14.6"
   },
@@ -16,29 +16,31 @@
     "postversion": "npm publish --access public"
   },
   "dependencies": {
-    "@vates/compose": "^2.0.0",
+    "@vates/compose": "^2.1.0",
     "@vates/disposable": "^0.1.1",
     "@vates/parse-duration": "^0.1.1",
     "@xen-orchestra/async-map": "^0.1.2",
-    "@xen-orchestra/fs": "^0.17.0",
-    "@xen-orchestra/log": "^0.2.0",
+    "@xen-orchestra/fs": "^0.19.3",
+    "@xen-orchestra/log": "^0.3.0",
     "@xen-orchestra/template": "^0.1.0",
-    "compare-versions": "^3.6.0",
+    "compare-versions": "^4.0.1",
     "d3-time-format": "^3.0.0",
     "end-of-stream": "^1.4.4",
-    "fs-extra": "^9.0.0",
+    "fs-extra": "^10.0.0",
     "golike-defer": "^0.5.1",
     "limit-concurrency-decorator": "^0.5.0",
     "lodash": "^4.17.20",
     "node-zone": "^0.4.0",
     "parse-pairs": "^1.1.0",
+    "promise-toolbox": "^0.20.0",
+    "proper-lockfile": "^4.1.2",
     "pump": "^3.0.0",
-    "promise-toolbox": "^0.19.2",
-    "vhd-lib": "^1.0.0",
+    "uuid": "^8.3.2",
+    "vhd-lib": "^3.0.0",
     "yazl": "^2.5.1"
   },
   "peerDependencies": {
-    "@xen-orchestra/xapi": "^0.6.2"
+    "@xen-orchestra/xapi": "^0.8.5"
   },
   "license": "AGPL-3.0-or-later",
   "author": {

@xen-orchestra/backups/tests.fixtures.js

@@ -0,0 +1,92 @@
+// a valid footer of a 2
+exports.VHDFOOTER = {
+  cookie: 'conectix',
+  features: 2,
+  fileFormatVersion: 65536,
+  dataOffset: 512,
+  timestamp: 0,
+  creatorApplication: 'caml',
+  creatorVersion: 1,
+  creatorHostOs: 0,
+  originalSize: 53687091200,
+  currentSize: 53687091200,
+  diskGeometry: { cylinders: 25700, heads: 16, sectorsPerTrackCylinder: 255 },
+  diskType: 3,
+  checksum: 4294962945,
+  uuid: Buffer.from('d8dbcad85265421e8b298d99c2eec551', 'utf-8'),
+  saved: '',
+  hidden: '',
+  reserved: '',
+}
+exports.VHDHEADER = {
+  cookie: 'cxsparse',
+  dataOffset: undefined,
+  tableOffset: 2048,
+  headerVersion: 65536,
+  maxTableEntries: 25600,
+  blockSize: 2097152,
+  checksum: 4294964241,
+  parentUuid: null,
+  parentTimestamp: 0,
+  reserved1: 0,
+  parentUnicodeName: '',
+  parentLocatorEntry: [
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+    {
+      platformCode: 0,
+      platformDataSpace: 0,
+      platformDataLength: 0,
+      reserved: 0,
+      platformDataOffset: 0,
+    },
+  ],
+  reserved2: '',
+}

@xen-orchestra/backups/writers/DeltaBackupWriter.js

@@ -3,7 +3,7 @@ const map = require('lodash/map.js')
 const mapValues = require('lodash/mapValues.js')
 const ignoreErrors = require('promise-toolbox/ignoreErrors.js')
 const { asyncMap } = require('@xen-orchestra/async-map')
-const { chainVhd, checkVhdChain, default: Vhd } = require('vhd-lib')
+const { chainVhd, checkVhdChain, openVhd, VhdAbstract } = require('vhd-lib')
 const { createLogger } = require('@xen-orchestra/log')
 const { dirname } = require('path')
 
@@ -16,6 +16,7 @@ const { MixinBackupWriter } = require('./_MixinBackupWriter.js')
 const { AbstractDeltaWriter } = require('./_AbstractDeltaWriter.js')
 const { checkVhd } = require('./_checkVhd.js')
 const { packUuid } = require('./_packUuid.js')
+const { Disposable } = require('promise-toolbox')
 
 const { warn } = createLogger('xo:backups:DeltaBackupWriter')
 
@@ -23,6 +24,7 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
   async checkBaseVdis(baseUuidToSrcVdi) {
     const { handler } = this._adapter
     const backup = this._backup
+    const adapter = this._adapter
 
     const backupDir = getVmBackupDir(backup.vm.uuid)
     const vdisDir = `${backupDir}/vdis/${backup.job.id}`
@@ -34,16 +36,21 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
           filter: _ => _[0] !== '.' && _.endsWith('.vhd'),
           prependDir: true,
         })
+        const packedBaseUuid = packUuid(baseUuid)
         await asyncMap(vhds, async path => {
           try {
             await checkVhdChain(handler, path)
+            // Warning, this should not be written as found = found || await adapter.isMergeableParent(packedBaseUuid, path)
+            //
+            // since all the checks of a path are done in parallel, found would be containing
+            // only the last answer of isMergeableParent which is probably not the right one
+            // this led to the support tickets  https://help.vates.fr/#ticket/zoom/4751 , 4729, 4665 and 4300
 
-            const vhd = new Vhd(handler, path)
-            await vhd.readHeaderAndFooter()
-            found = found || vhd.footer.uuid.equals(packUuid(baseUuid))
+            const isMergeable = await adapter.isMergeableParent(packedBaseUuid, path)
+            found = found || isMergeable
           } catch (error) {
             warn('checkBaseVdis', { error })
-            await ignoreErrors.call(handler.unlink(path))
+            await ignoreErrors.call(VhdAbstract.unlink(handler, path))
           }
         })
       } catch (error) {
@@ -113,22 +120,16 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
   }
 
   async _deleteOldEntries() {
-    return Task.run({ name: 'merge' }, async () => {
-      const adapter = this._adapter
-      const oldEntries = this._oldEntries
+    const adapter = this._adapter
+    const oldEntries = this._oldEntries
 
-      let size = 0
-      // delete sequentially from newest to oldest to avoid unnecessary merges
-      for (let i = oldEntries.length; i-- > 0; ) {
-        size += await adapter.deleteDeltaVmBackups([oldEntries[i]])
-      }
-      return {
-        size,
-      }
-    })
+    // delete sequentially from newest to oldest to avoid unnecessary merges
+    for (let i = oldEntries.length; i-- > 0; ) {
+      await adapter.deleteDeltaVmBackups([oldEntries[i]])
+    }
   }
 
-  async transfer({ timestamp, deltaExport, sizeContainers }) {
+  async _transfer({ timestamp, deltaExport, sizeContainers }) {
     const adapter = this._adapter
     const backup = this._backup
 
@@ -150,7 +151,7 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
               // don't do delta for it
               vdi.uuid
             : vdi.$snapshot_of$uuid
-        }/${basename}.vhd`
+        }/${adapter.getVhdFileName(basename)}`
     )
 
     const metadataFilename = `${backupDir}/${basename}.json`
@@ -194,7 +195,7 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
             await checkVhd(handler, parentPath)
           }
 
-          await adapter.outputStream(path, deltaExport.streams[`${id}.vhd`], {
+          await adapter.writeVhd(path, deltaExport.streams[`${id}.vhd`], {
             // no checksum for VHDs, because they will be invalidated by
             // merges and chainings
             checksum: false,
@@ -206,11 +207,11 @@ exports.DeltaBackupWriter = class DeltaBackupWriter extends MixinBackupWriter(Ab
           }
 
           // set the correct UUID in the VHD
-          const vhd = new Vhd(handler, path)
-          await vhd.readHeaderAndFooter()
-          vhd.footer.uuid = packUuid(vdi.uuid)
-          await vhd.readBlockAllocationTable() // required by writeFooter()
-          await vhd.writeFooter()
+          await Disposable.use(openVhd(handler, path), async vhd => {
+            vhd.footer.uuid = packUuid(vdi.uuid)
+            await vhd.readBlockAllocationTable() // required by writeFooter()
+            await vhd.writeFooter()
+          })
         })
       )
       return {

@xen-orchestra/backups/writers/DeltaReplicationWriter.js

@@ -77,7 +77,7 @@ exports.DeltaReplicationWriter = class DeltaReplicationWriter extends MixinRepli
     return asyncMapSettled(this._oldEntries, vm => vm.$destroy())
   }
 
-  async transfer({ timestamp, deltaExport, sizeContainers }) {
+  async _transfer({ timestamp, deltaExport, sizeContainers }) {
     const sr = this._sr
     const { job, scheduleId, vm } = this._backup
 
@@ -106,9 +106,11 @@ exports.DeltaReplicationWriter = class DeltaReplicationWriter extends MixinRepli
       targetVm.ha_restart_priority !== '' &&
         Promise.all([targetVm.set_ha_restart_priority(''), targetVm.add_tags('HA disabled')]),
       targetVm.set_name_label(`${vm.name_label} - ${job.name} - (${formatFilenameDate(timestamp)})`),
-      targetVm.update_blocked_operations(
-        'start',
-        'Start operation for this vm is blocked, clone it if you want to use it.'
+      asyncMap(['start', 'start_on'], op =>
+        targetVm.update_blocked_operations(
+          op,
+          'Start operation for this vm is blocked, clone it if you want to use it.'
+        )
       ),
       targetVm.update_other_config({
         'xo:backup:sr': srUuid,

@xen-orchestra/backups/writers/FullBackupWriter.js

@@ -25,7 +25,7 @@ exports.FullBackupWriter = class FullBackupWriter extends MixinBackupWriter(Abst
     )
   }
 
-  async run({ timestamp, sizeContainer, stream }) {
+  async _run({ timestamp, sizeContainer, stream }) {
     const backup = this._backup
     const settings = this._settings
 

@xen-orchestra/backups/writers/FullReplicationWriter.js

@@ -1,5 +1,5 @@
 const ignoreErrors = require('promise-toolbox/ignoreErrors.js')
-const { asyncMapSettled } = require('@xen-orchestra/async-map')
+const { asyncMap, asyncMapSettled } = require('@xen-orchestra/async-map')
 const { formatDateTime } = require('@xen-orchestra/xapi')
 
 const { formatFilenameDate } = require('../_filenameDate.js')
@@ -29,7 +29,7 @@ exports.FullReplicationWriter = class FullReplicationWriter extends MixinReplica
     )
   }
 
-  async run({ timestamp, sizeContainer, stream }) {
+  async _run({ timestamp, sizeContainer, stream }) {
     const sr = this._sr
     const settings = this._settings
     const { job, scheduleId, vm } = this._backup
@@ -64,9 +64,11 @@ exports.FullReplicationWriter = class FullReplicationWriter extends MixinReplica
     const targetVm = await xapi.getRecord('VM', targetVmRef)
 
     await Promise.all([
-      targetVm.update_blocked_operations(
-        'start',
-        'Start operation for this vm is blocked, clone it if you want to use it.'
+      asyncMap(['start', 'start_on'], op =>
+        targetVm.update_blocked_operations(
+          op,
+          'Start operation for this vm is blocked, clone it if you want to use it.'
+        )
       ),
       targetVm.update_other_config({
         'xo:backup:sr': srUuid,

@xen-orchestra/backups/writers/_AbstractDeltaWriter.js

@@ -13,7 +13,14 @@ exports.AbstractDeltaWriter = class AbstractDeltaWriter extends AbstractWriter {
     throw new Error('Not implemented')
   }
 
-  transfer({ timestamp, deltaExport, sizeContainers }) {
-    throw new Error('Not implemented')
+  async transfer({ timestamp, deltaExport, sizeContainers }) {
+    try {
+      return await this._transfer({ timestamp, deltaExport, sizeContainers })
+    } finally {
+      // ensure all streams are properly closed
+      for (const stream of Object.values(deltaExport.streams)) {
+        stream.destroy()
+      }
+    }
   }
 }

@xen-orchestra/backups/writers/_AbstractFullWriter.js

@@ -1,7 +1,12 @@
 const { AbstractWriter } = require('./_AbstractWriter.js')
 
 exports.AbstractFullWriter = class AbstractFullWriter extends AbstractWriter {
-  run({ timestamp, sizeContainer, stream }) {
-    throw new Error('Not implemented')
+  async run({ timestamp, sizeContainer, stream }) {
+    try {
+      return await this._run({ timestamp, sizeContainer, stream })
+    } finally {
+      // ensure stream is properly closed
+      stream.destroy()
+    }
   }
 }

@xen-orchestra/backups/writers/_MixinBackupWriter.js

@@ -1,34 +1,65 @@
 const { createLogger } = require('@xen-orchestra/log')
+const { join } = require('path')
 
-const { getVmBackupDir } = require('../_getVmBackupDir.js')
+const { BACKUP_DIR, getVmBackupDir } = require('../_getVmBackupDir.js')
+const MergeWorker = require('../merge-worker/index.js')
+const { formatFilenameDate } = require('../_filenameDate.js')
 
 const { warn } = createLogger('xo:backups:MixinBackupWriter')
 
 exports.MixinBackupWriter = (BaseClass = Object) =>
   class MixinBackupWriter extends BaseClass {
+    #lock
+    #vmBackupDir
+
     constructor({ remoteId, ...rest }) {
       super(rest)
 
       this._adapter = rest.backup.remoteAdapters[remoteId]
       this._remoteId = remoteId
-      this._lock = undefined
+
+      this.#vmBackupDir = getVmBackupDir(this._backup.vm.uuid)
     }
 
-    _cleanVm(options) {
-      return this._adapter
-        .cleanVm(getVmBackupDir(this._backup.vm.uuid), { ...options, onLog: warn, lock: false })
-        .catch(warn)
+    async _cleanVm(options) {
+      try {
+        return await this._adapter.cleanVm(this.#vmBackupDir, {
+          ...options,
+          fixMetadata: true,
+          onLog: warn,
+          lock: false,
+        })
+      } catch (error) {
+        warn(error)
+        return {}
+      }
     }
 
     async beforeBackup() {
       const { handler } = this._adapter
-      const vmBackupDir = getVmBackupDir(this._backup.vm.uuid)
+      const vmBackupDir = this.#vmBackupDir
       await handler.mktree(vmBackupDir)
-      this._lock = await handler.lock(vmBackupDir)
+      this.#lock = await handler.lock(vmBackupDir)
     }
 
     async afterBackup() {
-      await this._cleanVm({ remove: true, merge: true })
-      await this._lock.dispose()
+      const { disableMergeWorker } = this._backup.config
+
+      const { merge } = await this._cleanVm({ remove: true, merge: disableMergeWorker })
+      await this.#lock.dispose()
+
+      // merge worker only compatible with local remotes
+      const { handler } = this._adapter
+      if (merge && !disableMergeWorker && typeof handler._getRealPath === 'function') {
+        const taskFile =
+          join(MergeWorker.CLEAN_VM_QUEUE, formatFilenameDate(new Date())) +
+          '-' +
+          // add a random suffix to avoid collision in case multiple tasks are created at the same second
+          Math.random().toString(36).slice(2)
+
+        await handler.outputFile(taskFile, this._backup.vm.uuid)
+        const remotePath = handler._getRealPath()
+        await MergeWorker.run(remotePath)
+      }
     }
   }

@xen-orchestra/backups/writers/_checkVhd.js

@@ -1,5 +1,6 @@
-const Vhd = require('vhd-lib').default
+const openVhd = require('vhd-lib').openVhd
+const Disposable = require('promise-toolbox/Disposable')
 
 exports.checkVhd = async function checkVhd(handler, path) {
-  await new Vhd(handler, path).readHeaderAndFooter()
+  await Disposable.use(openVhd(handler, path), () => {})
 }

@xen-orchestra/cr-seed-cli/index.js

@@ -77,7 +77,11 @@ ${cliName} v${pkg.version}
       'xo:backup:sr': tgtSr.uuid,
       'xo:copy_of': srcSnapshotUuid,
     }),
-    tgtVm.update_blocked_operations('start', 'Start operation for this vm is blocked, clone it if you want to use it.'),
+    Promise.all(
+      ['start', 'start_on'].map(op =>
+        tgtVm.update_blocked_operations(op, 'Start operation for this vm is blocked, clone it if you want to use it.')
+      )
+    ),
     Promise.all(
       userDevices.map(userDevice => {
         const srcDisk = srcDisks[userDevice]

@xen-orchestra/cr-seed-cli/package.json

@@ -18,7 +18,7 @@
   "preferGlobal": true,
   "dependencies": {
     "golike-defer": "^0.5.1",
-    "xen-api": "^0.32.0"
+    "xen-api": "^0.35.1"
   },
   "scripts": {
     "postversion": "npm publish"

@xen-orchestra/cron/src/index.spec.js

@@ -2,6 +2,8 @@
 
 import { createSchedule } from './'
 
+jest.useFakeTimers()
+
 const wrap = value => () => value
 
 describe('issues', () => {

@xen-orchestra/defined/package.json

@@ -1,7 +1,7 @@
 {
   "private": false,
   "name": "@xen-orchestra/defined",
-  "version": "0.0.0",
+  "version": "0.0.1",
   "license": "ISC",
   "description": "Utilities to help handling (possibly) undefined values",
   "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/defined",

@xen-orchestra/emit-async/package.json

@@ -1,7 +1,7 @@
 {
   "private": false,
   "name": "@xen-orchestra/emit-async",
-  "version": "0.0.0",
+  "version": "0.1.0",
   "license": "ISC",
   "description": "Emit an event for async listeners to settle",
   "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/emit-async",

@xen-orchestra/fs/package.json

@@ -1,7 +1,7 @@
 {
   "private": false,
   "name": "@xen-orchestra/fs",
-  "version": "0.17.0",
+  "version": "0.19.3",
   "license": "AGPL-3.0-or-later",
   "description": "The File System for Xen Orchestra backups.",
   "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/fs",
@@ -17,23 +17,23 @@
     "node": ">=14"
   },
   "dependencies": {
-    "@marsaud/smb2": "^0.17.2",
+    "@marsaud/smb2": "^0.18.0",
     "@sindresorhus/df": "^3.1.1",
     "@sullux/aws-sdk": "^1.0.5",
     "@vates/coalesce-calls": "^0.1.0",
     "@xen-orchestra/async-map": "^0.1.2",
     "aws-sdk": "^2.686.0",
-    "decorator-synchronized": "^0.5.0",
+    "decorator-synchronized": "^0.6.0",
     "execa": "^5.0.0",
-    "fs-extra": "^9.0.0",
+    "fs-extra": "^10.0.0",
     "get-stream": "^6.0.0",
     "limit-concurrency-decorator": "^0.5.0",
     "lodash": "^4.17.4",
-    "promise-toolbox": "^0.19.2",
+    "promise-toolbox": "^0.20.0",
     "proper-lockfile": "^4.1.2",
     "readable-stream": "^3.0.6",
     "through2": "^4.0.2",
-    "xo-remote-parser": "^0.7.0"
+    "xo-remote-parser": "^0.8.0"
   },
   "devDependencies": {
     "@babel/cli": "^7.0.0",
@@ -45,7 +45,7 @@
     "async-iterator-to-stream": "^1.1.0",
     "babel-plugin-lodash": "^3.3.2",
     "cross-env": "^7.0.2",
-    "dotenv": "^8.0.0",
+    "dotenv": "^10.0.0",
     "rimraf": "^3.0.0"
   },
   "scripts": {

@xen-orchestra/fs/src/abstract.js

@@ -1,13 +1,13 @@
 import asyncMapSettled from '@xen-orchestra/async-map/legacy'
 import getStream from 'get-stream'
 import path, { basename } from 'path'
-import synchronized from 'decorator-synchronized'
 import { coalesceCalls } from '@vates/coalesce-calls'
 import { fromCallback, fromEvent, ignoreErrors, timeout } from 'promise-toolbox'
 import { limitConcurrency } from 'limit-concurrency-decorator'
 import { parse } from 'xo-remote-parser'
 import { pipeline } from 'stream'
 import { randomBytes } from 'crypto'
+import { synchronized } from 'decorator-synchronized'
 
 import normalizePath from './_normalizePath'
 import { createChecksumStream, validChecksumOfReadStream } from './checksum'
@@ -76,6 +76,7 @@ export default class RemoteHandlerAbstract {
 
     const sharedLimit = limitConcurrency(options.maxParallelOperations ?? DEFAULT_MAX_PARALLEL_OPERATIONS)
     this.closeFile = sharedLimit(this.closeFile)
+    this.copy = sharedLimit(this.copy)
     this.getInfo = sharedLimit(this.getInfo)
     this.getSize = sharedLimit(this.getSize)
     this.list = sharedLimit(this.list)
@@ -307,6 +308,17 @@ export default class RemoteHandlerAbstract {
     return p
   }
 
+  async copy(oldPath, newPath, { checksum = false } = {}) {
+    oldPath = normalizePath(oldPath)
+    newPath = normalizePath(newPath)
+
+    let p = timeout.call(this._copy(oldPath, newPath), this._timeout)
+    if (checksum) {
+      p = Promise.all([p, this._copy(checksumFile(oldPath), checksumFile(newPath))])
+    }
+    return p
+  }
+
   async rmdir(dir) {
     await timeout.call(this._rmdir(normalizePath(dir)).catch(ignoreEnoent), this._timeout)
   }
@@ -519,6 +531,9 @@ export default class RemoteHandlerAbstract {
   async _rename(oldPath, newPath) {
     throw new Error('Not implemented')
   }
+  async _copy(oldPath, newPath) {
+    throw new Error('Not implemented')
+  }
 
   async _rmdir(dir) {
     throw new Error('Not implemented')

@xen-orchestra/fs/src/index.js

@@ -27,3 +27,12 @@ export const getHandler = (remote, ...rest) => {
   }
   return new Handler(remote, ...rest)
 }
+
+export const getSyncedHandler = async (...opts) => {
+  const handler = getHandler(...opts)
+  await handler.sync()
+  return {
+    dispose: () => handler.forget(),
+    value: handler,
+  }
+}

@xen-orchestra/fs/src/local.js

@@ -33,6 +33,10 @@ export default class LocalHandler extends RemoteHandlerAbstract {
     return fs.close(fd)
   }
 
+  async _copy(oldPath, newPath) {
+    return fs.copy(this._getFilePath(oldPath), this._getFilePath(newPath))
+  }
+
   async _createReadStream(file, options) {
     if (typeof file === 'string') {
       const stream = fs.createReadStream(this._getFilePath(file), options)

@xen-orchestra/fs/src/s3.js

@@ -1,6 +1,7 @@
 import aws from '@sullux/aws-sdk'
 import assert from 'assert'
 import http from 'http'
+import https from 'https'
 import { parse } from 'xo-remote-parser'
 
 import RemoteHandlerAbstract from './abstract'
@@ -16,7 +17,7 @@ const IDEAL_FRAGMENT_SIZE = Math.ceil(MAX_OBJECT_SIZE / MAX_PARTS_COUNT) // the
 export default class S3Handler extends RemoteHandlerAbstract {
   constructor(remote, _opts) {
     super(remote)
-    const { host, path, username, password, protocol, region } = parse(remote.url)
+    const { allowUnauthorized, host, path, username, password, protocol, region } = parse(remote.url)
     const params = {
       accessKeyId: username,
       apiVersion: '2006-03-01',
@@ -29,8 +30,13 @@ export default class S3Handler extends RemoteHandlerAbstract {
       },
     }
     if (protocol === 'http') {
-      params.httpOptions.agent = new http.Agent()
+      params.httpOptions.agent = new http.Agent({ keepAlive: true })
       params.sslEnabled = false
+    } else if (protocol === 'https') {
+      params.httpOptions.agent = new https.Agent({
+        rejectUnauthorized: !allowUnauthorized,
+        keepAlive: true,
+      })
     }
     if (region !== undefined) {
       params.region = region
@@ -51,6 +57,27 @@ export default class S3Handler extends RemoteHandlerAbstract {
     return { Bucket: this._bucket, Key: this._dir + file }
   }
 
+  async _copy(oldPath, newPath) {
+    const size = await this._getSize(oldPath)
+    const multipartParams = await this._s3.createMultipartUpload({ ...this._createParams(newPath) })
+    const param2 = { ...multipartParams, CopySource: `/${this._bucket}/${this._dir}${oldPath}` }
+    try {
+      const parts = []
+      let start = 0
+      while (start < size) {
+        const range = `bytes=${start}-${Math.min(start + MAX_PART_SIZE, size) - 1}`
+        const partParams = { ...param2, PartNumber: parts.length + 1, CopySourceRange: range }
+        const upload = await this._s3.uploadPartCopy(partParams)
+        parts.push({ ETag: upload.CopyPartResult.ETag, PartNumber: partParams.PartNumber })
+        start += MAX_PART_SIZE
+      }
+      await this._s3.completeMultipartUpload({ ...multipartParams, MultipartUpload: { Parts: parts } })
+    } catch (e) {
+      await this._s3.abortMultipartUpload(multipartParams)
+      throw e
+    }
+  }
+
   async _isNotEmptyDir(path) {
     const result = await this._s3.listObjectsV2({
       Bucket: this._bucket,
@@ -125,16 +152,30 @@ export default class S3Handler extends RemoteHandlerAbstract {
     const splitPrefix = splitPath(prefix)
     const result = await this._s3.listObjectsV2({
       Bucket: this._bucket,
-      Prefix: splitPrefix.join('/'),
+      Prefix: splitPrefix.join('/') + '/', // need slash at the end with the use of delimiters
+      Delimiter: '/', // will only return path until delimiters
     })
-    const uniq = new Set()
+
+    if (result.isTruncated) {
+      const error = new Error('more than 1000 objects, unsupported in this implementation')
+      error.dir = dir
+      throw error
+    }
+
+    const uniq = []
+
+    // sub directories
+    for (const entry of result.CommonPrefixes) {
+      const line = splitPath(entry.Prefix)
+      uniq.push(line[line.length - 1])
+    }
+    // files
     for (const entry of result.Contents) {
       const line = splitPath(entry.Key)
-      if (line.length > splitPrefix.length) {
-        uniq.add(line[splitPrefix.length])
-      }
+      uniq.push(line[line.length - 1])
     }
-    return [...uniq]
+
+    return uniq
   }
 
   async _mkdir(path) {
@@ -147,25 +188,9 @@ export default class S3Handler extends RemoteHandlerAbstract {
     // nothing to do, directories do not exist, they are part of the files' path
   }
 
+  // s3 doesn't have a rename operation, so copy + delete source
   async _rename(oldPath, newPath) {
-    const size = await this._getSize(oldPath)
-    const multipartParams = await this._s3.createMultipartUpload({ ...this._createParams(newPath) })
-    const param2 = { ...multipartParams, CopySource: `/${this._bucket}/${this._dir}${oldPath}` }
-    try {
-      const parts = []
-      let start = 0
-      while (start < size) {
-        const range = `bytes=${start}-${Math.min(start + MAX_PART_SIZE, size) - 1}`
-        const partParams = { ...param2, PartNumber: parts.length + 1, CopySourceRange: range }
-        const upload = await this._s3.uploadPartCopy(partParams)
-        parts.push({ ETag: upload.CopyPartResult.ETag, PartNumber: partParams.PartNumber })
-        start += MAX_PART_SIZE
-      }
-      await this._s3.completeMultipartUpload({ ...multipartParams, MultipartUpload: { Parts: parts } })
-    } catch (e) {
-      await this._s3.abortMultipartUpload(multipartParams)
-      throw e
-    }
+    await this.copy(oldPath, newPath)
     await this._s3.deleteObject(this._createParams(oldPath))
   }
 
@@ -183,9 +208,21 @@ export default class S3Handler extends RemoteHandlerAbstract {
     }
     const params = this._createParams(file)
     params.Range = `bytes=${position}-${position + buffer.length - 1}`
-    const result = await this._s3.getObject(params)
-    result.Body.copy(buffer)
-    return { bytesRead: result.Body.length, buffer }
+    try {
+      const result = await this._s3.getObject(params)
+      result.Body.copy(buffer)
+      return { bytesRead: result.Body.length, buffer }
+    } catch (e) {
+      if (e.code === 'NoSuchKey') {
+        if (await this._isNotEmptyDir(file)) {
+          const error = new Error(`${file} is a directory`)
+          error.code = 'EISDIR'
+          error.path = file
+          throw error
+        }
+      }
+      throw e
+    }
   }
 
   async _rmdir(path) {
@@ -199,6 +236,28 @@ export default class S3Handler extends RemoteHandlerAbstract {
     // nothing to do, directories do not exist, they are part of the files' path
   }
 
+  // reimplement _rmtree to handle efficiantly path with more than 1000 entries in trees
+  // @todo : use parallel processing for unlink
+  async _rmtree(path) {
+    let NextContinuationToken
+    do {
+      const result = await this._s3.listObjectsV2({
+        Bucket: this._bucket,
+        Prefix: this._dir + path + '/',
+        ContinuationToken: NextContinuationToken,
+      })
+      NextContinuationToken = result.isTruncated ? result.NextContinuationToken : undefined
+      for (const { Key } of result.Contents) {
+        // _unlink will add the prefix, but Key contains everything
+        // also we don't need to check if we delete a directory, since the list only return files
+        await this._s3.deleteObject({
+          Bucket: this._bucket,
+          Key,
+        })
+      }
+    } while (NextContinuationToken !== undefined)
+  }
+
   async _write(file, buffer, position) {
     if (typeof file !== 'string') {
       file = file.fd

@xen-orchestra/log/README.md

@@ -66,6 +66,10 @@ configure([
     // if filter is a string, then it is pattern
     // (https://github.com/visionmedia/debug#wildcards) which is
     // matched against the namespace of the logs
+    //
+    // If it's an array, it will be handled as an array of filters
+    // and the transport will be used if any one of them match the
+    // current log
     filter: process.env.DEBUG,
 
     transport: transportConsole(),

@xen-orchestra/log/USAGE.md

@@ -48,6 +48,10 @@ configure([
     // if filter is a string, then it is pattern
     // (https://github.com/visionmedia/debug#wildcards) which is
     // matched against the namespace of the logs
+    //
+    // If it's an array, it will be handled as an array of filters
+    // and the transport will be used if any one of them match the
+    // current log
     filter: process.env.DEBUG,
 
     transport: transportConsole(),

@xen-orchestra/log/configure.js

@@ -4,6 +4,42 @@ const { compileGlobPattern } = require('./utils')
 
 // ===================================================================
 
+const compileFilter = filter => {
+  if (filter === undefined) {
+    return
+  }
+
+  const type = typeof filter
+  if (type === 'function') {
+    return filter
+  }
+  if (type === 'string') {
+    const re = compileGlobPattern(filter)
+    return log => re.test(log.namespace)
+  }
+
+  if (Array.isArray(filter)) {
+    const filters = filter.map(compileFilter).filter(_ => _ !== undefined)
+    const { length } = filters
+    if (length === 0) {
+      return
+    }
+    if (length === 1) {
+      return filters[0]
+    }
+    return log => {
+      for (let i = 0; i < length; ++i) {
+        if (filters[i](log)) {
+          return true
+        }
+      }
+      return false
+    }
+  }
+
+  throw new TypeError('unsupported `filter`')
+}
+
 const createTransport = config => {
   if (typeof config === 'function') {
     return config
@@ -19,26 +55,15 @@ const createTransport = config => {
     }
   }
 
-  let { filter } = config
-  let transport = createTransport(config.transport)
   const level = resolve(config.level)
+  const filter = compileFilter([config.filter, level === undefined ? undefined : log => log.level >= level])
+
+  let transport = createTransport(config.transport)
 
   if (filter !== undefined) {
-    if (typeof filter === 'string') {
-      const re = compileGlobPattern(filter)
-      filter = log => re.test(log.namespace)
-    }
-
     const orig = transport
     transport = function (log) {
-      if ((level !== undefined && log.level >= level) || filter(log)) {
-        return orig.apply(this, arguments)
-      }
-    }
-  } else if (level !== undefined) {
-    const orig = transport
-    transport = function (log) {
-      if (log.level >= level) {
+      if (filter(log)) {
         return orig.apply(this, arguments)
       }
     }

@xen-orchestra/log/package.json

@@ -1,7 +1,7 @@
 {
   "private": false,
   "name": "@xen-orchestra/log",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "license": "ISC",
   "description": "Logging system with decoupled producers/consumer",
   "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/log",
@@ -24,7 +24,7 @@
   },
   "dependencies": {
     "lodash": "^4.17.4",
-    "promise-toolbox": "^0.19.2"
+    "promise-toolbox": "^0.20.0"
   },
   "scripts": {
     "postversion": "npm publish"

@xen-orchestra/log/transports/console.js

@@ -20,36 +20,8 @@ if (process.stdout !== undefined && process.stdout.isTTY && process.stderr !== u
   }
 
   const NAMESPACE_COLORS = [
-    196,
-    202,
-    208,
-    214,
-    220,
-    226,
-    190,
-    154,
-    118,
-    82,
-    46,
-    47,
-    48,
-    49,
-    50,
-    51,
-    45,
-    39,
-    33,
-    27,
-    21,
-    57,
-    93,
-    129,
-    165,
-    201,
-    200,
-    199,
-    198,
-    197,
+    196, 202, 208, 214, 220, 226, 190, 154, 118, 82, 46, 47, 48, 49, 50, 51, 45, 39, 33, 27, 21, 57, 93, 129, 165, 201,
+    200, 199, 198, 197,
   ]
   formatNamespace = namespace => {
     // https://werxltd.com/wp/2010/05/13/javascript-implementation-of-javas-string-hashcode-method/

@xen-orchestra/mixin/package.json

@@ -19,7 +19,7 @@
     "node": ">=6"
   },
   "dependencies": {
-    "bind-property-descriptor": "^1.0.0",
+    "bind-property-descriptor": "^2.0.0",
     "lodash": "^4.17.21"
   },
   "scripts": {

@xen-orchestra/mixins/Config.js

@@ -1,5 +1,6 @@
 const get = require('lodash/get')
 const identity = require('lodash/identity')
+const isEqual = require('lodash/isEqual')
 const { createLogger } = require('@xen-orchestra/log')
 const { parseDuration } = require('@vates/parse-duration')
 const { watch } = require('app-conf')
@@ -12,7 +13,7 @@ module.exports = class Config {
     const watchers = (this._watchers = new Set())
 
     app.hooks.on('start', async () => {
-      app.hooks.on(
+      app.hooks.once(
         'stop',
         await watch({ appDir, appName, ignoreUnknownFormats: true }, (error, config) => {
           if (error != null) {
@@ -31,7 +32,7 @@ module.exports = class Config {
   get(path) {
     const value = get(this._config, path)
     if (value === undefined) {
-      throw new TypeError('missing config entry: ' + value)
+      throw new TypeError('missing config entry: ' + path)
     }
     return value
   }
@@ -48,7 +49,7 @@ module.exports = class Config {
     const watcher = config => {
       try {
         const value = processor(get(config, path))
-        if (value !== prev) {
+        if (!isEqual(value, prev)) {
           prev = value
           cb(value)
         }

@xen-orchestra/mixins/package.json

@@ -14,15 +14,15 @@
     "url": "https://vates.fr"
   },
   "license": "AGPL-3.0-or-later",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "engines": {
     "node": ">=12"
   },
   "dependencies": {
     "@vates/parse-duration": "^0.1.1",
-    "@xen-orchestra/emit-async": "^0.0.0",
-    "@xen-orchestra/log": "^0.2.0",
-    "app-conf": "^0.9.0",
+    "@xen-orchestra/emit-async": "^0.1.0",
+    "@xen-orchestra/log": "^0.3.0",
+    "app-conf": "^1.0.0",
     "lodash": "^4.17.21"
   },
   "scripts": {

@xen-orchestra/openflow/src/util/addrress-parser.js

@@ -28,9 +28,10 @@ export default {
     buffer.toString('hex', offset + 5, offset + 6),
 
   stringToEth: (string, buffer, offset) => {
-    const eth = /^([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2})$/.exec(
-      string
-    )
+    const eth =
+      /^([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2}):([0-9A-Fa-f]{2})$/.exec(
+        string
+      )
     assert(eth !== null)
     buffer.writeUInt8(parseInt(eth[1], 16), offset)
     buffer.writeUInt8(parseInt(eth[2], 16), offset + 1)
@@ -50,9 +51,10 @@ export default {
     ),
 
   stringToip4: (string, buffer, offset) => {
-    const ip = /^([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])$/.exec(
-      string
-    )
+    const ip =
+      /^([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])\.([1-9]?\d|1\d\d|2[0-4]\d|25[0-5])$/.exec(
+        string
+      )
     assert(ip !== null)
     buffer.writeUInt8(parseInt(ip[1], 10), offset)
     buffer.writeUInt8(parseInt(ip[2], 10), offset + 1)

@xen-orchestra/proxy-cli/package.json

@@ -23,22 +23,22 @@
     "xo-proxy-cli": "dist/index.js"
   },
   "engines": {
-    "node": ">=8.10"
+    "node": ">=12"
   },
   "dependencies": {
     "@iarna/toml": "^2.2.0",
     "@vates/read-chunk": "^0.1.2",
     "ansi-colors": "^4.1.1",
-    "app-conf": "^0.9.0",
+    "app-conf": "^1.0.0",
     "content-type": "^1.0.4",
     "cson-parser": "^4.0.7",
     "getopts": "^2.2.3",
-    "http-request-plus": "^0.10.0",
+    "http-request-plus": "^0.13.0",
     "json-rpc-protocol": "^0.13.1",
-    "promise-toolbox": "^0.19.2",
+    "promise-toolbox": "^0.20.0",
     "pump": "^3.0.0",
     "pumpify": "^2.0.1",
-    "split2": "^3.1.1"
+    "split2": "^4.1.0"
   },
   "devDependencies": {
     "@babel/cli": "^7.0.0",

@xen-orchestra/proxy-cli/src/index.js

@@ -36,7 +36,14 @@ async function main(argv) {
 
   const { hostname = 'localhost', port } = config?.http?.listen?.https ?? {}
 
-  const { _: args, file, help, host, raw, token } = getopts(argv, {
+  const {
+    _: args,
+    file,
+    help,
+    host,
+    raw,
+    token,
+  } = getopts(argv, {
     alias: { file: 'f', help: 'h' },
     boolean: ['help', 'raw'],
     default: {
@@ -140,16 +147,6 @@ ${pkg.name} v${pkg.version}`
     }
   }
 
-  const $import = ({ $import: path }) => {
-    const data = fs.readFileSync(path, 'utf8')
-    const ext = extname(path).slice(1).toLowerCase()
-    const parse = FORMATS[ext]
-    if (parse === undefined) {
-      throw new Error(`unsupported file: ${path}`)
-    }
-    return visit(parse(data))
-  }
-
   const seq = async seq => {
     const j = callPath.length
     for (let i = 0, n = seq.length; i < n; ++i) {
@@ -163,13 +160,17 @@ ${pkg.name} v${pkg.version}`
     if (Array.isArray(node)) {
       return seq(node)
     }
-    const keys = Object.keys(node)
-    return keys.length === 1 && keys[0] === '$import' ? $import(node) : call(node)
+    return call(node)
   }
 
-  let node
   if (file !== '') {
-    node = { $import: file }
+    const data = fs.readFileSync(file, 'utf8')
+    const ext = extname(file).slice(1).toLowerCase()
+    const parse = FORMATS[ext]
+    if (parse === undefined) {
+      throw new Error(`unsupported file: ${file}`)
+    }
+    await visit(parse(data))
   } else {
     const method = args[0]
     const params = {}
@@ -182,9 +183,8 @@ ${pkg.name} v${pkg.version}`
       params[param.slice(0, j)] = parseValue(param.slice(j + 1))
     }
 
-    node = { method, params }
+    await call({ method, params })
   }
-  await visit(node)
 }
 main(process.argv.slice(2)).then(
   () => {

@xen-orchestra/proxy/config.toml

@@ -18,7 +18,9 @@ keepAliveInterval = 10e3
 #
 # https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation
 dirMode = 0o700
+disableMergeWorker = false
 snapshotNameLabelTpl = '[XO Backup {job.name}] {vm.name_label}'
+vhdDirectoryCompression = 'brotli'
 
 [backups.defaultSettings]
 reportWhen = 'failure'
@@ -59,6 +61,13 @@ cert = '/var/lib/xo-proxy/certificate.pem'
 key = '/var/lib/xo-proxy/key.pem'
 port = 443
 
+[logs]
+# Display all logs matching this filter, regardless of their level
+#filter = 'xo:backups:*'
+
+# Display all logs with level >=, regardless of their namespace
+level = 'info'
+
 [remoteOptions]
 mountsDir = '/run/xo-proxy/mounts'
 
@@ -79,3 +88,20 @@ ignoreNobakVdis = false
 
 maxUncoalescedVdis = 1
 watchEvents = ['network', 'PIF', 'pool', 'SR', 'task', 'VBD', 'VDI', 'VIF', 'VM']
+
+
+
+#compact mode
+[reverseProxies]
+# '/http/' = 'http://localhost:8081/'
+#The target can have a path ( like `http://target/sub/directory/`),
+# parameters (`?param=one`) and hash (`#jwt:32154`) that are automatically added to all queries transfered by the proxy.
+# If a parameter is present in the configuration and in the query, only the config parameter is transferred.
+# '/another' = http://hiddenServer:8765/path/
+
+# And use the extended mode when required
+# The additionnal options of a proxy's configuraiton's section are used to instantiate the `https` Agent(respectively the `http`).
+# A notable option is `rejectUnauthorized` which allow to connect to a HTTPS backend  with an invalid/ self signed certificate
+#[reverseProxies.'/https/']
+# target = 'https://localhost:8080/'
+# rejectUnauthorized = false

@xen-orchestra/proxy/docs/api.md

@@ -93,10 +93,7 @@ declare namespace event {
 declare namespace backup {
   type SimpleIdPattern = { id: string | { __or: string[] } }
 
-declare namespace backup {
-  type SimpleIdPattern = { id: string | { __or: string[] } }
-
-  interface BackupJob {
+  interface BackupJob {
     id: string
     type: 'backup'
     compression?: 'native' | 'zstd' | ''
@@ -146,13 +143,13 @@ declare namespace backup {
   }
 
   function listXoMetadataBackups(_: { remotes: { [id: string]: Remote } }): { [remoteId: string]: object[] }
-
-  function run(_: {
-    job: BackupJob | MetadataBackupJob
+
+  function run(_: {
+    job: BackupJob | MetadataBackupJob
     remotes: { [id: string]: Remote }
-    schedule: Schedule
-    xapis?: { [id: string]: Xapi }
-    recordToXapi?: { [recordUuid: string]: string }
+    schedule: Schedule
+    xapis?: { [id: string]: Xapi }
+    recordToXapi?: { [recordUuid: string]: string }
     streamLogs: boolean = false
   }): string
 

@xen-orchestra/proxy/package.json

@@ -1,7 +1,7 @@
 {
   "private": true,
   "name": "@xen-orchestra/proxy",
-  "version": "0.13.1",
+  "version": "0.17.3",
   "license": "AGPL-3.0-or-later",
   "description": "XO Proxy used to remotely execute backup jobs",
   "keywords": [
@@ -18,48 +18,48 @@
     "url": "https://github.com/vatesfr/xen-orchestra.git"
   },
   "preferGlobal": true,
-  "main": "dist/",
   "bin": {
-    "xo-proxy": "dist/index.js"
+    "xo-proxy": "dist/index.mjs"
   },
   "engines": {
-    "node": ">=14.13"
+    "node": ">=14.18"
   },
   "dependencies": {
     "@iarna/toml": "^2.2.0",
     "@koa/router": "^10.0.0",
-    "@vates/compose": "^2.0.0",
-    "@vates/decorate-with": "^0.0.1",
+    "@vates/compose": "^2.1.0",
+    "@vates/decorate-with": "^1.0.0",
     "@vates/disposable": "^0.1.1",
     "@xen-orchestra/async-map": "^0.1.2",
-    "@xen-orchestra/backups": "^0.11.0",
-    "@xen-orchestra/fs": "^0.17.0",
-    "@xen-orchestra/log": "^0.2.0",
+    "@xen-orchestra/backups": "^0.18.3",
+    "@xen-orchestra/fs": "^0.19.3",
+    "@xen-orchestra/log": "^0.3.0",
     "@xen-orchestra/mixin": "^0.1.0",
-    "@xen-orchestra/mixins": "^0.1.0",
+    "@xen-orchestra/mixins": "^0.1.2",
     "@xen-orchestra/self-signed": "^0.1.0",
-    "@xen-orchestra/xapi": "^0.6.2",
+    "@xen-orchestra/xapi": "^0.8.5",
     "ajv": "^8.0.3",
-    "app-conf": "^0.9.0",
+    "app-conf": "^1.0.0",
     "async-iterator-to-stream": "^1.1.0",
-    "fs-extra": "^9.1.0",
+    "fs-extra": "^10.0.0",
     "get-stream": "^6.0.0",
     "getopts": "^2.2.3",
     "golike-defer": "^0.5.1",
     "http-server-plus": "^0.11.0",
+    "http2-proxy": "^5.0.53",
     "json-rpc-protocol": "^0.13.1",
-    "jsonrpc-websocket-client": "^0.6.0",
+    "jsonrpc-websocket-client": "^0.7.2",
     "koa": "^2.5.1",
     "koa-compress": "^5.0.1",
     "koa-helmet": "^5.1.0",
     "lodash": "^4.17.10",
     "node-zone": "^0.4.0",
     "parse-pairs": "^1.0.0",
-    "promise-toolbox": "^0.19.2",
+    "promise-toolbox": "^0.20.0",
     "source-map-support": "^0.5.16",
     "stoppable": "^1.0.6",
-    "xdg-basedir": "^4.0.0",
-    "xen-api": "^0.32.0",
+    "xdg-basedir": "^5.1.0",
+    "xen-api": "^0.35.1",
     "xo-common": "^0.7.0"
   },
   "devDependencies": {
@@ -73,7 +73,7 @@
     "@vates/toggle-scripts": "^1.0.0",
     "babel-plugin-transform-dev": "^2.0.1",
     "cross-env": "^7.0.2",
-    "index-modules": "^0.4.0"
+    "index-modules": "^0.4.3"
   },
   "scripts": {
     "_build": "index-modules --index-file index.mjs src/app/mixins && babel --delete-dir-on-start --keep-file-extension --source-maps --out-dir=dist/ src/",
@@ -84,7 +84,7 @@
     "prepack": "toggle-scripts +postinstall +preuninstall",
     "prepublishOnly": "yarn run build",
     "_preuninstall": "./scripts/systemd-service-installer",
-    "start": "./dist/index.js"
+    "start": "./dist/index.mjs"
   },
   "author": {
     "name": "Vates SAS",

@xen-orchestra/proxy/src/app/mixins/api.mjs

@@ -15,13 +15,29 @@ import { createLogger } from '@xen-orchestra/log'
 const { debug, warn } = createLogger('xo:proxy:api')
 
 const ndJsonStream = asyncIteratorToStream(async function* (responseId, iterable) {
-  yield format.response(responseId, { $responseType: 'ndjson' }) + '\n'
-  for await (const data of iterable) {
+  try {
+    let cursor, iterator
     try {
-      yield JSON.stringify(data) + '\n'
+      const getIterator = iterable[Symbol.iterator] ?? iterable[Symbol.asyncIterator]
+      iterator = getIterator.call(iterable)
+
+      cursor = await iterator.next()
+      yield format.response(responseId, { $responseType: 'ndjson' }) + '\n'
     } catch (error) {
-      warn('ndJsonStream', { error })
+      yield format.error(responseId, error)
+      throw error
     }
+
+    while (!cursor.done) {
+      try {
+        yield JSON.stringify(cursor.value) + '\n'
+      } catch (error) {
+        warn('ndJsonStream, item error', { error })
+      }
+      cursor = await iterator.next()
+    }
+  } catch (error) {
+    warn('ndJsonStream, fatal error', { error })
   }
 })
 
@@ -29,8 +45,8 @@ export default class Api {
   constructor(app, { appVersion, httpServer }) {
     this._ajv = new Ajv({ allErrors: true })
     this._methods = { __proto__: null }
-
-    const router = new Router({ prefix: '/api/v1' }).post('/', async ctx => {
+    const PREFIX = '/api/v1'
+    const router = new Router({ prefix: PREFIX }).post('/', async ctx => {
       // Before Node 13.0 there was an inactivity timeout of 2 mins, which may
       // not be enough for the API.
       ctx.req.setTimeout(0)
@@ -86,6 +102,7 @@ export default class Api {
           // breaks, send some data every 10s to keep it opened.
           const stopTimer = clearInterval.bind(
             undefined,
+            // @to check : can this add space inside binary data ?
             setInterval(() => stream.push(' '), keepAliveInterval)
           )
           stream.on('end', stopTimer).on('error', stopTimer)
@@ -102,7 +119,14 @@ export default class Api {
       .use(router.routes())
       .use(router.allowedMethods())
 
-    httpServer.on('request', koa.callback())
+    const callback = koa.callback()
+    httpServer.on('request', (req, res) => {
+      // only answers to query to the root url of this mixin
+      // do it before giving the request to Koa to ensure it's not modified
+      if (req.url.startsWith(PREFIX)) {
+        callback(req, res)
+      }
+    })
 
     this.addMethods({
       system: {

@xen-orchestra/proxy/src/app/mixins/authentication.mjs

@@ -1,5 +1,6 @@
+import assert from 'assert'
 import fse from 'fs-extra'
-import xdg from 'xdg-basedir'
+import { xdgConfig } from 'xdg-basedir'
 import { createLogger } from '@xen-orchestra/log'
 import { execFileSync } from 'child_process'
 
@@ -10,33 +11,48 @@ const { warn } = createLogger('xo:proxy:authentication')
 const isValidToken = t => typeof t === 'string' && t.length !== 0
 
 export default class Authentication {
-  constructor(_, { appName, config: { authenticationToken: token } }) {
-    if (!isValidToken(token)) {
-      token = JSON.parse(execFileSync('xenstore-read', ['vm-data/xo-proxy-authenticationToken']))
+  #token
 
-      if (!isValidToken(token)) {
-        throw new Error('missing authenticationToken in configuration')
-      }
+  constructor(app, { appName, config: { authenticationToken: token } }) {
+    const setToken = ({ token }) => {
+      assert(isValidToken(token), 'invalid authentication token: ' + token)
+
+      // save this token in the automatically handled conf file
+      fse.outputFileSync(
+        // this file must take precedence over normal user config
+        `${xdgConfig}/${appName}/config.z-auto.json`,
+        JSON.stringify({ authenticationToken: token }),
+        { mode: 0o600 }
+      )
+
+      this.#token = token
+    }
+
+    if (isValidToken(token)) {
+      this.#token = token
+    } else {
+      setToken({ token: JSON.parse(execFileSync('xenstore-read', ['vm-data/xo-proxy-authenticationToken'])) })
 
       try {
-        // save this token in the automatically handled conf file
-        fse.outputFileSync(
-          // this file must take precedence over normal user config
-          `${xdg.config}/${appName}/config.z-auto.json`,
-          JSON.stringify({ authenticationToken: token }),
-          { mode: 0o600 }
-        )
         execFileSync('xenstore-rm', ['vm-data/xo-proxy-authenticationToken'])
       } catch (error) {
         warn('failed to remove token from XenStore', { error })
       }
     }
 
-    this._token = token
+    app.api.addMethod('authentication.setToken', setToken, {
+      description: 'change the authentication token used by this XO Proxy',
+      params: {
+        token: {
+          type: 'string',
+          minLength: 1,
+        },
+      },
+    })
   }
 
   async findProfile(credentials) {
-    if (credentials?.authenticationToken === this._token) {
+    if (credentials?.authenticationToken === this.#token) {
       return new Profile()
     }
   }

@xen-orchestra/proxy/src/app/mixins/backups.mjs

@@ -1,5 +1,3 @@
-import Cancel from 'promise-toolbox/Cancel'
-import CancelToken from 'promise-toolbox/CancelToken'
 import Disposable from 'promise-toolbox/Disposable.js'
 import fromCallback from 'promise-toolbox/fromCallback.js'
 import { asyncMap } from '@xen-orchestra/async-map'
@@ -13,6 +11,7 @@ import { DurablePartition } from '@xen-orchestra/backups/DurablePartition.js'
 import { execFile } from 'child_process'
 import { formatVmBackups } from '@xen-orchestra/backups/formatVmBackups.js'
 import { ImportVmBackup } from '@xen-orchestra/backups/ImportVmBackup.js'
+import { JsonRpcError } from 'json-rpc-protocol'
 import { Readable } from 'stream'
 import { RemoteAdapter } from '@xen-orchestra/backups/RemoteAdapter.js'
 import { RestoreMetadataBackup } from '@xen-orchestra/backups/RestoreMetadataBackup.js'
@@ -97,8 +96,7 @@ export default class Backups {
           error.jobId = jobId
           throw error
         }
-        const source = CancelToken.source()
-        runningJobs[jobId] = source.cancel
+        runningJobs[jobId] = true
         try {
           return await run.apply(this, arguments)
         } finally {
@@ -111,7 +109,7 @@ export default class Backups {
         if (!__DEV__) {
           const license = await app.appliance.getSelfLicense()
           if (license === undefined) {
-            throw new Error('no valid proxy license')
+            throw new JsonRpcError('no valid proxy license')
           }
         }
         return run.apply(this, arguments)
@@ -166,6 +164,17 @@ export default class Backups {
             },
           },
         ],
+        deleteVmBackups: [
+          ({ filenames, remote }) =>
+            Disposable.use(this.getAdapter(remote), adapter => adapter.deleteVmBackups(filenames)),
+          {
+            description: 'delete VM backups',
+            params: {
+              filenames: { type: 'array', items: { type: 'string' } },
+              remote: { type: 'object' },
+            },
+          },
+        ],
         fetchPartitionFiles: [
           ({ disk: diskId, remote, partition: partitionId, paths }) =>
             Disposable.use(this.getAdapter(remote), adapter => adapter.fetchPartitionFiles(diskId, partitionId, paths)),
@@ -405,6 +414,7 @@ export default class Backups {
     return new RemoteAdapter(yield app.remotes.getHandler(remote), {
       debounceResource: app.debounceResource.bind(app),
       dirMode: app.config.get('backups.dirMode'),
+      vhdDirectoryCompression: app.config.get('backups.vhdDirectoryCompression'),
     })
   }
 

@xen-orchestra/proxy/src/app/mixins/logs.mjs

@@ -0,0 +1,17 @@
+import transportConsole from '@xen-orchestra/log/transports/console.js'
+import { configure } from '@xen-orchestra/log/configure.js'
+
+export default class Logs {
+  constructor(app) {
+    const transport = transportConsole()
+    app.config.watch('logs', ({ filter, level }) => {
+      configure([
+        {
+          filter: [process.env.DEBUG, filter],
+          level,
+          transport,
+        },
+      ])
+    })
+  }
+}

@xen-orchestra/proxy/src/app/mixins/reverseProxy.mjs

@@ -0,0 +1,120 @@
+import { urlToHttpOptions } from 'url'
+import proxy from 'http2-proxy'
+
+function removeSlash(str) {
+  return str.replace(/^\/|\/$/g, '')
+}
+
+function mergeUrl(relative, base) {
+  const res = new URL(base)
+  const relativeUrl = new URL(relative, base)
+  res.pathname = relativeUrl.pathname
+  relativeUrl.searchParams.forEach((value, name) => {
+    // we do not allow to modify params already specified by config
+    if (!res.searchParams.has(name)) {
+      res.searchParams.append(name, value)
+    }
+  })
+  res.hash = relativeUrl.hash.length > 0 ? relativeUrl.hash : res.hash
+  return res
+}
+
+export function backendToLocalPath(basePath, target, backendUrl) {
+  // keep redirect url relative to local server
+  const localPath = `${basePath}/${backendUrl.pathname.substring(target.pathname.length)}${backendUrl.search}${
+    backendUrl.hash
+  }`
+  return localPath
+}
+
+export function localToBackendUrl(basePath, target, localPath) {
+  let localPathWithoutBase = removeSlash(localPath).substring(basePath.length)
+  localPathWithoutBase = './' + removeSlash(localPathWithoutBase)
+  const url = mergeUrl(localPathWithoutBase, target)
+  return url
+}
+
+export default class ReverseProxy {
+  constructor(app, { httpServer }) {
+    app.config.watch('reverseProxies', proxies => {
+      this._proxies = Object.keys(proxies)
+        .sort((a, b) => b.length - a.length)
+        .map(path => {
+          let config = proxies[path]
+          if (typeof config === 'string') {
+            config = { target: config }
+          }
+          config.path = '/proxy/v1/' + removeSlash(path) + '/'
+
+          return config
+        })
+    })
+
+    httpServer.on('request', (req, res) => this._proxy(req, res))
+    httpServer.on('upgrade', (req, socket, head) => this._upgrade(req, socket, head))
+  }
+
+  _getConfigFromRequest(req) {
+    return this._proxies.find(({ path }) => req.url.startsWith(path))
+  }
+
+  _proxy(req, res) {
+    const config = this._getConfigFromRequest(req)
+
+    if (config === undefined) {
+      res.writeHead(404)
+      res.end('404')
+      return
+    }
+
+    const url = new URL(config.target)
+    const targetUrl = localToBackendUrl(config.path, url, req.originalUrl || req.url)
+    proxy.web(req, res, {
+      ...urlToHttpOptions(targetUrl),
+      ...config.options,
+      onReq: (req, { headers }) => {
+        headers['x-forwarded-for'] = req.socket.remoteAddress
+        headers['x-forwarded-proto'] = req.socket.encrypted ? 'https' : 'http'
+        if (req.headers.host !== undefined) {
+          headers['x-forwarded-host'] = req.headers.host
+        }
+      },
+      onRes: (req, res, proxyRes) => {
+        // rewrite redirect to pass through this proxy
+        if (proxyRes.statusCode === 301 || proxyRes.statusCode === 302) {
+          // handle relative/ absolute location
+          const redirectTargetLocation = new URL(proxyRes.headers.location, url)
+
+          // this proxy should only allow communication between known hosts. Don't open it too much
+          if (redirectTargetLocation.hostname !== url.hostname || redirectTargetLocation.protocol !== url.protocol) {
+            throw new Error(`Can't redirect from ${url.hostname} to ${redirectTargetLocation.hostname} `)
+          }
+          res.writeHead(proxyRes.statusCode, {
+            ...proxyRes.headers,
+            location: backendToLocalPath(config.path, url, redirectTargetLocation),
+          })
+          res.end()
+          return
+        }
+        // pass through the answer of the remote server
+        res.writeHead(proxyRes.statusCode, proxyRes.headers)
+        // pass through content
+        proxyRes.pipe(res)
+      },
+    })
+  }
+
+  _upgrade(req, socket, head) {
+    const config = this._getConfigFromRequest(req)
+    if (config === undefined) {
+      return
+    }
+
+    const { path, target, options } = config
+    const targetUrl = localToBackendUrl(path, target, req.originalUrl || req.url)
+    proxy.ws(req, socket, head, {
+      ...urlToHttpOptions(targetUrl),
+      ...options,
+    })
+  }
+}

@xen-orchestra/proxy/src/app/mixins/task.mjs

@@ -1,38 +0,0 @@
-import { asyncMapSettled } from '@xen-orchestra/async-map'
-
-export default class Task {
-  #tasks = new Map()
-
-  constructor(app) {
-    const tasks = new Map()
-    this.#tasks = tasks
-
-    app.api.addMethods({
-      task: {
-        *list() {
-          for (const id of tasks.keys()) {
-            yield { id }
-          }
-        },
-        cancel: [
-          ({ taskId }) => this.cancel(taskId),
-          {
-            params: {
-              taskId: { type: 'string' },
-            },
-          },
-        ],
-      },
-    })
-
-    app.hooks.on('stop', () => asyncMapSettled(tasks.values(), task => task.cancel()))
-  }
-
-  async cancel(taskId) {
-    await this.tasks.get(taskId).cancel()
-  }
-
-  register(task) {
-    this.#tasks.set(task.id, task)
-  }
-}

@xen-orchestra/proxy/tests/cert.pem

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDETCCAfkCFHXO1U7YJHI61bPNhYDvyBNJYH4LMA0GCSqGSIb3DQEBCwUAMEUx
+CzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRl
+cm5ldCBXaWRnaXRzIFB0eSBMdGQwHhcNMjIwMTEwMTI0MTU4WhcNNDkwNTI3MTI0
+MTU4WjBFMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UE
+CgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOC
+AQ8AMIIBCgKCAQEA1jMLdHuZu2R1fETyB2iRect1alwv76clp/7A8tx4zNaVA9qB
+BcHbI83mkozuyrXpsEUblTvvcWkheBPAvWD4gj0eWSDSiuf0edcIS6aky+Lr/n0T
+W/vL5kVNrgPTlsO8OyQcXjDeuUOR1xDWIa8G71Ynd6wtATB7oXe7kaV/Z6b2fENr
+4wlW0YEDnMHik59c9jXDshhQYDlErwZsSyHuLwkC7xuYO26SUW9fPcHJA3uOfxeG
+BrCxMuSMOJtdmslRWhLCjbk0PT12OYCCRlvuTvPHa8N57GEQbi4xAu+XATgO1DUm
+Dq/oCSj0TcWUXXOykN/PAC2cjIyqkU2e7orGaQIDAQABMA0GCSqGSIb3DQEBCwUA
+A4IBAQCTshhF3V5WVhnpFGHd+tPfeHmUVrUnbC+xW7fSeWpamNmTjHb7XB6uDR0O
+DGswhEitbbSOsCiwz4/zpfE3/3+X07O8NPbdHTVHCei6D0uyegEeWQ2HoocfZs3X
+8CORe8TItuvQAevV17D0WkGRoJGVAOiKo+izpjI55QXQ+FjkJ0bfl1iksnUJk0+I
+ZNmRRNjNyOxo7NAzomSBHfJ5rDE+E440F2uvXIE9OIwHRiq6FGvQmvGijPeeP5J0
+LzcSK98jfINFSsA/Wn5vWE+gfH9ySD2G3r2cDTS904T77PNiYH+cNSP6ujtmNzvK
+Bgoa3jXZPRBi82TUOb2jj5DB33bg
+-----END CERTIFICATE-----

@xen-orchestra/proxy/tests/key.pem

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEA1jMLdHuZu2R1fETyB2iRect1alwv76clp/7A8tx4zNaVA9qB
+BcHbI83mkozuyrXpsEUblTvvcWkheBPAvWD4gj0eWSDSiuf0edcIS6aky+Lr/n0T
+W/vL5kVNrgPTlsO8OyQcXjDeuUOR1xDWIa8G71Ynd6wtATB7oXe7kaV/Z6b2fENr
+4wlW0YEDnMHik59c9jXDshhQYDlErwZsSyHuLwkC7xuYO26SUW9fPcHJA3uOfxeG
+BrCxMuSMOJtdmslRWhLCjbk0PT12OYCCRlvuTvPHa8N57GEQbi4xAu+XATgO1DUm
+Dq/oCSj0TcWUXXOykN/PAC2cjIyqkU2e7orGaQIDAQABAoIBAQC65uVq6WLWGa1O
+FtbdUggGL1svyGrngYChGvB/uZMKoX57U1DbljDCCCrV23WNmbfkYBjWWervmZ1j
+qlC2roOJGQ1/Fd3A6O7w1YnegPUxFrt3XunijE55iiVi3uHknryDGlpKcfgVzfjW
+oVFHKPMzKYjcqnbGn+hwlwoq5y7JYFTOa57/dZbyommbodRyy9Dpn0OES0grQqwR
+VD1amrQ7XJhukcxQgYPuDc/jM3CuowoBsv9f+Q2zsPgr6CpHxxLLIs+kt8NQJT9v
+neg/pm8ojcwOa9qoILdtu6ue7ee3VE9cFnB1vutxS1+MPeI5wgTJjaYrgPCMxXBM
+2LdJJEmBAoGBAPA6LpuU1vv5R3x66hzenSk4LS1fj24K0WuBdTwFvzQmCr70oKdo
+Yywxt+ZkBw5aEtzQlB8GewolHobDJrzxMorU+qEXX3jP2BIPDVQl2orfjr03Yyus
+s5mYS/Qa6Zf1yObrjulTNm8oTn1WaG3TIvi8c5DyG2OK28N/9oMI1XGRAoGBAORD
+YKyII/S66gZsJSf45qmrhq1hHuVt1xae5LUPP6lVD+MCCAmuoJnReV8fc9h7Dvgd
+YPMINkWUTePFr3o4p1mh2ZC7ldczgDn6X4TldY2J3Zg47xJa5hL0L6JL4NiCGRIE
+FV5rLJxkGh/DDBfmC9hQQ6Yg6cHvyewso5xVnBtZAoGAI+OdWPMIl0ZrrqYyWbPM
+aP8SiMfRBtCo7tW9bQUyxpi0XEjxw3Dt+AlJfysMftFoJgMnTedK9H4NLHb1T579
+PQ6KjwyN39+1WSVUiXDKUJsLmSswLrMzdcvx9PscUO6QYCdrB2K+LCcqasFBAr9b
+ZyvIXCw/eUSihneUnYjxUnECgYAoPgCzKiU8ph9QFozOaUExNH4/3tl1lVHQOR8V
+FKUik06DtP35xwGlXJrLPF5OEhPnhjZrYk0/IxBAUb/ICmjmknQq4gdes0Ot9QgW
+A+Yfl+irR45ObBwXx1kGgd4YDYeh93pU9QweXj+Ezfw50mLQNgZXKYJMoJu2uX/2
+tdkZsQKBgQCTfDcW8qBntI6V+3Gh+sIThz+fjdv5+qT54heO4EHadc98ykEZX0M1
+sCWJiAQWM/zWXcsTndQDgDsvo23jpoulVPDitSEISp5gSe9FEN2njsVVID9h1OIM
+f30s5kwcJoiV9kUCya/BFtuS7kbuQfAyPU0v3I+lUey6VCW6A83OTg==
+-----END RSA PRIVATE KEY-----

@xen-orchestra/proxy/tests/localServer.mjs

@@ -0,0 +1,60 @@
+import { createServer as creatServerHttps } from 'https'
+import { createServer as creatServerHttp } from 'http'
+
+import { WebSocketServer } from 'ws'
+import fs from 'fs'
+
+const httpsServer = creatServerHttps({
+  key: fs.readFileSync('key.pem'),
+  cert: fs.readFileSync('cert.pem'),
+})
+const httpServer = creatServerHttp()
+
+const wss = new WebSocketServer({ noServer: true, perMessageDeflate: false })
+
+function upgrade(request, socket, head) {
+  const { pathname } = new URL(request.url)
+  // web socket server only on /foo url
+  if (pathname === '/foo') {
+    wss.handleUpgrade(request, socket, head, function done(ws) {
+      wss.emit('connection', ws, request)
+      ws.on('message', function message(data) {
+        ws.send(data)
+      })
+    })
+  } else {
+    socket.destroy()
+  }
+}
+
+function httpHandler(req, res) {
+  switch (req.url) {
+    case '/index.html':
+      res.end('hi')
+      return
+    case '/redirect':
+      res.writeHead(301, {
+        Location: 'index.html',
+      })
+      res.end()
+      return
+    case '/chainRedirect':
+      res.writeHead(301, {
+        Location: '/redirect',
+      })
+      res.end()
+      return
+    default:
+      res.writeHead(404)
+      res.end()
+  }
+}
+
+httpsServer.on('upgrade', upgrade)
+httpServer.on('upgrade', upgrade)
+
+httpsServer.on('request', httpHandler)
+httpServer.on('request', httpHandler)
+
+httpsServer.listen(8080)
+httpServer.listen(8081)

@xen-orchestra/proxy/tests/reverseProxy.unit.spec.mjs

@@ -0,0 +1,123 @@
+import ReverseProxy, { backendToLocalPath, localToBackendUrl } from '../dist/app/mixins/reverseProxy.mjs'
+import { deepEqual, strictEqual } from 'assert'
+
+function makeApp(reverseProxies) {
+  return {
+    config: {
+      get: () => reverseProxies,
+    },
+  }
+}
+
+const app = makeApp({
+  https: {
+    target: 'https://localhost:8080/remotePath/?baseParm=1#one=2&another=3',
+    oneOption: true,
+  },
+  http: 'http://localhost:8080/remotePath/?baseParm=1#one=2&another=3',
+})
+
+// test localToBackendUrl
+const expectedLocalToRemote = {
+  https: [
+    {
+      local: '/proxy/v1/https/',
+      remote: 'https://localhost:8080/remotePath/?baseParm=1#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub',
+      remote: 'https://localhost:8080/remotePath/sub?baseParm=1#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub/index.html',
+      remote: 'https://localhost:8080/remotePath/sub/index.html?baseParm=1#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub?param=1',
+      remote: 'https://localhost:8080/remotePath/sub?baseParm=1&param=1#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub?baseParm=willbeoverwritten&param=willstay',
+      remote: 'https://localhost:8080/remotePath/sub?baseParm=1&param=willstay#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub?param=1#another=willoverwrite',
+      remote: 'https://localhost:8080/remotePath/sub?baseParm=1&param=1#another=willoverwrite',
+    },
+  ],
+}
+const proxy = new ReverseProxy(app, { httpServer: { on: () => {} } })
+for (const proxyId in expectedLocalToRemote) {
+  for (const { local, remote } of expectedLocalToRemote[proxyId]) {
+    const config = proxy._getConfigFromRequest({ url: local })
+    const url = new URL(config.target)
+    strictEqual(localToBackendUrl(config.path, url, local).href, remote, 'error converting to backend')
+  }
+}
+
+// test backendToLocalPath
+const expectedRemoteToLocal = {
+  https: [
+    {
+      local: '/proxy/v1/https/',
+      remote: 'https://localhost:8080/remotePath/',
+    },
+    {
+      local: '/proxy/v1/https/sub/index.html',
+      remote: '/remotePath/sub/index.html',
+    },
+    {
+      local: '/proxy/v1/https/?baseParm=1#one=2&another=3',
+      remote: '?baseParm=1#one=2&another=3',
+    },
+    {
+      local: '/proxy/v1/https/sub?baseParm=1#one=2&another=3',
+      remote: 'https://localhost:8080/remotePath/sub?baseParm=1#one=2&another=3',
+    },
+  ],
+}
+
+for (const proxyId in expectedRemoteToLocal) {
+  for (const { local, remote } of expectedRemoteToLocal[proxyId]) {
+    const config = proxy._getConfigFromRequest({ url: local })
+    const targetUrl = new URL('https://localhost:8080/remotePath/?baseParm=1#one=2&another=3')
+    const remoteUrl = new URL(remote, targetUrl)
+    strictEqual(backendToLocalPath(config.path, targetUrl, remoteUrl), local, 'error converting to local')
+  }
+}
+
+// test _getConfigFromRequest
+
+const expectedConfig = [
+  {
+    local: '/proxy/v1/http/other',
+    config: {
+      target: 'http://localhost:8080/remotePath/?baseParm=1#one=2&another=3',
+      options: {},
+      path: '/proxy/v1/http',
+    },
+  },
+  {
+    local: '/proxy/v1/http',
+    config: undefined,
+  },
+
+  {
+    local: '/proxy/v1/other',
+    config: undefined,
+  },
+  {
+    local: '/proxy/v1/https/',
+    config: {
+      target: 'https://localhost:8080/remotePath/?baseParm=1#one=2&another=3',
+      options: {
+        oneOption: true,
+      },
+      path: '/proxy/v1/https',
+    },
+  },
+]
+
+for (const { local, config } of expectedConfig) {
+  deepEqual(proxy._getConfigFromRequest({ url: local }), config)
+}

@xen-orchestra/upload-ova/package.json

@@ -27,27 +27,25 @@
     "xo-upload-ova": "dist/index.js"
   },
   "engines": {
-    "node": ">=8.10"
+    "node": ">=10"
   },
   "dependencies": {
     "chalk": "^4.1.0",
     "exec-promise": "^0.7.0",
     "form-data": "^4.0.0",
-    "fs-extra": "^9.0.0",
-    "fs-promise": "^2.0.3",
+    "fs-extra": "^10.0.0",
     "get-stream": "^6.0.0",
-    "http-request-plus": "^0.10.0",
+    "http-request-plus": "^0.13.0",
     "human-format": "^0.11.0",
     "l33teral": "^3.0.3",
     "lodash": "^4.17.4",
-    "nice-pipe": "0.0.0",
     "pretty-ms": "^7.0.0",
     "progress-stream": "^2.0.0",
     "pw": "^0.0.4",
     "strip-indent": "^3.0.0",
     "xdg-basedir": "^4.0.0",
-    "xo-lib": "^0.10.1",
-    "xo-vmdk-to-vhd": "^2.0.0"
+    "xo-lib": "^0.11.1",
+    "xo-vmdk-to-vhd": "^2.0.3"
   },
   "devDependencies": {
     "@babel/cli": "^7.0.0",

@xen-orchestra/upload-ova/src/index.js

@@ -6,7 +6,7 @@ import chalk from 'chalk'
 import execPromise from 'exec-promise'
 import FormData from 'form-data'
 import { createReadStream } from 'fs'
-import { stat } from 'fs-promise'
+import { stat } from 'fs-extra'
 import getStream from 'get-stream'
 import hrp from 'http-request-plus'
 import humanFormat from 'human-format'
@@ -14,7 +14,6 @@ import l33t from 'l33teral'
 import isObject from 'lodash/isObject'
 import getKeys from 'lodash/keys'
 import startsWith from 'lodash/startsWith'
-import nicePipe from 'nice-pipe'
 import prettyMs from 'pretty-ms'
 import progressStream from 'progress-stream'
 import pw from 'pw'
@@ -22,10 +21,13 @@ import stripIndent from 'strip-indent'
 import { URL } from 'url'
 import Xo from 'xo-lib'
 import { parseOVAFile } from 'xo-vmdk-to-vhd'
+import { pipeline } from 'stream'
 
 import pkg from '../package'
 import { load as loadConfig, set as setConfig, unset as unsetConfig } from './config'
 
+const noop = Function.prototype
+
 function help() {
   return stripIndent(
     `
@@ -206,7 +208,7 @@ export async function upload(args) {
       url = new URL(result[key], baseUrl)
 
       const { size: length } = await stat(file)
-      const input = nicePipe([
+      const input = pipeline(
         createReadStream(file),
         progressStream(
           {
@@ -215,7 +217,8 @@ export async function upload(args) {
           },
           printProgress
         ),
-      ])
+        noop
+      )
       formData.append('file', input, { filename: 'file', knownLength: length })
       try {
         return await hrp.post(url.toString(), { body: formData, headers: formData.getHeaders() }).readAll('utf-8')

@xen-orchestra/xapi/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xen-orchestra/xapi",
-  "version": "0.6.2",
+  "version": "0.8.5",
   "homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/xapi",
   "bugs": "https://github.com/vatesfr/xen-orchestra/issues",
   "repository": {
@@ -25,7 +25,7 @@
     "xo-common": "^0.7.0"
   },
   "peerDependencies": {
-    "xen-api": "^0.32.0"
+    "xen-api": "^0.35.1"
   },
   "scripts": {
     "build": "cross-env NODE_ENV=production babel --source-maps --out-dir=dist/ src/",
@@ -38,13 +38,13 @@
     "prepublishOnly": "yarn run build"
   },
   "dependencies": {
-    "@vates/decorate-with": "^0.0.1",
+    "@vates/decorate-with": "^1.0.0",
     "@xen-orchestra/async-map": "^0.1.2",
-    "@xen-orchestra/log": "^0.2.0",
+    "@xen-orchestra/log": "^0.3.0",
     "d3-time-format": "^3.0.0",
     "golike-defer": "^0.5.1",
     "lodash": "^4.17.15",
-    "promise-toolbox": "^0.19.2"
+    "promise-toolbox": "^0.20.0"
   },
   "private": false,
   "license": "AGPL-3.0-or-later",

@xen-orchestra/xapi/src/index.js

@@ -15,16 +15,18 @@ exports.VDI_FORMAT_VHD = 'vhd'
 // xapi.call('host.get_servertime', host.$ref) for example
 exports.formatDateTime = utcFormat('%Y%m%dT%H:%M:%SZ')
 
-const parseDateTimeHelper = utcParse('%Y%m%dT%H:%M:%SZ')
+const dateTimeParsers = ['%Y%m%dT%H:%M:%SZ', '%Y-%m-%dT%H:%M:%S.%LZ'].map(utcParse)
 exports.parseDateTime = function (str, defaultValue) {
-  const date = parseDateTimeHelper(str)
-  if (date === null) {
-    if (arguments.length > 1) {
-      return defaultValue
+  for (const parser of dateTimeParsers) {
+    const date = parser(str)
+    if (date !== null) {
+      return date.getTime()
     }
-    throw new RangeError(`unable to parse XAPI datetime ${JSON.stringify(str)}`)
   }
-  return date.getTime()
+  if (arguments.length > 1) {
+    return defaultValue
+  }
+  throw new RangeError(`unable to parse XAPI datetime ${JSON.stringify(str)}`)
 }
 
 const hasProps = o => {
@@ -98,18 +100,16 @@ function removeWatcher(predicate, cb) {
 
 class Xapi extends Base {
   constructor({
-    callRetryWhenTooManyPendingTasks,
+    callRetryWhenTooManyPendingTasks = { delay: 5e3, tries: 10 },
     ignoreNobakVdis,
     maxUncoalescedVdis,
-    vdiDestroyRetryWhenInUse,
+    vdiDestroyRetryWhenInUse = { delay: 5e3, tries: 10 },
     ...opts
   }) {
     assert.notStrictEqual(ignoreNobakVdis, undefined)
 
     super(opts)
     this._callRetryWhenTooManyPendingTasks = {
-      delay: 5e3,
-      tries: 10,
       ...callRetryWhenTooManyPendingTasks,
       onRetry,
       when: { code: 'TOO_MANY_PENDING_TASKS' },
@@ -117,8 +117,6 @@ class Xapi extends Base {
     this._ignoreNobakVdis = ignoreNobakVdis
     this._maxUncoalescedVdis = maxUncoalescedVdis
     this._vdiDestroyRetryWhenInUse = {
-      delay: 5e3,
-      retries: 10,
       ...vdiDestroyRetryWhenInUse,
       onRetry,
       when: { code: 'VDI_IN_USE' },

@xen-orchestra/xapi/src/index.spec.js

@@ -0,0 +1,17 @@
+/* eslint-env jest */
+
+const { parseDateTime } = require('./')
+
+describe('parseDateTime()', () => {
+  it('parses legacy XAPI format', () => {
+    expect(parseDateTime('20220106T21:25:15Z')).toBe(1641504315000)
+  })
+
+  it('parses ISO 8601 with millisconds format', () => {
+    expect(parseDateTime('2022-01-06T17:32:35.000Z')).toBe(1641490355000)
+  })
+
+  it('throws when value cannot be parsed', () => {
+    expect(() => parseDateTime('foo bar')).toThrow('unable to parse XAPI datetime "foo bar"')
+  })
+})

@xen-orchestra/xapi/src/vdi.js

@@ -1,6 +1,7 @@
 const CancelToken = require('promise-toolbox/CancelToken.js')
 const pCatch = require('promise-toolbox/catch.js')
 const pRetry = require('promise-toolbox/retry.js')
+const { decorateWith } = require('@vates/decorate-with')
 
 const extractOpaqueRef = require('./_extractOpaqueRef.js')
 
@@ -11,10 +12,13 @@ module.exports = class Vdi {
     return extractOpaqueRef(await this.callAsync('VDI.clone', vdiRef))
   }
 
+  // work around a race condition in XCP-ng/XenServer where the disk is not fully unmounted yet
+  @decorateWith(pRetry.wrap, function () {
+    return this._vdiDestroyRetryWhenInUse
+  })
   async destroy(vdiRef) {
     await pCatch.call(
-      // work around a race condition in XCP-ng/XenServer where the disk is not fully unmounted yet
-      pRetry(() => this.callAsync('VDI.destroy', vdiRef), this._vdiDestroyRetryWhenInUse),
+      this.callAsync('VDI.destroy', vdiRef),
       // if this VDI is not found, consider it destroyed
       { code: 'HANDLE_INVALID' },
       noop

@xen-orchestra/xapi/src/vm.js

@@ -60,7 +60,7 @@ module.exports = class Vm {
       try {
         vdi = await this[vdiRefOrUuid.startsWith('OpaqueRef:') ? 'getRecord' : 'getRecordByUuid']('VDI', vdiRefOrUuid)
       } catch (error) {
-        warn(error)
+        warn('_assertHealthyVdiChain, could not fetch VDI', { error })
         return
       }
       cache[vdi.$ref] = vdi
@@ -81,7 +81,7 @@ module.exports = class Vm {
                   try {
                     vdi = await this.getRecord('VDI', vdiRef)
                   } catch (error) {
-                    warn(error)
+                    warn('_assertHealthyVdiChain, could not fetch VDI', { error })
                     return
                   }
                   cache[vdiRef] = vdi
@@ -99,6 +99,7 @@ module.exports = class Vm {
       // should coalesce
       const children = childrenMap[vdi.uuid]
       if (
+        children !== undefined && // unused unmanaged VDI, will be GC-ed
         children.length === 1 &&
         !children[0].managed && // some SRs do not coalesce the leaf
         tolerance-- <= 0
@@ -166,7 +167,7 @@ module.exports = class Vm {
       memory_static_min,
       name_description,
       name_label,
-      // NVRAM, // experimental
+      NVRAM,
       order,
       other_config = {},
       PCI_bus = '',
@@ -255,6 +256,7 @@ module.exports = class Vm {
       is_vmss_snapshot,
       name_description,
       name_label,
+      NVRAM,
       order,
       reference_label,
       shutdown_delay,

CHANGELOG.md

@@ -1,15 +1,364 @@
 # ChangeLog
 
-## **next**
+## **5.66.2** (2022-01-05)
+
+<img id="latest" src="https://badgen.net/badge/channel/latest/yellow" alt="Channel: latest" />
+
+### Bug fixes
+
+- [Import/Disk] Fix `JSON.parse` and `createReadableSparseStream is not a function` errors [#6068](https://github.com/vatesfr/xen-orchestra/issues/6068)
+- [Backup] Fix delta backup are almost always full backup instead of differentials [Forum#5256](https://xcp-ng.org/forum/topic/5256/s3-backup-try-it/69) [Forum#5371](https://xcp-ng.org/forum/topic/5371/delta-backup-changes-in-5-66) (PR [#6075](https://github.com/vatesfr/xen-orchestra/pull/6075))
 
 ### Released packages
 
-- xen-api 0.32
+- vhd-lib 3.0.0
+- xo-vmdk-to-vhd 2.0.3
+- @xen-orchestra/backups 0.18.3
+- @xen-orchestra/proxy 0.17.3
+- xo-server 5.86.3
+- xo-web 5.91.2
+
+## **5.66.1** (2021-12-23)
+
+### Bug fixes
+
+- [Dashboard/Health] Fix `error has occured` when a pool has no default SR
+- [Delta Backup] Fix unnecessary full backup when not using S3 [Forum #5371](https://xcp-ng.org/forum/topic/5371/delta-backup-changes-in-5-66)d (PR [#6070](https://github.com/vatesfr/xen-orchestra/pull/6070))
+- [Backup] Fix incorrect warnings `incorrect size [...] instead of undefined`
+
+### Released packages
+
+- @xen-orchestra/backups 0.18.2
+- @xen-orchestra/proxy 0.17.2
+- xo-server 5.86.2
+- xo-web 5.91.1
+
+## **5.66.0** (2021-12-21)
+
+### Enhancements
+
+- [About] Show commit instead of version numbers for source users (PR [#6045](https://github.com/vatesfr/xen-orchestra/pull/6045))
+- [Health] Display default SRs that aren't shared [#5871](https://github.com/vatesfr/xen-orchestra/issues/5871) (PR [#6033](https://github.com/vatesfr/xen-orchestra/pull/6033))
+- [Pool,VM/advanced] Ability to change the suspend SR [#4163](https://github.com/vatesfr/xen-orchestra/issues/4163) (PR [#6044](https://github.com/vatesfr/xen-orchestra/pull/6044))
+- [Home/VMs/Backup filter] Filter out VMs in disabled backup jobs (PR [#6037](https://github.com/vatesfr/xen-orchestra/pull/6037))
+- [Rolling Pool Update] Automatically disable High Availability during the update [#5711](https://github.com/vatesfr/xen-orchestra/issues/5711) (PR [#6057](https://github.com/vatesfr/xen-orchestra/pull/6057))
+- [Delta Backup on S3] Compress blocks by default ([Brotli](https://en.wikipedia.org/wiki/Brotli)) which reduces remote usage and increase backup speed (PR [#5932](https://github.com/vatesfr/xen-orchestra/pull/5932))
+
+### Bug fixes
+
+- [Tables/actions] Fix collapsed actions being clickable despite being disabled (PR [#6023](https://github.com/vatesfr/xen-orchestra/pull/6023))
+- [Backup] Remove incorrect size warning following a merge [Forum #5727](https://xcp-ng.org/forum/topic/4769/warnings-showing-in-system-logs-following-each-backup-job/4) (PR [#6010](https://github.com/vatesfr/xen-orchestra/pull/6010))
+- [Delta Backup] Preserve UEFI boot parameters [#6054](https://github.com/vatesfr/xen-orchestra/issues/6054) [Forum #5319](https://xcp-ng.org/forum/topic/5319/bug-uefi-boot-parameters-not-preserved-with-delta-backups)
+
+### Released packages
+
+- @xen-orchestra/mixins 0.1.2
+- @xen-orchestra/xapi 0.8.5
+- vhd-lib 2.1.0
+- xo-vmdk-to-vhd 2.0.2
+- @xen-orchestra/backups 0.18.1
+- @xen-orchestra/proxy 0.17.1
+- xo-server 5.86.1
+- xo-web 5.91.0
+
+## **5.65.3** (2021-12-20)
+
+<img id="stable" src="https://badgen.net/badge/channel/stable/green" alt="Channel: stable" />
+
+### Bug fixes
+
+- [Continuous Replication] Fix `could not find the base VM`
+- [Backup/Smart mode] Always ignore replicated VMs created by the current job
+- [Backup] Fix `Unexpected end of JSON input` during merge step
+- [Backup] Fix stuck jobs when using S3 remotes (PR [#6067](https://github.com/vatesfr/xen-orchestra/pull/6067))
+
+### Released packages
+
+- @xen-orchestra/fs 0.19.3
+- vhd-lib 2.0.4
+- @xen-orchestra/backups 0.17.1
+- xo-server 5.85.1
+
+## **5.65.2** (2021-12-10)
+
+### Bug fixes
+
+- [Backup] Fix `handler.rmTree` is not a function (Forum [5256](https://xcp-ng.org/forum/topic/5256/s3-backup-try-it/29) PR [#6041](https://github.com/vatesfr/xen-orchestra/pull/6041) )
+- [Backup] Fix `EEXIST` in logs when multiple merge tasks are created at the same time ([Forum #5301](https://xcp-ng.org/forum/topic/5301/warnings-errors-in-journalctl))
+- [Backup] Fix missing backup on restore (Forum [5256](https://xcp-ng.org/forum/topic/5256/s3-backup-try-it/29) (PR [#6048](https://github.com/vatesfr/xen-orchestra/pull/6048))
+
+### Released packages
+
+- @xen-orchestra/fs 0.19.2
+- vhd-lib 2.0.3
+- @xen-orchestra/backups 0.16.2
+- xo-server 5.84.3
+- @xen-orchestra/proxy 0.15.5
+
+## **5.65.1** (2021-12-03)
+
+### Bug fixes
+
+- [Delta Backup Restoration] Fix assertion error [Forum #5257](https://xcp-ng.org/forum/topic/5257/problems-building-from-source/16)
+- [Delta Backup Restoration] `TypeError: this disposable has already been disposed` [Forum #5257](https://xcp-ng.org/forum/topic/5257/problems-building-from-source/20)
+- [Backups] Fix: `Error: Chaining alias is forbidden xo-vm-backups/..alias.vhd to xo-vm-backups/....alias.vhd` when backuping a file to s3 [Forum #5226](https://xcp-ng.org/forum/topic/5256/s3-backup-try-it)
+- [Delta Backup Restoration] `VDI_IO_ERROR(Device I/O errors)` [Forum #5727](https://xcp-ng.org/forum/topic/5257/problems-building-from-source/4) (PR [#6031](https://github.com/vatesfr/xen-orchestra/pull/6031))
+- [Delta Backup] Fix `Cannot read property 'uuid' of undefined` when a VDI has been removed from a backed up VM (PR [#6034](https://github.com/vatesfr/xen-orchestra/pull/6034))
+
+### Released packages
+
+- @vates/compose 2.1.0
+- vhd-lib 2.0.2
+- xo-vmdk-to-vhd 2.0.1
+- @xen-orchestra/backups 0.16.1
+- @xen-orchestra/proxy 0.15.4
+- xo-server 5.84.2
+
+## **5.65.0** (2021-11-30)
+
+### Highlights
+
+- [VM] Ability to export a snapshot's memory (PR [#6015](https://github.com/vatesfr/xen-orchestra/pull/6015))
+- [Cloud config] Ability to create a network cloud config template and reuse it in the VM creation [#5931](https://github.com/vatesfr/xen-orchestra/issues/5931) (PR [#5979](https://github.com/vatesfr/xen-orchestra/pull/5979))
+- [Backup/logs] identify XAPI errors (PR [#6001](https://github.com/vatesfr/xen-orchestra/pull/6001))
+- [lite] Highlight selected VM (PR [#5939](https://github.com/vatesfr/xen-orchestra/pull/5939))
+
+### Enhancements
+
+- [S3] Ability to authorize self signed certificates for S3 remote (PR [#5961](https://github.com/vatesfr/xen-orchestra/pull/5961))
+
+### Bug fixes
+
+- [Import/VM] Fix the import of OVA files (PR [#5976](https://github.com/vatesfr/xen-orchestra/pull/5976))
+
+### Released packages
+
+- @vates/async-each 0.1.0
+- xo-remote-parser 0.8.4
+- @xen-orchestra/fs 0.19.0
+- @xen-orchestra/xapi patch
+- vhd-lib 2.0.1
+- @xen-orchestra/backups 0.16.0
+- xo-lib 0.11.1
+- @xen-orchestra/proxy 0.15.3
+- xo-server 5.84.1
+- vhd-cli 0.6.0
+- xo-web 5.90.0
+
+## **5.64.0** (2021-10-29)
+
+## Highlights
+
+- [Netbox] Support older versions of Netbox and prevent "active is not a valid choice" error [#5898](https://github.com/vatesfr/xen-orchestra/issues/5898) (PR [#5946](https://github.com/vatesfr/xen-orchestra/pull/5946))
+- [Tasks] Filter out short tasks using a default filter (PR [#5921](https://github.com/vatesfr/xen-orchestra/pull/5921))
+- [Host] Handle evacuation failure during host shutdown (PR [#5966](https://github.com/vatesfr/xen-orchestra/pull/#5966))
+- [Menu] Notify user when proxies need to be upgraded (PR [#5930](https://github.com/vatesfr/xen-orchestra/pull/5930))
+- [Servers] Ability to use an HTTP proxy between XO and a server (PR [#5958](https://github.com/vatesfr/xen-orchestra/pull/5958))
+- [VM/export] Ability to copy the export URL (PR [#5948](https://github.com/vatesfr/xen-orchestra/pull/5948))
+- [Pool/advanced] Ability to define network for importing/exporting VMs/VDIs (PR [#5957](https://github.com/vatesfr/xen-orchestra/pull/5957))
+- [Host/advanced] Add button to enable/disable the host (PR [#5952](https://github.com/vatesfr/xen-orchestra/pull/5952))
+- [Backups] Enable merge worker by default
+
+### Enhancements
+
+- [Jobs] Ability to copy a job ID (PR [#5951](https://github.com/vatesfr/xen-orchestra/pull/5951))
+
+### Bug fixes
+
+- [Backups] Delete unused snapshots related to other schedules (even no longer existing) (PR [#5949](https://github.com/vatesfr/xen-orchestra/pull/5949))
+- [Jobs] Fix `job.runSequence` method (PR [#5944](https://github.com/vatesfr/xen-orchestra/pull/5944))
+- [Netbox] Fix error when testing plugin on versions older than 2.10 (PR [#5963](https://github.com/vatesfr/xen-orchestra/pull/5963))
+- [Snapshot] Fix "Create VM from snapshot" creating a template instead of a VM (PR [#5955](https://github.com/vatesfr/xen-orchestra/pull/5955))
+- [Host/Logs] Improve the display of log content (PR [#5943](https://github.com/vatesfr/xen-orchestra/pull/5943))
+- [XOA licenses] Fix expiration date displaying "Invalid date" in some rare cases (PR [#5967](https://github.com/vatesfr/xen-orchestra/pull/5967))
+- [API/pool.listPoolsMatchingCriteria] Fix `checkSrName`/`checkPoolName` `is not a function` error
+
+### Released packages
+
+- xo-server-netbox 0.3.3
+- vhd-lib 1.3.0
+- xen-api 0.35.1
+- @xen-orchestra/xapi 0.8.0
+- @xen-orchestra/backups 0.15.1
+- @xen-orchestra/proxy 0.15.2
+- vhd-cli 0.5.0
+- xapi-explore-sr 0.4.0
+- xo-server 5.83.0
+- xo-web 5.89.0
+
+## **5.63.0** (2021-09-30)
+
+### Highlights
+
+- [Backup] Go back to previous page instead of going to the overview after editing a job: keeps current filters and page (PR [#5913](https://github.com/vatesfr/xen-orchestra/pull/5913))
+- [Health] Do not take into consideration duplicated MAC addresses from CR VMs (PR [#5916](https://github.com/vatesfr/xen-orchestra/pull/5916))
+- [Health] Ability to filter duplicated MAC addresses by running VMs (PR [#5917](https://github.com/vatesfr/xen-orchestra/pull/5917))
+- [Tables] Move the search bar and pagination to the top of the table (PR [#5914](https://github.com/vatesfr/xen-orchestra/pull/5914))
+- [Netbox] Handle nested prefixes by always assigning an IP to the smallest prefix it matches (PR [#5908](https://github.com/vatesfr/xen-orchestra/pull/5908))
+
+### Bug fixes
+
+- [SSH keys] Allow SSH key to be broken anywhere to avoid breaking page formatting (Thanks [@tstivers1990](https://github.com/tstivers1990)!) [#5891](https://github.com/vatesfr/xen-orchestra/issues/5891) (PR [#5892](https://github.com/vatesfr/xen-orchestra/pull/5892))
+- [Netbox] Better handling and error messages when encountering issues due to UUID custom field not being configured correctly [#5905](https://github.com/vatesfr/xen-orchestra/issues/5905) [#5806](https://github.com/vatesfr/xen-orchestra/issues/5806) [#5834](https://github.com/vatesfr/xen-orchestra/issues/5834) (PR [#5909](https://github.com/vatesfr/xen-orchestra/pull/5909))
+- [New VM] Don't send network config if untouched as all commented config can make Cloud-init fail [#5918](https://github.com/vatesfr/xen-orchestra/issues/5918) (PR [#5923](https://github.com/vatesfr/xen-orchestra/pull/5923))
+
+### Released packages
+
+- xen-api 0.34.3
+- vhd-lib 1.2.0
+- xo-server-netbox 0.3.1
+- @xen-orchestra/proxy 0.14.7
+- xo-server 5.82.3
+- xo-web 5.88.0
+
+## **5.62.1** (2021-09-17)
+
+### Bug fixes
+
+- [VM/Advanced] Fix conversion from UEFI to BIOS boot firmware (PR [#5895](https://github.com/vatesfr/xen-orchestra/pull/5895))
+- [VM/network] Support newline-delimited IP addresses reported by some guest tools
+- Fix VM/host stats, VM creation with Cloud-init, and VM backups, with NATted hosts [#5896](https://github.com/vatesfr/xen-orchestra/issues/5896)
+- [VM/import] Very small VMDK and OVA files were mangled upon import (PR [#5903](https://github.com/vatesfr/xen-orchestra/pull/5903))
+
+### Released packages
+
+- xen-api 0.34.2
+- @xen-orchestra/proxy 0.14.6
+- xo-server 5.82.2
+
+## **5.62.0** (2021-08-31)
+
+### Highlights
+
+- [Host] Add warning in case of unmaintained host version [#5840](https://github.com/vatesfr/xen-orchestra/issues/5840) (PR [#5847](https://github.com/vatesfr/xen-orchestra/pull/5847))
+- [Backup] Use default migration network if set when importing/exporting VMs/VDIs (PR [#5883](https://github.com/vatesfr/xen-orchestra/pull/5883))
+
+### Enhancements
+
+- [New network] Ability for pool's admin to create a new network within the pool (PR [#5873](https://github.com/vatesfr/xen-orchestra/pull/5873))
+- [Netbox] Synchronize primary IPv4 and IPv6 addresses [#5633](https://github.com/vatesfr/xen-orchestra/issues/5633) (PR [#5879](https://github.com/vatesfr/xen-orchestra/pull/5879))
+
+### Bug fixes
+
+- [VM/network] Fix an issue where multiple IPs would be displayed in the same tag when using old Xen tools. This also fixes Netbox's IP synchronization for the affected VMs. (PR [#5860](https://github.com/vatesfr/xen-orchestra/pull/5860))
+- [LDAP] Handle groups with no members (PR [#5862](https://github.com/vatesfr/xen-orchestra/pull/5862))
+- Fix empty button on small size screen (PR [#5874](https://github.com/vatesfr/xen-orchestra/pull/5874))
+- [Host] Fix `Cannot read property 'other_config' of undefined` error when enabling maintenance mode (PR [#5875](https://github.com/vatesfr/xen-orchestra/pull/5875))
+
+### Released packages
+
+- xen-api 0.34.1
+- @xen-orchestra/xapi 0.7.0
+- @xen-orchestra/backups 0.13.0
+- @xen-orchestra/fs 0.18.0
+- @xen-orchestra/log 0.3.0
+- @xen-orchestra/mixins 0.1.1
+- xo-server-auth-ldap 0.10.4
+- xo-server-netbox 0.3.0
+- xo-server 5.82.1
+- xo-web 5.87.0
+
+## **5.61.0** (2021-07-30)
+
+### Highlights
+
+- [SR/disks] Display base copies' active VDIs (PR [#5826](https://github.com/vatesfr/xen-orchestra/pull/5826))
+- [Netbox] Optionally allow self-signed certificates (PR [#5850](https://github.com/vatesfr/xen-orchestra/pull/5850))
+- [Host] When supported, use pool's default migration network to evacuate host [#5802](https://github.com/vatesfr/xen-orchestra/issues/5802) (PR [#5851](https://github.com/vatesfr/xen-orchestra/pull/5851))
+- [VM] shutdown/reboot: offer to force shutdown/reboot the VM if no Xen tools were detected [#5838](https://github.com/vatesfr/xen-orchestra/issues/5838) (PR [#5855](https://github.com/vatesfr/xen-orchestra/pull/5855))
+
+### Enhancements
+
+- [Netbox] Add information about a failed request to the error log to help better understand what happened [#5834](https://github.com/vatesfr/xen-orchestra/issues/5834) (PR [#5842](https://github.com/vatesfr/xen-orchestra/pull/5842))
+- [VM/console] Ability to rescan ISO SRs (PR [#5841](https://github.com/vatesfr/xen-orchestra/pull/5841))
+
+### Bug fixes
+
+- [VM/disks] Fix `an error has occured` when self service user was on VM disk view (PR [#5841](https://github.com/vatesfr/xen-orchestra/pull/5841))
+- [Backup] Protect replicated VMs from being started on specific hosts (PR [#5852](https://github.com/vatesfr/xen-orchestra/pull/5852))
+
+### Released packages
+
+- @xen-orchestra/backups 0.12.2
+- @xen-orchestra/proxy 0.14.4
+- xo-server-netbox 0.2.0
+- xo-web 5.86.0
+- xo-server 5.81.2
+
+## **5.60.0** (2021-06-30)
+
+### Highlights
+
+- [VM/disks] Ability to rescan ISO SRs (PR [#5814](https://github.com/vatesfr/xen-orchestra/pull/5814))
+- [VM/snapshots] Identify VM's current snapshot with an icon next to the snapshot's name (PR [#5824](https://github.com/vatesfr/xen-orchestra/pull/5824))
+
+### Enhancements
+
+- [OVA import] improve OVA import error reporting (PR [#5797](https://github.com/vatesfr/xen-orchestra/pull/5797))
+- [Backup] Distinguish error messages between cancelation and interrupted HTTP connection
+- [Jobs] Add `host.emergencyShutdownHost` to the list of methods that jobs can call (PR [#5818](https://github.com/vatesfr/xen-orchestra/pull/5818))
+- [Host/Load-balancer] Log VM and host names when a VM is migrated + category (density, performance, ...) (PR [#5808](https://github.com/vatesfr/xen-orchestra/pull/5808))
+- [VM/new disk] Auto-fill disk name input with generated unique name (PR [#5828](https://github.com/vatesfr/xen-orchestra/pull/5828))
+
+### Bug fixes
+
+- [IPs] Handle space-delimited IP address format provided by outdated guest tools [5801](https://github.com/vatesfr/xen-orchestra/issues/5801) (PR [5805](https://github.com/vatesfr/xen-orchestra/pull/5805))
+- [API/pool.listPoolsMatchingCriteria] fix `unknown error from the peer` error (PR [5807](https://github.com/vatesfr/xen-orchestra/pull/5807))
+- [Backup] Limit number of connections to hosts, which should reduce the occurences of `ECONNRESET`
+- [Plugins/perf-alert] All mode: only selects running hosts and VMs (PR [5811](https://github.com/vatesfr/xen-orchestra/pull/5811))
+- [New VM] Fix summary section always showing "0 B" for RAM (PR [#5817](https://github.com/vatesfr/xen-orchestra/pull/5817))
+- [Backup/Restore] Fix _start VM after restore_ [5820](https://github.com/vatesfr/xen-orchestra/issues/5820)
+- [Netbox] Fix a bug where some devices' IPs would get deleted from Netbox (PR [#5821](https://github.com/vatesfr/xen-orchestra/pull/5821))
+- [Netbox] Fix an issue where some IPv6 would be deleted just to be immediately created again (PR [#5822](https://github.com/vatesfr/xen-orchestra/pull/5822))
+
+### Released packages
+
+- @vates/decorate-with 0.1.0
+- xen-api 0.33.1
+- @xen-orchestra/xapi 0.6.4
+- @xen-orchestra/backups 0.12.0
+- @xen-orchestra/proxy 0.14.3
+- vhd-lib 1.1.0
+- vhd-cli 0.4.0
+- xo-server-netbox 0.1.2
+- xo-server-perf-alert 0.3.2
+- xo-server-load-balancer 0.7.0
+- xo-server 5.80.0
+- xo-web 5.84.0
+
+## **5.59.0** (2021-05-31)
+
+### Highlights
+
+- [Smart backup] Report missing pools [#2844](https://github.com/vatesfr/xen-orchestra/issues/2844) (PR [#5768](https://github.com/vatesfr/xen-orchestra/pull/5768))
+- [Metadata Backup] Add a warning on restoring a metadata backup (PR [#5769](https://github.com/vatesfr/xen-orchestra/pull/5769))
+- [Netbox][plugin](https://xen-orchestra.com/docs/advanced.html#netbox) to synchronize pools, VMs and IPs with [Netbox](https://netbox.readthedocs.io/en/stable/) (PR [#5783](https://github.com/vatesfr/xen-orchestra/pull/5783))
+
+### Enhancements
+
+- [SAML] Compatible with users created with other authentication providers (PR [#5781](https://github.com/vatesfr/xen-orchestra/pull/5781))
+
+### Bug fixes
+
+- [SDN Controller] Private network creation failure when the tunnels were created on different devices [Forum #4620](https://xcp-ng.org/forum/topic/4620/no-pif-found-in-center) (PR [#5793](https://github.com/vatesfr/xen-orchestra/pull/5793))
+
+### Released packages
+
+- @xen-orchestra/emit-async 0.1.0
+- @xen-orchestra/defined 0.0.1
+- xo-collection 0.5.0
+- @xen-orchestra/log 0.2.1
+- xen-api 0.33.0
+- @xen-orchestra/xapi 0.6.3
+- xo-server-auth-saml 0.9.0
+- xo-server-backup-reports 0.16.10
+- xo-server-netbox 0.1.1
+- xo-server-sdn-controller 1.0.5
+- xo-web 5.82.0
+- xo-server 5.79.5
 
 ## **5.58.1** (2021-05-06)
 
-<img id="latest" src="https://badgen.net/badge/channel/latest/yellow" alt="Channel: latest" />
-
 ### Bug fixes
 
 - [Backups] Better handling of errors in remotes, fix `task has already ended`
@@ -64,8 +413,6 @@
 
 ## **5.57.1** (2021-04-13)
 
-<img id="stable" src="https://badgen.net/badge/channel/stable/green" alt="Channel: stable" />
-
 ### Enhancements
 
 - [Host/Load-balancer] Add option to disable migration (PR [#5706](https://github.com/vatesfr/xen-orchestra/pull/5706))
@@ -112,7 +459,7 @@
 - [Proxy] _Redeploy_ now works when the bound VM is missing
 - [VM template] Fix confirmation modal doesn't appear on deleting a default template (PR [#5644](https://github.com/vatesfr/xen-orchestra/pull/5644))
 - [OVA VM Import] Fix imported VMs all having the same MAC addresses
-- [Disk import] Fix `an error has occurred` when importing wrong format or corrupted files [#5663](https://github.com/vatesfr/xen-orchestra/issues/5663) (PR [#5683](https://github.com/vatesfr/xen-orchestra/pull/5683))
+- [Disk import] Fix `an error has occurred` when importing wrong format or corrupted files [#5663](https://github.com/vatesfr/xen-orchestra/issues/5663) (PR [#5683](https://github.com/vatesfr/xen-orchestra/pull/5683))
 
 ### Released packages
 

CHANGELOG.unreleased.md

@@ -7,28 +7,22 @@
 
 > Users must be able to say: “Nice enhancement, I'm eager to test it”
 
-- [Metadata Backup] Add a warning on restoring a metadata backup (PR [#5769](https://github.com/vatesfr/xen-orchestra/pull/5769))
-- [SAML] Compatible with users created with other authentication providers (PR [#5781](https://github.com/vatesfr/xen-orchestra/pull/5781))
-- [Netbox] [Plugin](https://xen-orchestra.com/docs/advanced.html#netbox) to synchronize pools, VMs and IPs with [Netbox](https://netbox.readthedocs.io/en/stable/) (PR [#5783](https://github.com/vatesfr/xen-orchestra/pull/5783))
+- Limit number of concurrent VM migrations per pool to `3` [#6065](https://github.com/vatesfr/xen-orchestra/issues/6065) (PR [#6076](https://github.com/vatesfr/xen-orchestra/pull/6076))
+  Can be changed in `xo-server`'s configuration file: `xapiOptions.vmMigrationConcurrency`
+- [Proxy] Now ships a reverse proxy [PR#6072](https://github.com/vatesfr/xen-orchestra/pull/6072)
 
 ### Bug fixes
 
 > Users must be able to say: “I had this issue, happy to know it's fixed”
 
-- [Smart backup] Report missing pools [#2844](https://github.com/vatesfr/xen-orchestra/issues/2844) (PR [#5768](https://github.com/vatesfr/xen-orchestra/pull/5768))
+- [Backup] Detect and clear orphan merge states, fix `ENOENT` errors (PR [#6087](https://github.com/vatesfr/xen-orchestra/pull/6087))
 
 ### Packages to release
 
 > Packages will be released in the order they are here, therefore, they should
 > be listed by inverse order of dependency.
 >
-> Global order:
->
-> - @vates/...
-> - @xen-orchestra/...
-> - xo-server-...
-> - xo-server
-> - xo-web
+> Rule of thumb: add packages on top.
 >
 > The format is the following: - `$packageName` `$version`
 >
@@ -40,13 +34,7 @@
 >
 > In case of conflict, the highest (lowest in previous list) `$version` wins.
 
-- @xen-orchestra/emit-async minor
-- @xen-orchestra/defined patch
-- xo-collection minor
-- @xen-orchestra/log patch
-- xen-api minor
-- xo-server-auth-saml minor
-- xo-server-backup-reports patch
-- xo-server-netbox minor
-- xo-web minor
-- xo-server patch
+- @xen-orchestra/backups minor
+- @xen-orchestra/backups-cli minor
+- @xen-orchestra/proxy minor
+- xo-server minor

ISSUE_TEMPLATE.md

@@ -1,27 +0,0 @@
-<!--
-Welcome to the issue section of Xen Orchestra!
-
-Here you can:
-- report an issue
-- propose an enhancement
-- ask a question
-
-Please, respect this template as much as possible, it helps us sort
-the issues :)
--->
-
-### Context
-
-- **XO origin**: the sources / XO Appliance
-- **Versions**:
-  - Node: **FILL HERE**
-  - xo-web: **FILL HERE**
-  - xo-server: **FILL HERE**
-
-### Expected behavior
-
-<!-- What you expect to happen -->
-
-### Current behavior
-
-<!-- What is actually happening -->

docs/advanced.md

@@ -114,17 +114,18 @@ We need your feedback on this feature!
 
 The plugin "web-hooks" needs to be installed and loaded for this feature to work.
 
-You can trigger an HTTP POST request to a URL when a Xen Orchestra API method is called.
+You can trigger an HTTP POST request to a URL when a Xen Orchestra API method is called or when a backup job runs.
 
 - Go to Settings > Plugins > Web hooks
 - Add new hooks
 - For each hook, configure:
-  - Method: the XO API method that will trigger the HTTP request when called
+  - Method: the XO API method that will trigger the HTTP request when called. For backup jobs, choose `backupNg.runJob`.
   - Type:
     - pre: the request will be sent when the method is called
     - post: the request will be sent after the method action is completed
     - pre/post: both
   - URL: the full URL which the requests will be sent to
+  - Wait for response: you can choose to wait for the web hook response before the method is actually called ("pre" hooks only). This can be useful if you need to automatically run some tasks before a certain method is called.
 - Save the plugin configuration
 
 From now on, a request will be sent to the corresponding URLs when a configured method is called by an XO client.
@@ -326,6 +327,8 @@ Synchronize your pools, VMs, network interfaces and IP addresses with your [Netb
 
 ![](./assets/netbox.png)
 
+### Netbox side
+
 - Go to your Netbox interface
 - Configure prefixes:
   - Go to IPAM > Prefixes > Add
@@ -338,15 +341,32 @@ XO will try to find the right prefix for each IP address. If it can't find a pre
 - Generate a token:
   - Go to Admin > Tokens > Add token
   - Create a token with "Write enabled"
-- Add a UUID custom field:
+  - The owner of the token must have at least the following permissions:
+    - View permissions on:
+      - extras > custom-fields
+      - ipam > prefixes
+    - All permissions on:
+      - ipam > ip-addresses
+      - virtualization > cluster-types
+      - virtualization > clusters
+      - virtualization > interfaces
+      - virtualization > virtual-machines
+- Add a UUID custom field (for **Netbox 2.x**):
   - Got to Admin > Custom fields > Add custom field
-  - Create a custom field called "uuid"
+  - Create a custom field called "uuid" (lower case!)
   - Assign it to object types `virtualization > cluster` and `virtualization > virtual machine`
 
 ![](./assets/customfield.png)
 
+:::tip
+In Netbox 3.x, custom fields can be found directly in the site (no need to go in the admin section). It's available in "Other/Customization/Custom Fields". After creation of the `uuid` field, assign it to the object types `virtualization > cluster` and `virtualization > virtual machine`.
+:::
+
+### In Xen Orchestra
+
 - Go to Xen Orchestra > Settings > Plugins > Netbox and fill out the configuration:
   - Endpoint: the URL of your Netbox instance (e.g.: `https://netbox.company.net`)
+  - Unauthorized certificate: only for HTTPS, enable this option if your Netbox instance uses a self-signed SSL certificate
   - Token: the token you generated earlier
   - Pools: the pools you wish to automatically synchronize with Netbox
   - Interval: the time interval (in hours) between 2 auto-synchronizations. Leave empty if you don't want to synchronize automatically.

docs/backup_reports.md

@@ -87,3 +87,7 @@ You need to be an admin:
 ![Mattermost configuration](./assets/DocImg8.png)
 
 ![Mattermost](./assets/DocImg9.png)
+
+## Web hooks
+
+You can also configure web hooks to be sent to a custom server before and/or after a backup job runs. This won't send a formatted report but raw JSON data that you can use in custom scripts on your side. Follow the [web-hooks plugin documentation](./advanced.html#web-hooks) to configure it.

docs/backups.md

@@ -26,6 +26,12 @@ Each backups' job execution is identified by a `runId`. You can find this `runId
 
 ![](./assets/log-runId.png)
 
+## Exclude disks
+
+During a backup job, you can avoid saving all disks of the VM. To do that is trivial: just edit the VM disk name and add `[NOBAK]` before the current name, eg: `data-disk` will become `[NOBAK] data-disk` (with a space or not, doesn't matter).
+
+The disks marked with `[NOBAK]` will be now ignored in all following backups.
+
 ## Schedule
 
 :::tip
@@ -283,39 +289,42 @@ When it's done exporting, we'll remove the snapshot. Note: this operation will t
 
 ### Concurrency
 
+Concurrency is a parameter that let you define how many VMs your backup job will manage simultaneously.
+
+:::tip
+
+- Default concurrency value is 2 if left empty.
+  :::
+
 Let's say you want to backup 50 VMs (each with 1x disk) at 3:00 AM. There are **2 different strategies**:
 
 1. backup VM #1 (snapshot, export, delete snapshots) **then** backup VM #2 -> _fully sequential strategy_
 2. snapshot all VMs, **then** export all snapshots, **then** delete all snapshots for finished exports -> _fully parallel strategy_
 
-The first purely sequential strategy will lead to a big problem: **you can't predict when a snapshot of your data will occur**. Because you can't predict the first VM export time (let's say 3 hours), then your second VM will have its snapshot taken 3 hours later, at 6 AM. We assume that's not what you meant when you specified "backup everything at 3 AM". You would end up with data from 6 AM (and later) for other VMs.
-
-Strategy number 2 is better in this aspect: all the snapshots will be taken at 3 AM. However **it's risky without limits**: it means potentially doing 50 snapshots or more at once on the same storage. **Since XenServer doesn't have a queue**, it will try to do all of them at once. This is also prone to race conditions and could cause crashes on your storage.
-
-So what's the best choice? Continue below to learn how to best configure concurrency for your needs.
-
-#### Best choice
-
-By default the _parallel strategy_ is, on paper, the most logical one. But we need to give it some limits on concurrency.
+The first purely sequential strategy will lead to the fact that: **you can't predict when a snapshot of your data will occur**. Because you can't predict the first VM export time (let's say 3 hours), then your second VM will have its snapshot taken 3 hours later, at 6 AM.
 
 :::tip
-Xen Orchestra can be connected to multiple pools at once. So the concurrency number applies **per pool**.
+If you need your backup to be done at a specific time you should consider creating a specific backup task for this VM.
 :::
 
-Each step has its own concurrency to fit its requirements:
+Strategy number 2 is to parallelise: all the snapshots will be taken at 3 AM. However **it's risky without limits**: it means potentially doing 50 snapshots or more at once on the same storage. **Since XenServer doesn't have a queue**, it will try to do all of them at once. This is also prone to race conditions and could cause crashes on your storage.
 
-- **snapshot process** needs to be performed with the lowest concurrency possible. 2 is a good compromise: one snapshot is fast, but a stuck snapshot won't block the whole job. That's why a concurrency of 2 is not too bad on your storage. Basically, at 3 AM, we'll do all the VM snapshots needed, 2 at a time.
-- **disk export process** is bottlenecked by XCP-ng/XenServer - so to get the most of it, you can use up to 12 in parallel. As soon a snapshot is done, the export process will start, until reaching 12 at once. Then as soon as one in those 12 is finished, another one will appear until there is nothing more to export.
-- **VM export process:** the 12 disk export limit mentioned above applies to VDI exports, which happen during delta exports. For full VM exports (for example, for full backup job types), there is a built in limit of 2. This means if you have a full backup job of 6 VMs, only 2 will be exported at once.
-- **snapshot deletion** can't happen all at once because the previous step durations are random - no need to implement concurrency on this one.
+By default the _parallel strategy_ is, on paper, the most logical one. But you need to be careful and give it some limits on concurrency.
 
-This is how it currently works in Xen Orchestra. But sometimes, you also want to have _sequential_ backups combined with the _parallel strategy_. That's why we introduced a sequential option in the advanced section of backup-ng:
-
-:::tip
-0 means it will be fully **parallel** for all VMs.
+:::danger
+High concurrency could impact your dom0 and network performances.
 :::
 
-If you job contains 50 VMs for example, you could specify a sequential backup with a limit of "25 at once" (enter 25 in the concurrency field). This means at 3 AM, we'll do 25 snapshots (2 at a time), then exports. As soon as the first VM backup is completely finished (snapshot removed), then we'll start the 26th and so on, to always keep a max of 25x VM backups going in parallel.
+You should be aware of your hardware limitation when defining the best concurrency for your XCP-ng infrastructure, never put concurrency too high or you could impact your VMs performances.
+The best way to define the best concurrency for you is by increasing it slowly and watching the result on backup time.
+
+So to summarize, if you set your concurrency at 6 and you have 20 Vms to backup the process will be the following:
+
+- We start the backup of the first 6 VMs.
+- When one VM backup as ended we will launch the next VM backup.
+- We're keep launching new VM backup until the 20 VMs are finished, keeping 6 backups running.
+
+Removing the snapshot will trigger the coalesce process for the first VM, this is an automated action not triggered directly by the backup job.
 
 ## Backup modifier tags
 

docs/delta_backups.md

@@ -43,12 +43,6 @@ Just go into your "Backup" view, and select Delta Backup. Then, it's the same as
 
 Unlike other types of backup jobs which delete the associated snapshot when the job is done and it has been exported, delta backups always keep a snapshot of every VM in the backup job, and uses it for the delta. Do not delete these snapshots!
 
-## Exclude disks
-
-During a delta backup job, you can avoid saving all disks of the VM. To do that is trivial: just edit the VM disk name and add `[NOBAK]` before the current name, eg: `data-disk` will become `[NOBAK] data-disk` (with a space or not, doesn't matter).
-
-The disks marked with `[NOBAK]` will be now ignored in all following backups.
-
 ## Delta backup initial seed
 
 If you don't want to do an initial full directly toward the destination, you can create a local delta backup first, then transfer the files to your destination.

docs/from_the_sources.md

@@ -14,13 +14,13 @@ As you may have seen in other parts of the documentation, XO is composed of two
 
 ### NodeJS
 
-XO needs Node.js. **Please use Node LTS version 14**.
+XO needs Node.js. **Please always use latest Node LTS**.
 
 We'll consider at this point that you've got a working node on your box. E.g:
 
 ```
 $ node -v
-v14.16.0
+v14.17.0
 ```
 
 If not, see [this page](https://nodejs.org/en/download/package-manager/) for instructions on how to install Node.
@@ -46,7 +46,7 @@ apt-get install build-essential redis-server libpng-dev git python-minimal libvh
 You need to use the `git` source code manager to fetch the code. Ideally, you should run XO as a non-root user, and if you choose to, you need to set up `sudo` to be able to mount NFS remotes. As your chosen non-root (or root) user, run the following:
 
 ```
-git clone -b master http://github.com/vatesfr/xen-orchestra
+git clone -b master https://github.com/vatesfr/xen-orchestra
 ```
 
 > Note: xo-server and xo-web have been migrated to the [xen-orchestra](https://github.com/vatesfr/xen-orchestra) mono-repository - so you only need the single clone command above

docs/full_backups.md

@@ -1,6 +1,6 @@
 # Full backups
 
-You can schedule full backups of your VMs, by exporting them to the local XOA file-system, or directly to an NFS or SMB share. The "rentention" parameter allows you to modify how many backups are retained (by removing the oldest one).
+You can schedule full backups of your VMs, by exporting them to the local XOA file-system, or directly to an NFS or SMB share. The "retention" parameter allows you to modify how many backups are retained (by removing the oldest one).
 
 [![](./assets/backupexample.png)](https://xen-orchestra.com/blog/backup-your-xenserver-vms-with-xen-orchestra/)
 

docs/installation.md

@@ -20,7 +20,7 @@ Once you have started the VM, you can access the web UI by putting the IP you co
 :::tip
 
 - Default Web UI credentials are `admin@admin.net` / `admin`
-- Default console/SSH credentials are `xoa` / `xoa` (first login)
+- Default console/SSH credentials are not set, you need to set them [as described here](troubleshooting.md#set-or-recover-xoa-vm-password).
   :::
 
 ### Registration
@@ -83,13 +83,13 @@ As you may have seen in other parts of the documentation, XO is composed of two
 
 #### NodeJS
 
-XO needs Node.js. **Please use Node LTS 14**.
+XO needs Node.js. **Please always use latest Node LTS**.
 
 We'll consider at this point that you've got a working node on your box. E.g:
 
 ```
 $ node -v
-v14.16.0
+v14.17.0
 ```
 
 If not, see [this page](https://nodejs.org/en/download/package-manager/) for instructions on how to install Node.

docs/load_balancing.md

@@ -94,3 +94,21 @@ The global situation (resource usage) is examined **every minute**.
 :::tip
 TODO: more details to come here
 :::
+
+## VM anti-affinity
+
+VM anti-affinity is a feature that prevents VMs with the same user tags from running on the same host. This functionality is available directly in the load-balancer plugin.
+This way, you can avoid having pairs of redundant VMs or similar running on the same host.
+
+Let's look at a simple example: you have multiple VMs running MySQL and PostgreSQL with high availability/replication. Obviously, you don't want to lose the replicated database inside the VMs on the same physical host. Just create your plan like this:
+
+![](./assets/antiaffinity.png)
+
+- Simple plan: means no active load balancing mechanism used
+- Anti-affinity: we added our 2x tags, meaning any VMs with one of these tags will never run on the same host (if possible) with another VM having the same tag
+
+You can also use the performance plan with the anti-affinity mode activated to continue to migrate non-tagged VMs.
+
+:::tip
+This feature is not limited by the number of VMs using the same tag, i.e. if you have 6 VMs with the same anti-affinity tag and 2 hosts, the plugin will always try to place 3 VMs on each host. It will distribute as much as possible the VMs fairly and it takes precedence (in the majority of the cases) over the performance algorithm.
+:::

docs/manage_infrastructure.md

@@ -320,6 +320,7 @@ You can learn more about XenServer [resource management on the Citrix Website](h
 :::tip
 XCP-ng doesn't limit VMs to 32 vCPU
 :::
+
 ### VDI live migration
 
 Thanks to Xen Storage Motion, it's easy to move a VM disk from one storage location to another, while the VM is running! This feature can help you migrate from your local storage to a SAN, or just upgrade your SAN without any downtime.
@@ -491,10 +492,12 @@ If you are behind a proxy, please update your `xo-server` configuration to add a
 ::: danger
 As specified in the [documentation](https://xcp-ng.org/docs/requirements.html#pool-requirements) your pool shouldn't consist of hosts from different CPU vendors.
 :::
+
 ::: warning
 - Even with matching CPU vendors, in the case of different CPU models XCP-ng will scale the pool CPU ability to the CPU having the least instructions.
 - All the hosts in a pool must run the same XCP-ng version.
 :::
+
 ### Creating a pool
 
 First you should add your new host to XOA by going to New > Server as described in [the relevant chapter](manage_infrastructure.md#add-a-host).