feat(xo-server): 5.19.3

This field is only present on delta backups

CHANGELOG.md

@@ -1,9 +1,104 @@
 # ChangeLog
 
+## **5.20.0** (planned 2018-05-31)
+
+### Enhancements
+
+- Add VDI UUID in SR coalesce view [#2903](https://github.com/vatesfr/xen-orchestra/issues/2903)
+- Create new VDI from SR view not attached to any VM [#2229](https://github.com/vatesfr/xen-orchestra/issues/2229)
+- [Patches] ignore XS upgrade in missing patches counter [#2866](https://github.com/vatesfr/xen-orchestra/issues/2866)
+- [Health] List VM snapshots related to non-existing backup jobs/schedules [#2828](https://github.com/vatesfr/xen-orchestra/issues/2828)
+
+### Bugs
+
+## **5.19.0** (2018-05-01)
+
+### Enhancements
+
+- Expose vendor device in VM advanced tab [#2883](https://github.com/vatesfr/xen-orchestra/issues/2883)
+- Networks created in XO are missing the "automatic" parameter [#2818](https://github.com/vatesfr/xen-orchestra/issues/2818)
+- Performance alert disk space monitoring XS [#2737](https://github.com/vatesfr/xen-orchestra/issues/2737)
+- Add ability to create NFSv4 storage repository [#2706](https://github.com/vatesfr/xen-orchestra/issues/2706)
+- [SortedTable] Support link actions [#2691](https://github.com/vatesfr/xen-orchestra/issues/2691)
+- Additional sort option: by host name [#2680](https://github.com/vatesfr/xen-orchestra/issues/2680)
+- Expose XenTools version numbers in data model and UI [#2650](https://github.com/vatesfr/xen-orchestra/issues/2650)
+- RRDs stats for SR object [#2644](https://github.com/vatesfr/xen-orchestra/issues/2644)
+- composite jobs [#2367](https://github.com/vatesfr/xen-orchestra/issues/2367)
+- Better error message [#2344](https://github.com/vatesfr/xen-orchestra/issues/2344)
+- Avoid using backup tag with special characters [#2336](https://github.com/vatesfr/xen-orchestra/issues/2336)
+- Prefix/suffix for temporary files [#2333](https://github.com/vatesfr/xen-orchestra/issues/2333)
+- Continuous Replication - better interface matching on destination [#2093](https://github.com/vatesfr/xen-orchestra/issues/2093)
+- Creation of LVMoHBA SRs [#1992](https://github.com/vatesfr/xen-orchestra/issues/1992)
+- [Delta backup] Improve restoration by creating a virtual full VHD [#1943](https://github.com/vatesfr/xen-orchestra/issues/1943)
+- VM Backups should be done in a dedicated remote directory [#1752](https://github.com/vatesfr/xen-orchestra/issues/1752)
+- Add Pool / SR filter in backup view [#1762](https://github.com/vatesfr/xen-orchestra/issues/1762)
+- Hide/Disable upgrade button when no upgrade exists [#1594](https://github.com/vatesfr/xen-orchestra/issues/1594)
+- "Upgrade" button should display "Downgrade" when trial is over [#1483](https://github.com/vatesfr/xen-orchestra/issues/1483)
+
+### Bugs
+
+- Allowed-ips don't works displaying index.js:1 Uncaught TypeError: (0 , z.isIp) is not a function [#2891](https://github.com/vatesfr/xen-orchestra/issues/2891)
+- Error on "usage-report" [#2876](https://github.com/vatesfr/xen-orchestra/issues/2876)
+- SR selection combo only listing local storage [#2875](https://github.com/vatesfr/xen-orchestra/issues/2875)
+- [Backup NG - Delta] Issue while importing delta [#2857](https://github.com/vatesfr/xen-orchestra/issues/2857)
+- Create New SR page broken with past commit [#2853](https://github.com/vatesfr/xen-orchestra/issues/2853)
+- [Backup NG] a target should only be preset once [#2848](https://github.com/vatesfr/xen-orchestra/issues/2848)
+- Auth Method iSCSI [#2835](https://github.com/vatesfr/xen-orchestra/issues/2835)
+- [Backup NG] ENOENT with Delta Backup [#2833](https://github.com/vatesfr/xen-orchestra/issues/2833)
+- Different backup logs [#2732](https://github.com/vatesfr/xen-orchestra/issues/2732)
+- Creating network fails silently when omitting Description [#2719](https://github.com/vatesfr/xen-orchestra/issues/2719)
+- Can't create ISO NFS SR via XOA [#1845](https://github.com/vatesfr/xen-orchestra/issues/1845)
+
+## **5.18.0** (2018-03-31)
+
+### Enhancements
+
+- Support huge VHDs [#2785](https://github.com/vatesfr/xen-orchestra/issues/2785)
+- Usage report extended usage [#2770](https://github.com/vatesfr/xen-orchestra/issues/2770)
+- Improve host available RAM display [#2750](https://github.com/vatesfr/xen-orchestra/issues/2750)
+- Hide IP field during VM creation if not configured [#2739](https://github.com/vatesfr/xen-orchestra/issues/2739)
+- [Home] Delete VMs modal should autofocus the input field [#2736](https://github.com/vatesfr/xen-orchestra/issues/2736)
+- Backup restore view load icon [#2692](https://github.com/vatesfr/xen-orchestra/issues/2692)
+- Deleting default templates doesn't work [#2666](https://github.com/vatesfr/xen-orchestra/issues/2666)
+- DR clean previous "failed" snapshots [#2656](https://github.com/vatesfr/xen-orchestra/issues/2656)
+- [Home] Put sort criteria in URL like the filter [#2585](https://github.com/vatesfr/xen-orchestra/issues/2585)
+- Allow disconnect VDI in SR disk view [#2505](https://github.com/vatesfr/xen-orchestra/issues/2505)
+- Add confirmation modal for manual backup run [#2355](https://github.com/vatesfr/xen-orchestra/issues/2355)
+- Multiple schedule for backup jobs [#2286](https://github.com/vatesfr/xen-orchestra/issues/2286)
+- Checks before web update [#2250](https://github.com/vatesfr/xen-orchestra/issues/2250)
+- Backup logs should truly reflect if the job is running [#2206](https://github.com/vatesfr/xen-orchestra/issues/2206)
+- Hook/action if an export stream is cut [#1929](https://github.com/vatesfr/xen-orchestra/issues/1929)
+- Backup paths should not contain tags but job ids [#1854](https://github.com/vatesfr/xen-orchestra/issues/1854)
+- Add a button to delete a backup [#1751](https://github.com/vatesfr/xen-orchestra/issues/1751)
+- Dashboard available for Pool and Host level  [#1631](https://github.com/vatesfr/xen-orchestra/issues/1631)
+- UI Enhancement - VM list - Allways show the Toolbar  [#1581](https://github.com/vatesfr/xen-orchestra/issues/1581)
+- xoa-updater --register:  unable to define proxy using the CLI [#873](https://github.com/vatesfr/xen-orchestra/issues/873)
+
+
+### Bugs
+
+- [Backup NG] CR/DR fail with multiple VMs [#2807](https://github.com/vatesfr/xen-orchestra/issues/2807)
+- HTTPS Crash [#2803](https://github.com/vatesfr/xen-orchestra/issues/2803)
+- Backup NG "cannot fork the stream after it has been created" [#2790](https://github.com/vatesfr/xen-orchestra/issues/2790)
+- [XOSAN] Make temporary `boundObjectId` unique [#2758](https://github.com/vatesfr/xen-orchestra/issues/2758)
+- First VIF ignored at VM creation [#2794](https://github.com/vatesfr/xen-orchestra/issues/2794)
+- VM creation from snapshot does not work [#2748](https://github.com/vatesfr/xen-orchestra/issues/2748)
+- Error: no such object with CentOS 7 template [#2747](https://github.com/vatesfr/xen-orchestra/issues/2747)
+- [Tasks] Filter does not work [#2740](https://github.com/vatesfr/xen-orchestra/issues/2740)
+- Pagination broken when listing pool VMs [#2730](https://github.com/vatesfr/xen-orchestra/issues/2730)
+- All jobs show error icon with message "This backup's creator no longer exists" [#2728](https://github.com/vatesfr/xen-orchestra/issues/2728)
+- [Basic backup] Continous Replication VM names [#2727](https://github.com/vatesfr/xen-orchestra/issues/2727)
+- Continuous replication clone removed [#2724](https://github.com/vatesfr/xen-orchestra/issues/2724)
+- [Backup] "See matching VMs" issue [#2704](https://github.com/vatesfr/xen-orchestra/issues/2704)
+- How to exclude CR targets from a smart backup using tags? [#2613](https://github.com/vatesfr/xen-orchestra/issues/2613)
+- Successful VM import reported as failed [#2056](https://github.com/vatesfr/xen-orchestra/issues/2056)
+- Delta backup: issue if a disk is once again backed up [#1824](https://github.com/vatesfr/xen-orchestra/issues/1824)
+
 ## **5.17.0** (2018-03-02)
 
 ### Enhancements
 
+- Username field labeled inconsistently [#2651](https://github.com/vatesfr/xen-orchestra/issues/2651)
 - Add modal confirmation for host emergency mode [#2230](https://github.com/vatesfr/xen-orchestra/issues/2230)
 - Authorize stats fetching in RO mode [#2678](https://github.com/vatesfr/xen-orchestra/issues/2678)
 - Limit VM.export concurrency [#2669](https://github.com/vatesfr/xen-orchestra/issues/2669)
@@ -22,6 +117,9 @@
 - Cant attach XenTools on User side.  [#2503](https://github.com/vatesfr/xen-orchestra/issues/2503)
 - Pool filter for health view [#2302](https://github.com/vatesfr/xen-orchestra/issues/2302)
 - [Smart Backup] Improve feedback [#2253](https://github.com/vatesfr/xen-orchestra/issues/2253)
+- Backup jobs stuck if no space left on NFS remote [#2116](https://github.com/vatesfr/xen-orchestra/issues/2116)
+- Link between backup and XS tasks [#1193](https://github.com/vatesfr/xen-orchestra/issues/1193)
+- Move delta backup grouping to server side [#1008](https://github.com/vatesfr/xen-orchestra/issues/1008)
 
 ### Bugs
 
@@ -41,6 +139,7 @@
 - Self-service: allow VIF create [#2593](https://github.com/vatesfr/xen-orchestra/issues/2593)
 - Ghost tasks [#2579](https://github.com/vatesfr/xen-orchestra/issues/2579)
 - Autopatching: ignore 7.3 update patch for 7.2 [#2564](https://github.com/vatesfr/xen-orchestra/issues/2564)
+- Better Handling of suspending VMs from the Home screen [#2547](https://github.com/vatesfr/xen-orchestra/issues/2547)
 - Allow deleting VMs for which `destroy` is blocked [#2525](https://github.com/vatesfr/xen-orchestra/issues/2525)
 - Better confirmation on mass destructive actions [#2522](https://github.com/vatesfr/xen-orchestra/issues/2522)
 - Move VM In to/Out of Self Service Group [#1913](https://github.com/vatesfr/xen-orchestra/issues/1913)
@@ -50,6 +149,8 @@
 - Remove CoffeeScript in xo-server [#189](https://github.com/vatesfr/xen-orchestra/issues/189)
 - Better Handling of suspending VMs from the Home screen [#2547](https://github.com/vatesfr/xen-orchestra/issues/2547)
 - [xen-api] Stronger reconnection policy [#2410](https://github.com/vatesfr/xen-orchestra/issues/2410)
+- home view - allow selecting more than 25 items [#1210](https://github.com/vatesfr/xen-orchestra/issues/1210)
+- Performances alerts [#511](https://github.com/vatesfr/xen-orchestra/issues/511)
 
 ### Bugs
 
@@ -79,6 +180,7 @@
 - Graphs in VM list view [#2469](https://github.com/vatesfr/xen-orchestra/issues/2469)
 - [Delta Backups] Do not include merge duration in transfer speed stat [#2426](https://github.com/vatesfr/xen-orchestra/issues/2426)
 - Warning for disperse mode [#2537](https://github.com/vatesfr/xen-orchestra/issues/2537)
+- Select components: auto select value if only 1 choice possible [#1479](https://github.com/vatesfr/xen-orchestra/issues/1479)
 
 ### Bugs
 

packages/xo-server/package.json

@@ -1,6 +1,6 @@
 {
   "name": "xo-server",
-  "version": "5.18.3",
+  "version": "5.19.3",
   "license": "AGPL-3.0",
   "description": "Server part of Xen-Orchestra",
   "keywords": [
@@ -116,7 +116,8 @@
     "xo-collection": "^0.4.1",
     "xo-common": "^0.1.1",
     "xo-remote-parser": "^0.3",
-    "xo-vmdk-to-vhd": "0.0.12"
+    "xo-vmdk-to-vhd": "0.0.12",
+    "yazl": "^2.4.3"
   },
   "devDependencies": {
     "@babel/cli": "7.0.0-beta.44",

packages/xo-server/src/api/backup-ng.js

@@ -1,3 +1,7 @@
+import { basename } from 'path'
+
+import { safeDateFormat } from '../utils'
+
 export function createJob ({ schedules, ...job }) {
   job.userId = this.user.id
   return this.createBackupNgJob(job, schedules)
@@ -171,3 +175,88 @@ importVmBackup.params = {
     type: 'string',
   },
 }
+
+// -----------------------------------------------------------------------------
+
+export function listPartitions ({ remote, disk }) {
+  return this.listBackupNgDiskPartitions(remote, disk)
+}
+
+listPartitions.permission = 'admin'
+
+listPartitions.params = {
+  disk: {
+    type: 'string',
+  },
+  remote: {
+    type: 'string',
+  },
+}
+
+export function listFiles ({ remote, disk, partition, path }) {
+  return this.listBackupNgPartitionFiles(remote, disk, partition, path)
+}
+
+listFiles.permission = 'admin'
+
+listFiles.params = {
+  disk: {
+    type: 'string',
+  },
+  partition: {
+    type: 'string',
+    optional: true,
+  },
+  path: {
+    type: 'string',
+  },
+  remote: {
+    type: 'string',
+  },
+}
+
+async function handleFetchFiles (req, res, { remote, disk, partition, paths }) {
+  const zipStream = await this.fetchBackupNgPartitionFiles(
+    remote,
+    disk,
+    partition,
+    paths
+  )
+
+  res.setHeader('content-disposition', 'attachment')
+  res.setHeader('content-type', 'application/octet-stream')
+  return zipStream
+}
+
+export async function fetchFiles (params) {
+  const { paths } = params
+  let filename = `restore_${safeDateFormat(new Date())}`
+  if (paths.length === 1) {
+    filename += `_${basename(paths[0])}`
+  }
+  filename += '.zip'
+
+  return this.registerHttpRequest(handleFetchFiles, params, {
+    suffix: encodeURI(`/${filename}`),
+  }).then(url => ({ $getFrom: url }))
+}
+
+fetchFiles.permission = 'admin'
+
+fetchFiles.params = {
+  disk: {
+    type: 'string',
+  },
+  partition: {
+    optional: true,
+    type: 'string',
+  },
+  paths: {
+    items: { type: 'string' },
+    minLength: 1,
+    type: 'array',
+  },
+  remote: {
+    type: 'string',
+  },
+}

packages/xo-server/src/lvm.js

@@ -1,16 +1,15 @@
 import execa from 'execa'
 import splitLines from 'split-lines'
 import { createParser } from 'parse-pairs'
-import { isArray, map } from 'lodash'
 
 // ===================================================================
 
 const parse = createParser({
   keyTransform: key => key.slice(5).toLowerCase(),
 })
-const makeFunction = command => (fields, ...args) =>
-  execa
-    .stdout(command, [
+const makeFunction = command => async (fields, ...args) => {
+  return splitLines(
+    await execa.stdout(command, [
       '--noheading',
       '--nosuffix',
       '--nameprefixes',
@@ -21,17 +20,8 @@ const makeFunction = command => (fields, ...args) =>
       String(fields),
       ...args,
     ])
-    .then(stdout =>
-      map(
-        splitLines(stdout),
-        isArray(fields)
-          ? parse
-          : line => {
-            const data = parse(line)
-            return data[fields]
-          }
-      )
-    )
+  ).map(Array.isArray(fields) ? parse : line => parse(line)[fields])
+}
 
 export const lvs = makeFunction('lvs')
 export const pvs = makeFunction('pvs')

packages/xo-server/src/vhd-merge.js

@@ -977,7 +977,7 @@ export const createReadStream = asyncIteratorToStream(function * (handler, path)
           }
         }
       }
-      yield * emitBlockSectors(owner, 0, sectorsPerBlock)
+      yield * emitBlockSectors(owner, 0, sectorsPerBlockData)
     }
 
     yield footer

packages/xo-server/src/xapi/index.js

@@ -1026,8 +1026,6 @@ export default class Xapi extends XapiBase {
           sr: mapVdisSrs[vdi.uuid] || srId,
         })
         $defer.onFailure(() => this._deleteVdi(newVdi))
-
-        return newVdi
       }
 
       await asyncMap(vbds[vdiId], vbd =>

packages/xo-server/src/xo-mixins/backups-ng/index.js

@@ -539,6 +539,19 @@ export default class BackupNg {
             // inject an id usable by importVmBackupNg()
             backups.forEach(backup => {
               backup.id = `${remoteId}/${backup._filename}`
+
+              const { vdis, vhds } = backup
+              backup.disks =
+                vhds === undefined
+                  ? []
+                  : Object.keys(vhds).map(vdiId => {
+                      const vdi = vdis[vdiId]
+                      return {
+                        id: `${dirname(backup._filename)}/${vhds[vdiId]}`,
+                        name: vdi.name_label,
+                        uuid: vdi.uuid,
+                      }
+                    })
             })
 
             backupsByVm[vmUuid] = backups
@@ -1096,7 +1109,11 @@ export default class BackupNg {
         })
       )
     } catch (error) {
-      if (error == null || error.code !== 'ENOENT') {
+      let code
+      if (
+        error == null ||
+        ((code = error.code) !== 'ENOENT' && code !== 'ENOTDIR')
+      ) {
         throw error
       }
     }

packages/xo-server/src/xo-mixins/backups.js

@@ -12,7 +12,6 @@ import {
   endsWith,
   filter,
   find,
-  findIndex,
   includes,
   once,
   range,
@@ -22,7 +21,10 @@ import {
 } from 'lodash'
 
 import createSizeStream from '../size-stream'
-import vhdMerge, { chainVhd } from '../vhd-merge'
+import vhdMerge, {
+  chainVhd,
+  createReadStream as createVhdReadStream,
+} from '../vhd-merge'
 import xapiObjectToXo from '../xapi-object-to-xo'
 import { lvs, pvs } from '../lvm'
 import {
@@ -566,33 +568,6 @@ export default class {
     return mergedDataSize
   }
 
-  async _listDeltaVdiDependencies (handler, filePath) {
-    const dir = dirname(filePath)
-    const filename = basename(filePath)
-    const backups = await this._listVdiBackups(handler, dir)
-
-    // Search file. (delta or full backup)
-    const i = findIndex(
-      backups,
-      backup => getVdiTimestamp(backup) === getVdiTimestamp(filename)
-    )
-
-    if (i === -1) {
-      throw new Error('VDI to import not found in this remote.')
-    }
-
-    // Search full backup.
-    let j
-
-    for (j = i; j >= 0 && isDeltaVdiBackup(backups[j]); j--);
-
-    if (j === -1) {
-      throw new Error(`Unable to found full vdi backup of: ${filePath}`)
-    }
-
-    return backups.slice(j, i + 1)
-  }
-
   // -----------------------------------------------------------------
 
   async _listDeltaVmBackups (handler, dir) {
@@ -840,17 +815,10 @@ export default class {
 
       await Promise.all(
         mapToArray(delta.vdis, async (vdi, id) => {
-          const vdisFolder = `${basePath}/${dirname(vdi.xoPath)}`
-          const backups = await this._listDeltaVdiDependencies(
+          streams[`${id}.vhd`] = await createVhdReadStream(
             handler,
             `${basePath}/${vdi.xoPath}`
           )
-
-          streams[`${id}.vhd`] = await Promise.all(
-            mapToArray(backups, async backup =>
-              handler.createReadStream(`${vdisFolder}/${backup}`)
-            )
-          )
         })
       )
 
@@ -1038,13 +1006,13 @@ export default class {
             // VHD path may need to be fixed.
             return endsWith(vhdPath, '_delta.vhd')
               ? pFromCallback(cb => stat(vhdPath, cb)).then(
-                () => vhdPath,
-                error => {
-                  if (error && error.code === 'ENOENT') {
-                    return `${vhdPath.slice(0, -10)}_full.vhd`
+                  () => vhdPath,
+                  error => {
+                    if (error && error.code === 'ENOENT') {
+                      return `${vhdPath.slice(0, -10)}_full.vhd`
+                    }
                   }
-                }
-              )
+                )
               : vhdPath
           })
           .then(vhdPath => execa('vhdimount', [vhdPath, mountDir]))

packages/xo-server/src/xo-mixins/file-restore-ng.js

@@ -0,0 +1,332 @@
+import defer from 'golike-defer'
+import execa from 'execa'
+import splitLines from 'split-lines'
+import { createParser as createPairsParser } from 'parse-pairs'
+import { normalize } from 'path'
+import { readdir, rmdir, stat } from 'fs-extra'
+import { ZipFile } from 'yazl'
+
+import { lvs, pvs } from '../lvm'
+import { resolveSubpath, tmpDir } from '../utils'
+
+const IGNORED_PARTITION_TYPES = {
+  // https://github.com/jhermsmeier/node-mbr/blob/master/lib/partition.js#L38
+  0x05: true,
+  0x0f: true,
+  0x15: true,
+  0x5e: true,
+  0x5f: true,
+  0x85: true,
+  0x91: true,
+  0x9b: true,
+  0xc5: true,
+  0xcf: true,
+  0xd5: true,
+
+  0x82: true, // swap
+}
+
+const PARTITION_TYPE_NAMES = {
+  0x07: 'NTFS',
+  0x0c: 'FAT',
+  0x83: 'linux',
+}
+
+const RE_VHDI = /^vhdi(\d+)$/
+
+const parsePartxLine = createPairsParser({
+  keyTransform: key => (key === 'UUID' ? 'id' : key.toLowerCase()),
+  valueTransform: (value, key) =>
+    key === 'start' || key === 'size'
+      ? +value
+      : key === 'type' ? PARTITION_TYPE_NAMES[+value] || value : value,
+})
+
+const listLvmLogicalVolumes = defer(
+  async ($defer, devicePath, partition, results = []) => {
+    const pv = await mountLvmPhysicalVolume(devicePath, partition)
+    $defer(pv.unmount)
+
+    const lvs = await pvs(['lv_name', 'lv_path', 'lv_size', 'vg_name'], pv.path)
+    const partitionId = partition !== undefined ? partition.id : ''
+    lvs.forEach((lv, i) => {
+      const name = lv.lv_name
+      if (name !== '') {
+        results.push({
+          id: `${partitionId}/${lv.vg_name}/${name}`,
+          name,
+          size: lv.lv_size,
+        })
+      }
+    })
+    return results
+  }
+)
+
+async function mountLvmPhysicalVolume (devicePath, partition) {
+  const args = []
+  if (partition !== undefined) {
+    args.push('-o', partition.start * 512)
+  }
+  args.push('--show', '-f', devicePath)
+  const path = (await execa.stdout('losetup', args)).trim()
+  await execa('pvscan', ['--cache', path])
+
+  return {
+    path,
+    unmount: async () => {
+      try {
+        const vgNames = await pvs('vg_name', path)
+        await execa('vgchange', ['-an', ...vgNames])
+      } finally {
+        await execa('losetup', ['-d', path])
+      }
+    },
+  }
+}
+
+const mountPartition = defer(async ($defer, devicePath, partition) => {
+  const options = ['loop', 'ro']
+
+  if (partition !== undefined) {
+    const { start } = partition
+    if (start !== undefined) {
+      options.push(`offset=${start * 512}`)
+    }
+  }
+
+  const path = await tmpDir()
+  $defer.onFailure(rmdir, path)
+
+  const mount = options =>
+    execa('mount', [
+      `--options=${options.join(',')}`,
+      `--source=${devicePath}`,
+      `--target=${path}`,
+    ])
+
+  // `norecovery` option is used for ext3/ext4/xfs, if it fails it might be
+  // another fs, try without
+  try {
+    await mount([...options, 'norecovery'])
+  } catch (error) {
+    await mount(options)
+  }
+  const unmount = async () => {
+    await execa('umount', ['--lazy', path])
+    return rmdir(path)
+  }
+  $defer.onFailure(unmount)
+
+  return { path, unmount }
+})
+
+// - [x] list partitions
+// - [x] list files in a partition
+// - [x] list files in a bare partition
+// - [x] list LVM partitions
+//
+// - [ ] partitions with unmount debounce
+// - [ ] handle directory restore
+// - [ ] handle multiple entries restore (both dirs and files)
+//       - [ ] by default use common path as root
+// - [ ] handle LVM partitions on multiple disks
+// - [ ] find mounted disks/partitions on start (in case of interruptions)
+//
+// - [ ] manual mount/unmount (of disk) for advance file restore
+//       - could it stay mounted during the backup process?
+//       - [ ] mountDisk (VHD)
+//       - [ ] unmountDisk (only for manual mount)
+//       - [ ] getMountedDisks
+//       - [ ] mountPartition (optional)
+//       - [ ] getMountedPartitions
+//       - [ ] unmountPartition
+export default class BackupNgFileRestore {
+  constructor (app) {
+    this._app = app
+    this._mounts = { __proto__: null }
+  }
+
+  @defer
+  async fetchBackupNgPartitionFiles (
+    $defer,
+    remoteId,
+    diskId,
+    partitionId,
+    paths
+  ) {
+    const disk = await this._mountDisk(remoteId, diskId)
+    $defer.onFailure(disk.unmount)
+
+    const partition = await this._mountPartition(disk.path, partitionId)
+    $defer.onFailure(partition.unmount)
+
+    const zip = new ZipFile()
+    paths.forEach(file => {
+      zip.addFile(resolveSubpath(partition.path, file), normalize('./' + file))
+    })
+    zip.end()
+    return zip.outputStream.on('end', () =>
+      partition.unmount().then(disk.unmount)
+    )
+  }
+
+  @defer
+  async listBackupNgDiskPartitions ($defer, remoteId, diskId) {
+    const disk = await this._mountDisk(remoteId, diskId)
+    $defer(disk.unmount)
+    return this._listPartitions(disk.path)
+  }
+
+  @defer
+  async listBackupNgPartitionFiles (
+    $defer,
+    remoteId,
+    diskId,
+    partitionId,
+    path
+  ) {
+    const disk = await this._mountDisk(remoteId, diskId)
+    $defer(disk.unmount)
+
+    const partition = await this._mountPartition(disk.path, partitionId)
+    $defer(partition.unmount)
+
+    path = resolveSubpath(partition.path, path)
+
+    const entriesMap = {}
+    await Promise.all(
+      (await readdir(path)).map(async name => {
+        try {
+          const stats = await stat(`${path}/${name}`)
+          entriesMap[stats.isDirectory() ? `${name}/` : name] = {}
+        } catch (error) {
+          if (error == null || error.code !== 'ENOENT') {
+            throw error
+          }
+        }
+      })
+    )
+    return entriesMap
+  }
+
+  async _findPartition (devicePath, partitionId) {
+    const partitions = await this._listPartitions(devicePath, false)
+    const partition = partitions.find(_ => _.id === partitionId)
+    if (partition === undefined) {
+      throw new Error(`partition ${partitionId} not found`)
+    }
+    return partition
+  }
+
+  async _listPartitions (devicePath, inspectLvmPv = true) {
+    const stdout = await execa.stdout('partx', [
+      '--bytes',
+      '--output=NR,START,SIZE,NAME,UUID,TYPE',
+      '--pairs',
+      devicePath,
+    ])
+
+    const promises = []
+    const partitions = []
+    splitLines(stdout).forEach(line => {
+      const partition = parsePartxLine(line)
+      let { type } = partition
+      if (type == null || (type = +type) in IGNORED_PARTITION_TYPES) {
+        return
+      }
+
+      if (inspectLvmPv && type === 0x8e) {
+        promises.push(listLvmLogicalVolumes(devicePath, partition, partitions))
+        return
+      }
+
+      partitions.push(partition)
+    })
+
+    await Promise.all(promises)
+
+    return partitions
+  }
+
+  @defer
+  async _mountDisk ($defer, remoteId, diskId) {
+    const handler = await this._app.getRemoteHandler(remoteId)
+    if (handler._getFilePath === undefined) {
+      throw new Error(`this remote is not supported`)
+    }
+
+    const diskPath = handler._getFilePath(diskId)
+    const mountDir = await tmpDir()
+    $defer.onFailure(rmdir, mountDir)
+
+    await execa('vhdimount', [diskPath, mountDir])
+    const unmount = async () => {
+      await execa('fusermount', ['-uz', mountDir])
+      return rmdir(mountDir)
+    }
+    $defer.onFailure(unmount)
+
+    let max = 0
+    let maxEntry
+    const entries = await readdir(mountDir)
+    entries.forEach(entry => {
+      const matches = RE_VHDI.exec(entry)
+      if (matches !== null) {
+        const value = +matches[1]
+        if (value > max) {
+          max = value
+          maxEntry = entry
+        }
+      }
+    })
+    if (max === 0) {
+      throw new Error('no disks found')
+    }
+
+    return {
+      path: `${mountDir}/${maxEntry}`,
+      unmount,
+    }
+  }
+
+  @defer
+  async _mountPartition ($defer, devicePath, partitionId) {
+    if (partitionId === undefined) {
+      return mountPartition(devicePath)
+    }
+
+    if (partitionId.includes('/')) {
+      const [pvId, vgName, lvName] = partitionId.split('/')
+      const lvmPartition =
+        pvId !== '' ? await this._findPartition(devicePath, pvId) : undefined
+
+      const pv = await mountLvmPhysicalVolume(devicePath, lvmPartition)
+
+      const unmountQueue = [pv.unmount]
+      const unmount = async () => {
+        let fn
+        while ((fn = unmountQueue.pop()) !== undefined) {
+          await fn()
+        }
+      }
+      $defer.onFailure(unmount)
+
+      await execa('vgchange', ['-ay', vgName])
+      unmountQueue.push(() => execa('vgchange', ['-an', vgName]))
+
+      const partition = await mountPartition(
+        (await lvs(['lv_name', 'lv_path'], vgName)).find(
+          _ => _.lv_name === lvName
+        ).lv_path
+      )
+      unmountQueue.push(partition.unmount)
+      return { ...partition, unmount }
+    }
+
+    return mountPartition(
+      devicePath,
+      await this._findPartition(devicePath, partitionId)
+    )
+  }
+}

packages/xo-server/src/xo-mixins/jobs/index.js

@@ -225,9 +225,10 @@ export default class Jobs {
 
     runningJobs[id] = runJobId
 
+    let session
     try {
       const app = this._app
-      const session = app.createUserConnection()
+      session = app.createUserConnection()
       session.set('user_id', job.userId)
 
       const status = await executor({
@@ -255,6 +256,9 @@ export default class Jobs {
       throw error
     } finally {
       delete runningJobs[id]
+      if (session !== undefined) {
+        session.close()
+      }
     }
   }
 

packages/xo-server/src/xo.js

@@ -140,7 +140,11 @@ export default class Xo extends EventEmitter {
     }).then(
       result => {
         if (result != null) {
-          res.end(JSON.stringify(result))
+          if (typeof result.pipe === 'function') {
+            result.pipe(res)
+          } else {
+            res.end(JSON.stringify(result))
+          }
         }
       },
       error => {

packages/xo-web/package.json

@@ -1,7 +1,7 @@
 {
   "private": false,
   "name": "xo-web",
-  "version": "5.18.3",
+  "version": "5.19.0",
   "license": "AGPL-3.0",
   "description": "Web interface client for Xen-Orchestra",
   "keywords": [

packages/xo-web/src/common/intl/messages.js

@@ -540,10 +540,11 @@ const messages = {
 
   // ----- SR advanced tab -----
 
+  srUnhealthyVdiDepth: 'Depth',
   srUnhealthyVdiNameLabel: 'Name',
   srUnhealthyVdiSize: 'Size',
-  srUnhealthyVdiDepth: 'Depth',
   srUnhealthyVdiTitle: 'VDI to coalesce ({total, number})',
+  srUnhealthyVdiUuid: 'UUID',
 
   // ----- SR stats tab -----
 
@@ -1037,6 +1038,8 @@ const messages = {
   vmNameLabel: 'Name',
   vmNameDescription: 'Description',
   vmContainer: 'Resident on',
+  vmSnapshotsRelatedToNonExistentBackups:
+    'VM snapshots related to non-existent backups',
   alarmMessage: 'Alarms',
   noAlarms: 'No alarms',
   alarmDate: 'Date',

packages/xo-web/src/common/render-xo-item.js

@@ -5,6 +5,7 @@ import { startsWith } from 'lodash'
 import Icon from './icon'
 import propTypes from './prop-types-decorator'
 import { createGetObject } from './selectors'
+import { FormattedDate } from 'react-intl'
 import { isSrWritable } from './xo'
 import { connectStore, formatSize } from './utils'
 
@@ -203,10 +204,29 @@ const xoItemToRender = {
         : group.name_label}
     </span>
   ),
+
+  backup: backup => (
+    <span>
+      <span className='tag tag-info' style={{ textTransform: 'capitalize' }}>
+        {backup.mode}
+      </span>{' '}
+      <span className='tag tag-warning'>{backup.remote.name}</span>{' '}
+      <FormattedDate
+        value={new Date(backup.timestamp)}
+        month='long'
+        day='numeric'
+        year='numeric'
+        hour='2-digit'
+        minute='2-digit'
+        second='2-digit'
+      />
+    </span>
+  ),
 }
 
-const renderXoItem = (item, { className } = {}) => {
-  const { id, type, label } = item
+const renderXoItem = (item, { className, type: xoType } = {}) => {
+  const { id, label } = item
+  const type = xoType || item.type
 
   if (item.removed) {
     return (
@@ -245,6 +265,9 @@ const renderXoItem = (item, { className } = {}) => {
 
 export { renderXoItem as default }
 
+export const getRenderXoItemOfType = type => (item, options = {}) =>
+  renderXoItem(item, { ...options, type })
+
 const GenericXoItem = connectStore(() => {
   const getObject = createGetObject()
 

packages/xo-web/src/common/selectors.js

@@ -495,7 +495,8 @@ export const createDoesHostNeedRestart = hostSelector => {
       )
     )
     .find([
-      ({ guidance }) =>
+      ({ guidance, upgrade }) =>
+        upgrade ||
         find(
           guidance,
           action => action === 'restartHost' || action === 'restartXapi'

packages/xo-web/src/common/xo-line-chart/index.js

@@ -1,10 +1,22 @@
 import ChartistGraph from 'react-chartist'
 import ChartistLegend from 'chartist-plugin-legend'
 import ChartistTooltip from 'chartist-plugin-tooltip'
+import humanFormat from 'human-format'
 import React from 'react'
 import { injectIntl } from 'react-intl'
 import { messages } from 'intl'
-import { find, flatten, floor, get, map, max, size, sum, values } from 'lodash'
+import {
+  find,
+  flatten,
+  floor,
+  get,
+  map,
+  max,
+  round,
+  size,
+  sum,
+  values,
+} from 'lodash'
 
 import propTypes from '../prop-types-decorator'
 import { computeArraysSum } from '../xo-stats'
@@ -87,9 +99,9 @@ const makeLabelInterpolationFnc = (intl, nValues, endTimestamp, interval) => {
   return (value, index) =>
     index % labelSpace === 0
       ? intl.formatTime(
-        (endTimestamp - (nValues - index - 1) * interval) * 1000,
-        format
-      )
+          (endTimestamp - (nValues - index - 1) * interval) * 1000,
+          format
+        )
       : null
 }
 
@@ -441,19 +453,19 @@ export const PoolPifLineChart = injectIntl(
 
     const series = addSumSeries
       ? map(ios, io => ({
-        name: `${intl.formatMessage(messages.poolAllHosts)} (${io})`,
-        data: computeArraysSum(
-          map(data, ({ stats }) => computeArraysSum(stats.pifs[io]))
-        ),
-      }))
+          name: `${intl.formatMessage(messages.poolAllHosts)} (${io})`,
+          data: computeArraysSum(
+            map(data, ({ stats }) => computeArraysSum(stats.pifs[io]))
+          ),
+        }))
       : flatten(
-        map(data, ({ stats, host }) =>
-          map(ios, io => ({
-            name: `${host} (${io})`,
-            data: computeArraysSum(stats.pifs[io]),
-          }))
+          map(data, ({ stats, host }) =>
+            map(ios, io => ({
+              name: `${host} (${io})`,
+              data: computeArraysSum(stats.pifs[io]),
+            }))
+          )
         )
-      )
 
     return (
       <ChartistGraph
@@ -604,7 +616,11 @@ export const IopsLineChart = injectIntl(
             nValues: length,
             endTimestamp,
             interval,
-            valueTransform: value => `${value.toPrecision(3)} /s`,
+            valueTransform: value =>
+              humanFormat(value, {
+                decimals: 3,
+                unit: 'IOPS',
+              }),
           }),
           ...options,
         }}
@@ -721,7 +737,7 @@ export const IowaitChart = injectIntl(
             nValues: length,
             endTimestamp,
             interval,
-            valueTransform: value => `${value.toPrecision(2)}%`,
+            valueTransform: value => `${round(value, 2)}%`,
           }),
           ...options,
         }}

packages/xo-web/src/common/xo/index.js

@@ -1918,6 +1918,22 @@ export const fetchFiles = (remote, disk, partition, paths, format) =>
     window.location = `.${url}`
   })
 
+// File restore NG  ----------------------------------------------------
+
+export const listPartitions = (remote, disk) =>
+  _call('backupNg.listPartitions', resolveIds({ remote, disk }))
+
+export const listFiles = (remote, disk, path, partition) =>
+  _call('backupNg.listFiles', resolveIds({ remote, disk, path, partition }))
+
+export const fetchFilesNg = (remote, disk, partition, paths, format) =>
+  _call(
+    'backupNg.fetchFiles',
+    resolveIds({ remote, disk, partition, paths, format })
+  ).then(({ $getFrom: url }) => {
+    window.location = `.${url}`
+  })
+
 // -------------------------------------------------------------------
 
 export const probeSrNfs = (host, server) =>

packages/xo-web/src/xo-app/backup-ng/file-restore/index.js

@@ -1,8 +1,233 @@
+import _ from 'intl'
+import ActionButton from 'action-button'
 import Component from 'base-component'
+import Icon from 'icon'
 import React from 'react'
+import SortedTable from 'sorted-table'
+import Upgrade from 'xoa-upgrade'
+import { addSubscriptions, noop } from 'utils'
+import { confirm } from 'modal'
+import { error } from 'notification'
+import { FormattedDate } from 'react-intl'
+import {
+  deleteBackups,
+  fetchFilesNg as fetchFiles,
+  listVmBackups,
+  subscribeRemotes,
+} from 'xo'
+import {
+  assign,
+  filter,
+  flatMap,
+  forEach,
+  keyBy,
+  map,
+  reduce,
+  toArray,
+} from 'lodash'
+
+import DeleteBackupsModalBody from '../restore/delete-backups-modal-body'
+import RestoreFileModalBody from './restore-file-modal'
+
+// -----------------------------------------------------------------------------
+
+const BACKUPS_COLUMNS = [
+  {
+    name: _('backupVmNameColumn'),
+    itemRenderer: ({ last }) => last.vm.name_label,
+    sortCriteria: 'last.vm.name_label',
+  },
+  {
+    name: _('backupVmDescriptionColumn'),
+    itemRenderer: ({ last }) => last.vm.name_description,
+    sortCriteria: 'last.vm.name_description',
+  },
+  {
+    name: _('firstBackupColumn'),
+    itemRenderer: ({ first }) => (
+      <FormattedDate
+        value={new Date(first.timestamp)}
+        month='long'
+        day='numeric'
+        year='numeric'
+        hour='2-digit'
+        minute='2-digit'
+        second='2-digit'
+      />
+    ),
+    sortCriteria: 'first.timestamp',
+    sortOrder: 'desc',
+  },
+  {
+    name: _('lastBackupColumn'),
+    itemRenderer: ({ last }) => (
+      <FormattedDate
+        value={new Date(last.timestamp)}
+        month='long'
+        day='numeric'
+        year='numeric'
+        hour='2-digit'
+        minute='2-digit'
+        second='2-digit'
+      />
+    ),
+    sortCriteria: 'last.timestamp',
+    default: true,
+    sortOrder: 'desc',
+  },
+  {
+    name: _('availableBackupsColumn'),
+    itemRenderer: ({ count }) => count,
+    sortCriteria: 'count',
+  },
+]
+
+// -----------------------------------------------------------------------------
+
+@addSubscriptions({
+  remotes: subscribeRemotes,
+})
+export default class Restore extends Component {
+  state = {
+    backupDataByVm: {},
+  }
+
+  componentWillReceiveProps (props) {
+    if (props.remotes !== this.props.remotes) {
+      this._refreshBackupList(props.remotes)
+    }
+  }
+
+  _refreshBackupList = async (_ = this.props.remotes) => {
+    const remotes = keyBy(filter(_, { enabled: true }), 'id')
+    const backupsByRemote = await listVmBackups(toArray(remotes))
+
+    const backupDataByVm = {}
+    forEach(backupsByRemote, (backups, remoteId) => {
+      const remote = remotes[remoteId]
+      forEach(backups, (vmBackups, vmId) => {
+        vmBackups = filter(vmBackups, { mode: 'delta' })
+        if (vmBackups.length === 0) {
+          return
+        }
+        if (backupDataByVm[vmId] === undefined) {
+          backupDataByVm[vmId] = { backups: [] }
+        }
+
+        backupDataByVm[vmId].backups.push(
+          ...map(vmBackups, bkp => ({ ...bkp, remote }))
+        )
+      })
+    })
+    let first, last
+    forEach(backupDataByVm, (data, vmId) => {
+      first = { timestamp: Infinity }
+      last = { timestamp: 0 }
+      let count = 0 // Number since there's only 1 mode in file restore
+      forEach(data.backups, backup => {
+        if (backup.timestamp > last.timestamp) {
+          last = backup
+        }
+        if (backup.timestamp < first.timestamp) {
+          first = backup
+        }
+        count++
+      })
+
+      assign(data, { first, last, count, id: vmId })
+    })
+    this.setState({ backupDataByVm })
+  }
+
+  // Actions -------------------------------------------------------------------
+
+  _restore = ({ backups, last }) =>
+    confirm({
+      title: _('restoreFilesFromBackup', { name: last.vm.name_label }),
+      body: (
+        <RestoreFileModalBody vmName={last.vm.name_label} backups={backups} />
+      ),
+    }).then(({ remote, disk, partition, paths, format }) => {
+      if (
+        remote === undefined ||
+        disk === undefined ||
+        paths === undefined ||
+        paths.length === 0
+      ) {
+        return error(_('restoreFiles'), _('restoreFilesError'))
+      }
+      return fetchFiles(remote, disk, partition, paths, format)
+    }, noop)
+
+  _delete = data =>
+    confirm({
+      title: _('deleteVmBackupsTitle', { vm: data.last.vm.name_label }),
+      body: <DeleteBackupsModalBody backups={data.backups} />,
+      icon: 'delete',
+    })
+      .then(deleteBackups, noop)
+      .then(() => this._refreshBackupList())
+
+  _bulkDelete = datas =>
+    confirm({
+      title: _('deleteVmBackupsBulkTitle'),
+      body: <p>{_('deleteVmBackupsBulkMessage', { nVms: datas.length })}</p>,
+      icon: 'delete',
+      strongConfirm: {
+        messageId: 'deleteVmBackupsBulkConfirmText',
+        values: {
+          nBackups: reduce(datas, (sum, data) => sum + data.backups.length, 0),
+        },
+      },
+    })
+      .then(() => deleteBackups(flatMap(datas, 'backups')), noop)
+      .then(() => this._refreshBackupList())
+
+  // ---------------------------------------------------------------------------
+
+  _actions = [
+    {
+      handler: this._bulkDelete,
+      icon: 'delete',
+      individualHandler: this._delete,
+      label: _('deleteVmBackups'),
+      level: 'danger',
+    },
+  ]
+
+  _individualActions = [
+    {
+      handler: this._restore,
+      icon: 'restore',
+      label: _('restoreVmBackups'),
+      level: 'primary',
+    },
+  ]
 
-export default class FileRestore extends Component {
   render () {
-    return <p className='text-danger'>Available soon</p>
+    return (
+      <Upgrade place='restoreBackup' available={2}>
+        <div>
+          <div className='mb-1'>
+            <ActionButton
+              btnStyle='primary'
+              handler={this._refreshBackupList}
+              icon='refresh'
+            >
+              {_('restoreResfreshList')}
+            </ActionButton>
+          </div>
+          <em>
+            <Icon icon='info' /> {_('restoreDeltaBackupsInfo')}
+          </em>
+          <SortedTable
+            actions={this._actions}
+            collection={this.state.backupDataByVm}
+            columns={BACKUPS_COLUMNS}
+            individualActions={this._individualActions}
+          />
+        </div>
+      </Upgrade>
+    )
   }
 }

packages/xo-web/src/xo-app/backup-ng/file-restore/restore-file-modal.js

@@ -0,0 +1,379 @@
+import _ from 'intl'
+import ActionButton from 'action-button'
+import Component from 'base-component'
+import endsWith from 'lodash/endsWith'
+import Icon from 'icon'
+import React from 'react'
+import replace from 'lodash/replace'
+import Select from 'form/select'
+import Tooltip from 'tooltip'
+import { Container, Col, Row } from 'grid'
+import { createSelector } from 'reselect'
+import { formatSize } from 'utils'
+import { filter, includes, isEmpty, map } from 'lodash'
+import { getRenderXoItemOfType } from 'render-xo-item'
+import { listPartitions, listFiles } from 'xo'
+
+const BACKUP_RENDERER = getRenderXoItemOfType('backup')
+
+const partitionOptionRenderer = partition => (
+  <span>
+    {partition.name} {partition.type}{' '}
+    {partition.size && `(${formatSize(+partition.size)})`}
+  </span>
+)
+
+const diskOptionRenderer = disk => <span>{disk.name}</span>
+
+const fileOptionRenderer = file => <span>{file.name}</span>
+
+const formatFilesOptions = (rawFiles, path) => {
+  const files =
+    path !== '/'
+      ? [
+          {
+            name: '..',
+            id: '..',
+            path: getParentPath(path),
+            content: {},
+          },
+        ]
+      : []
+
+  return files.concat(
+    map(rawFiles, (file, name) => ({
+      name,
+      id: `${path}${name}`,
+      path: `${path}${name}`,
+      content: file,
+    }))
+  )
+}
+
+const getParentPath = path => replace(path, /^(\/+.+)*(\/+.+)/, '$1/')
+
+// -----------------------------------------------------------------------------
+
+export default class RestoreFileModalBody extends Component {
+  state = {
+    format: 'zip',
+  }
+
+  get value () {
+    const { state } = this
+
+    return {
+      disk: state.disk,
+      format: state.format,
+      partition: state.partition,
+      paths: state.selectedFiles && map(state.selectedFiles, 'path'),
+      remote: state.backup.remote.id,
+    }
+  }
+
+  _listFiles = () => {
+    const { backup, disk, partition, path } = this.state
+    this.setState({ scanningFiles: true })
+
+    return listFiles(backup.remote.id, disk, path, partition).then(
+      rawFiles =>
+        this.setState({
+          files: formatFilesOptions(rawFiles, path),
+          scanningFiles: false,
+          listFilesError: false,
+        }),
+      error => {
+        this.setState({
+          scanningFiles: false,
+          listFilesError: true,
+        })
+        throw error
+      }
+    )
+  }
+
+  _getSelectableFiles = createSelector(
+    () => this.state.files,
+    () => this.state.selectedFiles,
+    (available, selected) =>
+      filter(available, file => !includes(selected, file))
+  )
+
+  _onBackupChange = backup => {
+    this.setState({
+      backup,
+      disk: undefined,
+      partition: undefined,
+      file: undefined,
+      selectedFiles: undefined,
+      scanDiskError: false,
+      listFilesError: false,
+    })
+  }
+
+  _onDiskChange = disk => {
+    this.setState({
+      partition: undefined,
+      file: undefined,
+      selectedFiles: undefined,
+      scanDiskError: false,
+      listFilesError: false,
+    })
+
+    if (!disk) {
+      return
+    }
+
+    listPartitions(this.state.backup.remote.id, disk).then(
+      partitions => {
+        if (isEmpty(partitions)) {
+          this.setState(
+            {
+              disk,
+              path: '/',
+            },
+            this._listFiles
+          )
+
+          return
+        }
+
+        this.setState({
+          disk,
+          partitions,
+        })
+      },
+      error => {
+        this.setState({
+          disk,
+          scanDiskError: true,
+        })
+        throw error
+      }
+    )
+  }
+
+  _onPartitionChange = partition => {
+    this.setState(
+      {
+        partition,
+        path: '/',
+        file: undefined,
+        selectedFiles: undefined,
+      },
+      partition && this._listFiles
+    )
+  }
+
+  _onFileChange = file => {
+    if (file == null) {
+      return
+    }
+
+    // Ugly workaround to keep the ReactSelect open after selecting a folder
+    // FIXME: Remove once something better is implemented in react-select:
+    // https://github.com/JedWatson/react-select/issues/1989
+    const select = document.activeElement
+    select.blur()
+    select.focus()
+
+    const isFile = file.id !== '..' && !endsWith(file.path, '/')
+    if (isFile) {
+      const { selectedFiles } = this.state
+      if (!includes(selectedFiles, file)) {
+        this.setState({
+          selectedFiles: (selectedFiles || []).concat(file),
+        })
+      }
+    } else {
+      this.setState(
+        {
+          path: file.id === '..' ? getParentPath(this.state.path) : file.path,
+        },
+        this._listFiles
+      )
+    }
+  }
+
+  _unselectFile = file => {
+    this.setState({
+      selectedFiles: filter(
+        this.state.selectedFiles,
+        ({ id }) => id !== file.id
+      ),
+    })
+  }
+
+  _unselectAllFiles = () => {
+    this.setState({
+      selectedFiles: undefined,
+    })
+  }
+
+  _selectAllFolderFiles = () => {
+    this.setState({
+      selectedFiles: (this.state.selectedFiles || []).concat(
+        filter(this._getSelectableFiles(), ({ path }) => !endsWith(path, '/'))
+      ),
+    })
+  }
+
+  // ---------------------------------------------------------------------------
+
+  render () {
+    const { backups } = this.props
+    const {
+      backup,
+      disk,
+      format,
+      partition,
+      partitions,
+      path,
+      scanDiskError,
+      listFilesError,
+      scanningFiles,
+      selectedFiles,
+    } = this.state
+    const noPartitions = isEmpty(partitions)
+
+    return (
+      <div>
+        <Select
+          labelKey='name'
+          onChange={this._onBackupChange}
+          optionRenderer={BACKUP_RENDERER}
+          options={backups}
+          placeholder={_('restoreFilesSelectBackup')}
+          value={backup}
+          valueKey='id'
+        />
+        {backup && [
+          <br />,
+          <Select
+            labelKey='name'
+            onChange={this._onDiskChange}
+            optionRenderer={diskOptionRenderer}
+            options={backup.disks}
+            placeholder={_('restoreFilesSelectDisk')}
+            value={disk}
+            valueKey='id'
+          />,
+        ]}
+        {scanDiskError && (
+          <span>
+            <Icon icon='error' /> {_('restoreFilesDiskError')}
+          </span>
+        )}
+        {disk &&
+          !scanDiskError &&
+          !noPartitions && [
+            <br />,
+            <Select
+              labelKey='name'
+              onChange={this._onPartitionChange}
+              optionRenderer={partitionOptionRenderer}
+              options={partitions}
+              placeholder={_('restoreFilesSelectPartition')}
+              value={partition}
+              valueKey='id'
+            />,
+          ]}
+        {(partition || (disk && !scanDiskError && noPartitions)) && [
+          <br />,
+          <Container>
+            <Row>
+              <Col size={10}>
+                <pre>
+                  {path} {scanningFiles && <Icon icon='loading' />}
+                  {listFilesError && <Icon icon='error' />}
+                </pre>
+              </Col>
+              <Col size={2}>
+                <span className='pull-right'>
+                  <Tooltip content={_('restoreFilesSelectAllFiles')}>
+                    <ActionButton
+                      handler={this._selectAllFolderFiles}
+                      icon='add'
+                      size='small'
+                    />
+                  </Tooltip>
+                </span>
+              </Col>
+            </Row>
+          </Container>,
+          <Select
+            labelKey='name'
+            onChange={this._onFileChange}
+            optionRenderer={fileOptionRenderer}
+            options={this._getSelectableFiles()}
+            placeholder={_('restoreFilesSelectFiles')}
+            value={null}
+            valueKey='id'
+          />,
+          <br />,
+          <div>
+            <span className='mr-1'>
+              <input
+                checked={format === 'zip'}
+                name='format'
+                onChange={this.linkState('format')}
+                type='radio'
+                value='zip'
+              />{' '}
+              ZIP
+            </span>
+            <span>
+              <input
+                checked={format === 'tar'}
+                name='format'
+                onChange={this.linkState('format')}
+                type='radio'
+                value='tar'
+              />{' '}
+              TAR
+            </span>
+          </div>,
+          <br />,
+          selectedFiles && selectedFiles.length ? (
+            <Container>
+              <Row>
+                <Col className='pl-0 pb-1' size={10}>
+                  <em>
+                    {_('restoreFilesSelectedFiles', {
+                      files: selectedFiles.length,
+                    })}
+                  </em>
+                </Col>
+                <Col size={2} className='text-xs-right'>
+                  <ActionButton
+                    handler={this._unselectAllFiles}
+                    icon='remove'
+                    size='small'
+                    tooltip={_('restoreFilesUnselectAll')}
+                  />
+                </Col>
+              </Row>
+              {map(selectedFiles, file => (
+                <Row key={file.id}>
+                  <Col size={10}>
+                    <pre>{file.path}</pre>
+                  </Col>
+                  <Col size={2} className='text-xs-right'>
+                    <ActionButton
+                      handler={this._unselectFile}
+                      handlerParam={file}
+                      icon='remove'
+                      size='small'
+                    />
+                  </Col>
+                </Row>
+              ))}
+            </Container>
+          ) : (
+            <em>{_('restoreFilesNoFilesSelected')}</em>
+          ),
+        ]}
+      </div>
+    )
+  }
+}

packages/xo-web/src/xo-app/backup-ng/index.js

@@ -184,7 +184,7 @@ const HEADER = (
       </Col>
       <Col mediumSize={9}>
         <NavTabs className='pull-right'>
-          <NavLink exact to='/backup-ng'>
+          <NavLink exact to='/backup-ng/overview'>
             <Icon icon='menu-backup-overview' /> {_('backupOverviewPage')}
           </NavLink>
           <NavLink to='/backup-ng/new'>
@@ -203,9 +203,10 @@ const HEADER = (
   </Container>
 )
 
-export default routes(Overview, {
+export default routes('overview', {
   ':id/edit': Edit,
   new: New,
+  overview: Overview,
   restore: Restore,
   'file-restore': FileRestore,
 })(({ children }) => (

packages/xo-web/src/xo-app/backup-ng/restore/delete-backups-modal-body.js

@@ -2,10 +2,12 @@ import _ from 'intl'
 import classNames from 'classnames'
 import Component from 'base-component'
 import React from 'react'
-import { FormattedDate } from 'react-intl'
 import { forEach, map, orderBy } from 'lodash'
 import { createFilter, createSelector } from 'selectors'
 import { Toggle } from 'form'
+import { getRenderXoItemOfType } from 'render-xo-item'
+
+const BACKUP_RENDERER = getRenderXoItemOfType('backup')
 
 const _escapeDot = id => id.replace('.', '\0')
 
@@ -61,22 +63,7 @@ export default class DeleteBackupsModalBody extends Component {
               onClick={this.toggleState(_escapeDot(backup.id))}
               type='button'
             >
-              <span
-                className='tag tag-info'
-                style={{ textTransform: 'capitalize' }}
-              >
-                {backup.mode}
-              </span>{' '}
-              <span className='tag tag-warning'>{backup.remote.name}</span>{' '}
-              <FormattedDate
-                value={new Date(backup.timestamp)}
-                month='long'
-                day='numeric'
-                year='numeric'
-                hour='2-digit'
-                minute='2-digit'
-                second='2-digit'
-              />
+              {BACKUP_RENDERER(backup)}
             </button>
           ))}
         </div>

packages/xo-web/src/xo-app/backup-ng/restore/restore-backups-modal-body.js

@@ -2,9 +2,11 @@ import _ from 'intl'
 import React from 'react'
 import Component from 'base-component'
 import StateButton from 'state-button'
+import { getRenderXoItemOfType } from 'render-xo-item'
 import { Select, Toggle } from 'form'
 import { SelectSr } from 'select-objects'
-import { FormattedDate } from 'react-intl'
+
+const BACKUP_RENDERER = getRenderXoItemOfType('backup')
 
 export default class RestoreBackupsModalBody extends Component {
   get value () {
@@ -15,26 +17,7 @@ export default class RestoreBackupsModalBody extends Component {
       <div>
         <div className='mb-1'>
           <Select
-            optionRenderer={backup => (
-              <span>
-                <span
-                  className='tag tag-info'
-                  style={{ textTransform: 'capitalize' }}
-                >
-                  {backup.mode}
-                </span>{' '}
-                <span className='tag tag-warning'>{backup.remote.name}</span>{' '}
-                <FormattedDate
-                  value={new Date(backup.timestamp)}
-                  month='long'
-                  day='numeric'
-                  year='numeric'
-                  hour='2-digit'
-                  minute='2-digit'
-                  second='2-digit'
-                />
-              </span>
-            )}
+            optionRenderer={BACKUP_RENDERER}
             options={this.props.data.backups}
             onChange={this.linkState('backup')}
             placeholder={_('importBackupModalSelectBackup')}

packages/xo-web/src/xo-app/dashboard/health/index.js

@@ -10,14 +10,20 @@ import TabButton from 'tab-button'
 import Tooltip from 'tooltip'
 import Upgrade from 'xoa-upgrade'
 import xml2js from 'xml2js'
-import { Card, CardHeader, CardBlock } from 'card'
 import { confirm } from 'modal'
-import { connectStore, formatSize, noop, resolveIds } from 'utils'
-import { Container, Row, Col } from 'grid'
-import { flatten, get, includes, isEmpty, map, mapValues } from 'lodash'
-import { FormattedRelative, FormattedTime } from 'react-intl'
-import { fromCallback } from 'promise-toolbox'
 import { SelectPool } from 'select-objects'
+import { fromCallback } from 'promise-toolbox'
+import { Container, Row, Col } from 'grid'
+import { Card, CardHeader, CardBlock } from 'card'
+import { FormattedRelative, FormattedTime } from 'react-intl'
+import { flatten, get, includes, isEmpty, map, mapValues } from 'lodash'
+import {
+  addSubscriptions,
+  connectStore,
+  formatSize,
+  noop,
+  resolveIds,
+} from 'utils'
 import {
   deleteMessage,
   deleteOrphanedVdis,
@@ -26,6 +32,7 @@ import {
   deleteVdi,
   deleteVm,
   isSrWritable,
+  subscribeSchedules,
 } from 'xo'
 import {
   areObjectsFetched,
@@ -383,6 +390,9 @@ const ALARM_COLUMNS = [
   },
 ]
 
+@addSubscriptions({
+  schedules: subscribeSchedules,
+})
 @connectStore(() => {
   const getOrphanVdiSnapshots = createGetObjectsOfType('VDI-snapshot')
     .filter([_ => !_.$snapshot_of && _.$VBDs.length === 0])
@@ -390,6 +400,15 @@ const ALARM_COLUMNS = [
   const getOrphanVmSnapshots = createGetObjectsOfType('VM-snapshot')
     .filter([snapshot => !snapshot.$snapshot_of])
     .sort()
+  const getLoneBackupSnapshots = createGetObjectsOfType('VM-snapshot').filter(
+    createSelector(
+      createCollectionWrapper((_, props) => map(props.schedules, 'id')),
+      scheduleIds => _ => {
+        const scheduleId = _.other['xo:backup:schedule']
+        return scheduleId !== undefined && !includes(scheduleIds, scheduleId)
+      }
+    )
+  )
   const getUserSrs = createGetObjectsOfType('SR').filter([isSrWritable])
   const getVdiSrs = createGetObjectsOfType('SR').pick(
     createSelector(getOrphanVdiSnapshots, snapshots => map(snapshots, '$SR'))
@@ -405,6 +424,7 @@ const ALARM_COLUMNS = [
     vdiOrphaned: getOrphanVdiSnapshots,
     vdiSr: getVdiSrs,
     vmOrphaned: getOrphanVmSnapshots,
+    vmBackupSnapshots: getLoneBackupSnapshots,
   }
 })
 export default class Health extends Component {
@@ -490,6 +510,11 @@ export default class Health extends Component {
     this._getPoolPredicate
   )
 
+  _getVmBackupSnapshots = createFilter(
+    () => this.props.vmBackupSnapshots,
+    this._getPoolPredicate
+  )
+
   _getAlertMessages = createFilter(
     () => this.props.alertMessages,
     this._getPoolPredicate
@@ -610,6 +635,24 @@ export default class Health extends Component {
             </Card>
           </Col>
         </Row>
+        <Row className='snapshot-vms'>
+          <Col>
+            <Card>
+              <CardHeader>
+                <Icon icon='vm' /> {_('vmSnapshotsRelatedToNonExistentBackups')}
+              </CardHeader>
+              <CardBlock>
+                <NoObjects
+                  collection={this._getVmBackupSnapshots()}
+                  columns={VM_COLUMNS}
+                  component={SortedTable}
+                  emptyMessage={_('noSnapshots')}
+                  shortcutsTarget='.snapshot-vms'
+                />
+              </CardBlock>
+            </Card>
+          </Col>
+        </Row>
         <Row>
           <Col>
             <Card>

packages/xo-web/src/xo-app/host/index.js

@@ -246,6 +246,13 @@ export default class Host extends Component {
                 }
               />{' '}
               <Text value={host.name_label} onChange={this._setNameLabel} />
+              {this.props.needsRestart && (
+                <Tooltip content={_('rebootUpdateHostLabel')}>
+                  <Link to={`/hosts/${host.id}/patches`}>
+                    <Icon icon='alarm' />
+                  </Link>
+                </Tooltip>
+              )}
             </h2>
             <span>
               <Text
@@ -291,12 +298,6 @@ export default class Host extends Component {
                     {missingPatches.length}
                   </span>
                 )}
-                {this.props.needsRestart &&
-                  isEmpty(missingPatches) && (
-                    <Tooltip content={_('rebootUpdateHostLabel')}>
-                      <Icon icon='alarm' />
-                    </Tooltip>
-                  )}
               </NavLink>
               <NavLink to={`/hosts/${host.id}/logs`}>
                 {_('logsTabName')}

packages/xo-web/src/xo-app/host/tab-patches.js

@@ -181,16 +181,15 @@ export default class HostPatches extends Component {
       <Container>
         <Row>
           <Col className='text-xs-right'>
-            {this.props.needsRestart &&
-              isEmpty(missingPatches) && (
-                <TabButton
-                  btnStyle='warning'
-                  handler={restartHost}
-                  handlerParam={host}
-                  icon='host-reboot'
-                  labelId='rebootUpdateHostLabel'
-                />
-              )}
+            {this.props.needsRestart && (
+              <TabButton
+                btnStyle='warning'
+                handler={restartHost}
+                handlerParam={host}
+                icon='host-reboot'
+                labelId='rebootUpdateHostLabel'
+              />
+            )}
             <TabButton
               disabled={!hasMissingPatches}
               btnStyle={hasMissingPatches ? 'primary' : undefined}

packages/xo-web/src/xo-app/menu/index.js

@@ -203,9 +203,31 @@ export default class Menu extends Component {
         ],
       },
       isAdmin && {
-        to: '/backup-ng',
+        to: '/backup-ng/overview',
         icon: 'menu-backup',
-        label: ['Backup NG'],
+        label: <span>Backup NG</span>,
+        subMenu: [
+          {
+            to: '/backup-ng/overview',
+            icon: 'menu-backup-overview',
+            label: 'backupOverviewPage',
+          },
+          {
+            to: '/backup-ng/new',
+            icon: 'menu-backup-new',
+            label: 'backupNewPage',
+          },
+          {
+            to: '/backup-ng/restore',
+            icon: 'menu-backup-restore',
+            label: 'backupRestorePage',
+          },
+          {
+            to: '/backup-ng/file-restore',
+            icon: 'menu-backup-file-restore',
+            label: 'backupFileRestorePage',
+          },
+        ],
       },
       isAdmin && {
         to: 'xoa/update',

packages/xo-web/src/xo-app/sr/tab-advanced.js

@@ -14,20 +14,25 @@ import { flowRight, isEmpty, keys, sum, values } from 'lodash'
 
 const COLUMNS = [
   {
+    itemRenderer: _ => <span>{_.name_label}</span>,
     name: _('srUnhealthyVdiNameLabel'),
-    itemRenderer: vdi => <span>{vdi.name_label}</span>,
-    sortCriteria: vdi => vdi.name_label,
+    sortCriteria: 'name_label',
   },
   {
-    name: _('srUnhealthyVdiSize'),
     itemRenderer: vdi => formatSize(vdi.size),
-    sortCriteria: vdi => vdi.size,
+    name: _('srUnhealthyVdiSize'),
+    sortCriteria: 'size',
   },
   {
-    name: _('srUnhealthyVdiDepth'),
     itemRenderer: (vdi, chains) => chains[vdi.uuid],
+    name: _('srUnhealthyVdiDepth'),
     sortCriteria: (vdi, chains) => chains[vdi.uuid],
   },
+  {
+    itemRenderer: _ => <Copiable tagName='div'>{_.uuid}</Copiable>,
+    name: _('srUnhealthyVdiUuid'),
+    sortCriteria: 'uuid',
+  },
 ]
 
 const UnhealthyVdiChains = flowRight(

yarn.lock

@@ -2566,7 +2566,7 @@ buffer-alloc-unsafe@^1.0.0:
   version "1.0.0"
   resolved "https://registry.yarnpkg.com/buffer-alloc-unsafe/-/buffer-alloc-unsafe-1.0.0.tgz#474aa88f34e7bc75fa311d2e6457409c5846c3fe"
 
-buffer-crc32@^0.2.1, buffer-crc32@^0.2.13:
+buffer-crc32@^0.2.1, buffer-crc32@^0.2.13, buffer-crc32@~0.2.3:
   version "0.2.13"
   resolved "https://registry.yarnpkg.com/buffer-crc32/-/buffer-crc32-0.2.13.tgz#0d333e3f00eac50aa1454abd30ef8c2a5d9a7242"
 
@@ -12011,6 +12011,12 @@ yargs@~3.10.0:
     decamelize "^1.0.0"
     window-size "0.1.0"
 
+yazl@^2.4.3:
+  version "2.4.3"
+  resolved "https://registry.yarnpkg.com/yazl/-/yazl-2.4.3.tgz#ec26e5cc87d5601b9df8432dbdd3cd2e5173a071"
+  dependencies:
+    buffer-crc32 "~0.2.3"
+
 zip-stream@^1.2.0:
   version "1.2.0"
   resolved "https://registry.yarnpkg.com/zip-stream/-/zip-stream-1.2.0.tgz#a8bc45f4c1b49699c6b90198baacaacdbcd4ba04"