Migrate server to ESM

Sorry for the very big commit that may lead to git log issues and merge
conflicts, but it's a major step forward:

 * Server can be faster at startup because imports() are async and we can
   easily lazy import big modules
 * Angular doesn't seem to support ES import (with .js extension), so we
   had to correctly organize peertube into a monorepo:
    * Use yarn workspace feature
    * Use typescript reference projects for dependencies
    * Shared projects have been moved into "packages", each one is now a
      node module (with a dedicated package.json/tsconfig.json)
    * server/tools have been moved into apps/ and is now a dedicated app
      bundled and published on NPM so users don't have to build peertube
      cli tools manually
    * server/tests have been moved into packages/ so we don't compile
      them every time we want to run the server
 * Use isolatedModule option:
   * Had to move from const enum to const
     (https://www.typescriptlang.org/docs/handbook/enums.html#objects-vs-enums)
   * Had to explictely specify "type" imports when used in decorators
 * Prefer tsx (that uses esbuild under the hood) instead of ts-node to
   load typescript files (tests with mocha or scripts):
     * To reduce test complexity as esbuild doesn't support decorator
       metadata, we only test server files that do not import server
       models
     * We still build tests files into js files for a faster CI
 * Remove unmaintained peertube CLI import script
 * Removed some barrels to speed up execution (less imports)

support/doc/development/lib.md

@@ -18,7 +18,7 @@ The complete types package is generated via:
 
 ```
 npm run generate-types-package 4.x.x
-cd packages/types/dist
+cd packages/types-generator/dist
 npm publish --access=public
 ```
 

support/doc/development/localization.md

@@ -26,7 +26,7 @@ Nothing to do here, Github will automatically send a webhook to Weblate that wil
 
 ## Support a new language
 
- * Add it to [/shared/models/i18n/i18n.ts](/shared/models/i18n/i18n.ts)
+ * Add it to [/packages/models/i18n/i18n.ts](/packages/models/i18n/i18n.ts)
  * Add it to [/scripts/build/client.sh](/scripts/build/client.sh)
  * Add it to [/client/angular.json](/client/angular.json)
  * Add it to [/scripts/i18n/update.sh](/scripts/i18n/update.sh)

support/doc/development/monitoring.md

@@ -13,11 +13,11 @@ npm run build -- --analyze-bundle && npm run client-report
 To benchmark the REST API and save result in `benchmark.json`:
 
 ```
-node dist/scripts/benchmark.js -o benchmark.json
+npm run benchmark-server -- -o benchmark.json
 ```
 
 You can also grep on a specific test:
 
 ```
-node dist/scripts/benchmark.js --grep homepage
+npm run benchmark-server -- --grep homepage
 ```

support/doc/development/server.md

@@ -1,11 +1,11 @@
-# Server code
+ # Server code
 
 ## Database model typing
 
 Sequelize models contain optional fields corresponding to table joins.
 For example, `VideoModel` has a `VideoChannel?: VideoChannelModel` field. It can be filled if the SQL query joined with the `videoChannel` table or empty if not.
 It can be difficult in TypeScript to understand if a function argument expects associations to be filled or not.
-To improve clarity and reduce bugs, PeerTube defines multiple versions of a database model depending on its associations in `server/types/models/`.
+To improve clarity and reduce bugs, PeerTube defines multiple versions of a database model depending on its associations in `server/server/types/models/`.
 These models start with `M` and by default do not include any association. `MVideo` for example corresponds to `VideoModel` without any association, where `VideoChannel` attribute doesn't exist. On the other hand, `MVideoWithChannel` is a `MVideo` that has a `VideoChannel` field. This way, a function that accepts `video: MVideoWithChannel` argument expects a video with channel populated. Main PeerTube code should never use `...Model` (`VideoModel`) database type, but always `M...` instead (`MVideo`, `MVideoChannel` etc).
 
 ## Add a new feature walkthrough
@@ -16,67 +16,67 @@ Some of these may be optional (for example your new endpoint may not need to sen
  * Configuration:
    - Add you new configuration key in `config/default.yaml` and `config/production.yaml`
    - If you configuration needs to be different in dev or tests environments, also update `config/dev.yaml` and `config/test.yaml`
-   - Load your configuration in `server/initializers/config.ts`
-   - Check new configuration keys are set in `server/initializers/checker-before-init.ts`
-   - You can also ensure configuration consistency in `server/initializers/checker-after-init.ts`
+   - Load your configuration in `server/server/initializers/config.ts`
+   - Check new configuration keys are set in `server/server/initializers/checker-before-init.ts`
+   - You can also ensure configuration consistency in `server/server/initializers/checker-after-init.ts`
    - If you want your configuration to be available in the client:
-     + Add your field in `shared/models/server/server-config.model.ts`
-     + Update `server/lib/server-config-manager.ts` to include your new configuration
+     + Add your field in `packages/models/server/server/server-config.model.ts`
+     + Update `server/server/lib/server-config-manager.ts` to include your new configuration
    - If you want your configuration to be updatable by the web admin in the client:
-     + Add your field in `shared/models/server/custom-config.model.ts`
-     + Add the configuration to the config object in the `server/controllers/api/config.ts` controller
+     + Add your field in `packages/models/server/server/custom-config.model.ts`
+     + Add the configuration to the config object in the `server/server/controllers/api/config.ts` controller
  * Controllers:
    - Create the controller file and fill it with your REST API routes
    - Import and use your controller in the parent controller
  * Middlewares:
-   - Create your validator middleware in `server/middlewares/validators` that will be used by your controllers
-   - Add your new middleware file `server/middlewares/validators/index.ts` so it's easier to import
-   - Create the entry in `server/types/express.d.ts` to attach the database model loaded by your middleware to the express response
+   - Create your validator middleware in `server/server/middlewares/validators` that will be used by your controllers
+   - Add your new middleware file `server/server/middlewares/validators/index.ts` so it's easier to import
+   - Create the entry in `server/server/types/express.d.ts` to attach the database model loaded by your middleware to the express response
  * Validators:
-   - Create your validators that will be used by your middlewares in `server/helpers/custom-validators`
+   - Create your validators that will be used by your middlewares in `server/server/helpers/custom-validators`
  * Typescript models:
-   - Create the API models (request parameters or response) in `shared/models`
+   - Create the API models (request parameters or response) in `packages/models`
    - Add your models in `index.ts` of current directory to facilitate the imports
  * Sequelize model (BDD):
    - If you need to create a new table:
-     + Create the Sequelize model in `server/models/`:
+     + Create the Sequelize model in `server/server/models/`:
        * Create the `@Column`
        * Add some indexes if you need
        * Create static methods to load a specific from the database `loadBy...`
        * Create static methods to load a list of models from the database `listBy...`
        * Create the instance method `toFormattedJSON` that creates the JSON to send to the REST API from the model
-     + Add your new Sequelize model to `server/initializers/database.ts`
-     + Create a new file in `server/types` to define multiple versions of your Sequelize model depending on database associations
-     + Add this new file to `server/types/*/index.ts` to facilitate the imports
+     + Add your new Sequelize model to `server/server/initializers/database.ts`
+     + Create a new file in `server/server/types` to define multiple versions of your Sequelize model depending on database associations
+     + Add this new file to `server/server/types/*/index.ts` to facilitate the imports
      + Create database migrations:
-       * Create the migration file in `server/initializers/migrations` using raw SQL (copy the same SQL query as at PeerTube startup)
-       * Update `LAST_MIGRATION_VERSION` in `server/initializers/constants.ts`
+       * Create the migration file in `server/server/initializers/migrations` using raw SQL (copy the same SQL query as at PeerTube startup)
+       * Update `LAST_MIGRATION_VERSION` in `server/server/initializers/constants.ts`
    - If updating database schema (adding/removing/renaming a column):
-     + Update the sequelize models in `server/models/`
+     + Update the sequelize models in `server/server/models/`
      + Add migrations:
        * Create the migration file in `initializers/migrations` using Sequelize Query Interface (`.addColumn`, `.dropTable`, `.changeColumn`)
-       * Update `LAST_MIGRATION_VERSION` in `server/initializers/constants.ts`
+       * Update `LAST_MIGRATION_VERSION` in `server/server/initializers/constants.ts`
  * Notifications:
-   - Create the new notification model in `shared/models/users/user-notification.model.ts`
-   - Create the notification logic in `server/lib/notifier/shared`:
+   - Create the new notification model in `packages/models/users/user-notification.model.ts`
+   - Create the notification logic in `server/server/lib/notifier/shared`:
      + Email subject has a common prefix (defined by the admin in PeerTube configuration)
-   - Add your notification to `server/lib/notifier/notifier.ts`
-   - Create the email template in `server/lib/emails`:
+   - Add your notification to `server/server/lib/notifier/notifier.ts`
+   - Create the email template in `server/server/lib/emails`:
      + A text version is automatically generated from the HTML
      + The template usually extends `../common/grettings` that already says "Hi" and "Cheers". You just have to write the title and the content blocks that will be inserted in the appropriate places in the HTML template
    - If you need to associate a new table with `userNotification`:
      + Associate the new table in `UserNotificationModel` (don't forget the index)
-     + Add the object property in the API model definition (`shared/models/users/user-notification.model.ts`)
+     + Add the object property in the API model definition (`packages/models/users/user-notification.model.ts`)
      + Add the object in `UserNotificationModel.toFormattedJSON`
      + Handle this new notification type in client (`UserNotificationsComponent`)
      + Handle the new object property in client model (`UserNotification`)
  * Tests:
-   - Create your command class in `shared/server-commands/` that will wrap HTTP requests to your new endpoint
+   - Create your command class in `packages/server-commands/` that will wrap HTTP requests to your new endpoint
    - Add your command file in `index.ts` of current directory
-   - Instantiate your command class in `shared/server-commands/server/server.ts`
-   - Create your test file in `server/tests/api/check-params` to test middleware validators/authentification/user rights (offensive tests)
-   - Add it to `server/tests/api/check-params/index.ts`
-   - Create your test file in `server/tests/api` to test your new endpoints
+   - Instantiate your command class in `packages/server-commands/server/server/server.ts`
+   - Create your test file in `server/server/tests/api/check-params` to test middleware validators/authentification/user rights (offensive tests)
+   - Add it to `server/server/tests/api/check-params/index.ts`
+   - Create your test file in `server/server/tests/api` to test your new endpoints
    - Add it to `index.ts` of current directory
-   - Add your notification test in `server/tests/api/notifications`
+   - Add your notification test in `server/server/tests/api/notifications`
  * Update REST API documentation in `support/doc/api/openapi.yaml`

support/doc/development/tests.md

@@ -8,7 +8,7 @@ Prepare PostgreSQL user so PeerTube can delete/create the test databases:
 sudo -u postgres createuser you_username --createdb --superuser
 ```
 
-Prepare databases:
+Prepare the databases:
 
 ```bash
 npm run clean:server:test
@@ -45,22 +45,19 @@ sudo apt-get install parallel libimage-exiftool-perl
 
 ### Test
 
-To run all test suites:
+To run all test suites (can be long!):
 
 ```bash
 npm run test # See scripts/test.sh to run a particular suite
 ```
 
-Most of tests can be run using:
+To run a specific test:
 
 ```bash
-TS_NODE_TRANSPILE_ONLY=true npm run mocha -- --timeout 30000 --exit -r ts-node/register -r tsconfig-paths/register --bail server/tests/api/videos/video-transcoder.ts
-```
+npm run mocha -- --exit --bail packages/tests/src/your-test.ts
 
-`server/tests/api/activitypub` tests will need different options:
-
-```
-TS_NODE_FILES=true mocha -- --timeout 30000 --exit -r ts-node/register -r tsconfig-paths/register --bail server/tests/api/activitypub/security.ts
+# For example
+npm run mocha -- --exit --bail packages/tests/src/api/videos/single-server.ts
 ```
 
 ### Configuration