Commit Graph

58 Commits

Author SHA1 Message Date
Chocobozzz
1bfb791e05
Integrate transcription in PeerTube 2024-06-28 08:44:58 +02:00
lutangar
ef14cf4a5c
feat(transcription): groundwork
chore: fiddling around some more

chore: add ctranslate2 and timestamped

chore: add performance markers

chore: refactor test

chore: change worflow name

chore: ensure Python3

chore(duration): convert to chai/mocha syntahx

chore(transcription): add individual tests for others transcribers

chore(transcription): implement formats test of all implementations

Also compare result of other implementation to the reference implementation

chore(transcription): add more test case with other language and models size and local model

chore(test): wip ctranslate 2 adapat

chore(transcription): wip transcript file and benchmark

chore(test): clean a bit

chore(test): clean a bit

chore(test): refacto timestamed spec

chore(test): update workflow

chore(test): fix glob expansion with sh

chore(test): extract some hw info

chore(test): fix async tests

chore(benchmark): add model info

feat(transcription): allow use of a local mode in timestamped-whisper

feat(transcription): extract run and profiling info in own value object

feat(transcription): extract run concept in own class an run more bench

chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): creates own package readme

docs(transcription): add local model usage

docs(transcription): update README

fix(transcription): use fr video for better comparison

chore(transcription): make openai comparison passed

docs(timestamped): clea

chore(transcription): change transcribers transcribe method signature

Introduce whisper builtin model.

fix(transcription): activate language detection

Forbid transcript creation without a language.
Add `languageDetection` flag to an engine and some assertions.

Fix an issue in `whisper-ctranslate2` :
https://github.com/Softcatala/whisper-ctranslate2/pull/93

chore(transcription): use PeerTube time helpers instead of custom ones

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

chore(transcription): use PeerTube UUID helpers

chore(transcription): enable CER evaluation

Thanks to this recent fix in Jiwer <3
https://github.com/jitsi/jiwer/issues/873

chore(jiwer): creates JiWer package

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ?

chore(JiWer): add usage in README

docs(jiwer): update JiWer readme

chore(transcription): use FunMOOC video in fixtures

chore(transcription): add proper english video fixture

chore(transcription): use os tmp directory where relevant

chore(transcription): fix jiwer cli test reference.txt

chore(transcription): move benchmark out of tests

chore(transcription): remove transcription workflow

docs(transcription): add benchmark info

fix(transcription): use ms precision in other transcribers

chore(transcription): simplify most of the tests

chore(transcription): remove slashes when building path with join

chore(transcription): make fromPath method async

chore(transcription): assert path to model is a directory for CTranslate2 transcriber

chore(transcription): ctranslate2 assertion

chore(transcription): ctranslate2 assertion

chore(transcription): add preinstall script for Python dependencies

chore(transcription): add download and unzip utils functions

chore(transcription): add download and unzip utils functions

chore(transcription): download & unzip models fixtures

chore(transcription): zip

chore(transcription): raise download file test timeout

chore(transcription): simplify download file test

chore(transcription): add transcriptions test to CI

chore(transcription): raise test preconditions timeout

chore(transcription): run preinstall scripts before running ci

chore(transcription): create dedicated tmp folder for transcriber tests

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): use short video for local model test

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): setup verbosity based on NODE_ENV value
2024-06-28 08:43:40 +02:00
Chocobozzz
3a4992633e
Migrate server to ESM
Sorry for the very big commit that may lead to git log issues and merge
conflicts, but it's a major step forward:

 * Server can be faster at startup because imports() are async and we can
   easily lazy import big modules
 * Angular doesn't seem to support ES import (with .js extension), so we
   had to correctly organize peertube into a monorepo:
    * Use yarn workspace feature
    * Use typescript reference projects for dependencies
    * Shared projects have been moved into "packages", each one is now a
      node module (with a dedicated package.json/tsconfig.json)
    * server/tools have been moved into apps/ and is now a dedicated app
      bundled and published on NPM so users don't have to build peertube
      cli tools manually
    * server/tests have been moved into packages/ so we don't compile
      them every time we want to run the server
 * Use isolatedModule option:
   * Had to move from const enum to const
     (https://www.typescriptlang.org/docs/handbook/enums.html#objects-vs-enums)
   * Had to explictely specify "type" imports when used in decorators
 * Prefer tsx (that uses esbuild under the hood) instead of ts-node to
   load typescript files (tests with mocha or scripts):
     * To reduce test complexity as esbuild doesn't support decorator
       metadata, we only test server files that do not import server
       models
     * We still build tests files into js files for a faster CI
 * Remove unmaintained peertube CLI import script
 * Removed some barrels to speed up execution (less imports)
2023-08-11 15:02:33 +02:00
Chocobozzz
d732ec7b46
Rename "videos" config to "web-videos" 2023-07-11 11:23:51 +02:00
kontrollanten
156cdbac22
object-storage: @aws-sdk/lib-storage for multipart (#4903)
* object-storage: @aws-sdk/lib-storage for multipart

* gitignore: add .DS_Store

* test(object-storage): remove only

* test(object-storage/multipart): generate video

* fix lint issue

* test(obj-storage/video): ensure file size

* Styling

Co-authored-by: Chocobozzz <me@florianbigard.com>
2022-04-19 15:22:18 +02:00
Chocobozzz
6627dbc957
Move types package in packages/ 2021-12-28 10:18:15 +01:00
lutangar
b8fa3e8c73 refactor(types): create dedicated folder for types package src
fix guide examples and add types package readme

refactor(tsconfig): move back base tsconfig to base directory
2021-12-16 10:08:43 +01:00
lutangar
8b03e2ce1a feat(types): create peertube-types package
Create dedicated Typescript "types" configuration file for each "projects".
Create a types folder which includes every useful ts definition:
- client
- server
- shared
Add tooling to create a proper package, extract dependencies, etc...
Add CI Github task.

refactor(types): publish types package in release script
2021-12-16 10:08:43 +01:00
lutangar
06aad80165 chore(refactor): remove shared folder dependencies to the server
Many files from the `shared` folder were importing files from the `server` folder.
When attempting to use Typescript project references to describe dependencies,
it highlighted a circular dependency beetween `shared` <-> `server`.

The Typescript project forbid such usages.
Using project references greatly improve performance by rebuilding only
the updated project and not all source files.
> see https://www.typescriptlang.org/docs/handbook/project-references.html
2021-12-16 10:08:43 +01:00
Chocobozzz
662351e626 Update embed webpack config 2021-05-27 16:00:13 +02:00
kontrollanten
ca87329289
Don't pause video upon modal open (#3909)
* client: dont pause video upon modal open

closes #3816

* client(video/watch): remove obsolete code

* gitignore: add vscode and vim swp files
2021-04-08 13:37:57 +02:00
Chocobozzz
c7f36e4f48
Fix transcoding 2020-01-29 16:54:52 +01:00
Julien Maulny
e063f46a0b Add generated files to gitignore to have a clean stage 2019-10-19 11:54:14 +02:00
LoveIsGrief
6aa5414813 Autoplay next recommended video (#2137)
* Start working on autoplay of next video

* Ignore changes made by gitpod

* Apply changes from PR#1370

* Correct the spelling of recommendations

* Fix linting errors

* Move boolean check to existing onEnded handler

* Pick a random video until the recommendations are improved

* Add simple tests for autoPlayNextVideo

* Fix lint

...again
2019-09-24 08:48:01 +02:00
Chocobozzz
7e9d3f259d Move CLI dependencies in their own package.json 2019-05-27 14:56:06 +02:00
Chocobozzz
3acc508440
Upgrade sequelize 2019-04-24 16:26:21 +02:00
Chocobozzz
7f8f8bdb4a
HLS is only supported by ffmpeg 4
Because of c8f625f529
2019-02-12 11:14:36 +01:00
Felix Ableitner
edb4ffc7e0 Set bitrate limits for transcoding (fixes #638) (#1135)
* Set bitrate limits for transcoding (fixes #638)

* added optimization script and test, changed stuff

* fix test, improve docs

* re-add optimize-old-videos script

* added documentation

* Don't optimize videos without valid UUID, or redundancy videos

* move getUUIDFromFilename

* fix tests?

* update torrent and file size, some more fixes/improvements

* use higher bitrate for high fps video, adjust bitrates

* add test video

* don't throw error if resolution is undefined

* generate test fixture on the fly

* use random noise video for bitrate test, add promise

* shorten test video to avoid timeout

* use existing function to optimize video

* various fixes

* increase test timeout

* limit test fixture size, add link

* test fixes

* add await

* more test fixes, add -b:v parameter

* replace ffmpeg wiki link

* fix ffmpeg params

* fix unit test

* add test fixture to .gitgnore

* add video transcoding fps model

* add missing file
2018-10-08 16:26:04 +02:00
Brad Johnson
7f5f4152a4 Refactor: Separated "Other Videos" section into a dedicated component/service (#969)
* Separated "Other Videos" section into a dedicated component/service

I'm currently working on some proof-of-concepts for recommendation
providers that could work with PeerTube to provide useful video
suggestions to the user.

As a first step, I want to have great clarity about how PeerTube,
itself, will surface these videos to the user.

With this branch, I'm refactoring the "recommendations" to make it
easier to swap out different recommender implementations quickly.

Stop recommender from including the video that's being watched.

Ensure always 5 recommendations

* Treat recommendations as a stream of values, rather than a single async value.

* Prioritize readability over HTTP response size early-optimization.

* Simplify pipe
2018-08-31 17:19:21 +02:00
Chocobozzz
288178bf02
Try to fix travis tests 2018-08-10 17:49:12 +02:00
Jorropo
c674185eb2 Add npm debug log to gitignore and organize 2018-07-26 14:46:18 +02:00
Chocobozzz
9c2e0dbfa9
Update iso639 translations for french and deutch 2018-06-19 14:02:57 +02:00
Chocobozzz
989e526abf
Prepare i18n files 2018-06-05 08:43:01 +02:00
Chocobozzz
fd5e57bbe2
Docker config directory as volume 2018-05-07 18:41:10 +02:00
Chocobozzz
29173696f7
Update gitignore 2018-05-07 13:06:56 +02:00
Chocobozzz
4919b6304f
Fix nginx configuration that do not work with import-videos script 2018-03-01 17:14:57 +01:00
Chocobozzz
c8a5b03785
Fix gitignore 2018-02-16 19:21:42 +01:00
Chocobozzz
ecb4e35f4e
Add ability to reset our password 2018-01-30 13:27:07 +01:00
Dimitri Gilbert
8c4890cbfe Issue #196 : Allow to cancel an upload (#221)
* issue #196

* fixed missattribution of var

* fix styling issue

* renamed videoUpload to videoUploadObservable

* added created path to gitignore

* changed uploadCancel method name to cancelUpload
2018-01-23 15:01:38 +01:00
Chocobozzz
f05a1c30c1
Don't show videos of remote instance after unfollow 2018-01-18 15:42:20 +01:00
Chocobozzz
fd206f0b2d
Add ability to update some configuration keys 2018-01-17 10:41:27 +01:00
Chocobozzz
519b8df5f8
Bumped to version 0.0.7-alpha 2018-01-15 11:55:27 +01:00
Chocobozzz
6b2ef589ed
Prepare production workflow 2018-01-15 11:28:41 +01:00
Chocobozzz
a22bfc3ebc
Refractor peertube videojs plugin 2018-01-09 15:29:05 +01:00
Chocobozzz
c5911fd347
Begin to add avatar to actors 2017-12-29 19:10:13 +01:00
Chocobozzz
08535e56aa
Prepare release script 2017-10-31 18:57:32 +01:00
Chocobozzz
93e1258c7c Move video file metadata in their own table
Will be used for user video quotas and multiple video resolutions
2017-08-25 11:36:23 +02:00
Chocobozzz
407c4473ad Fix upgrade script \n 2017-07-23 09:43:44 +02:00
Chocobozzz
f981dae861 Add previews cache system between pods 2017-07-12 11:56:02 +02:00
Chocobozzz
65fcc3119c First typescript iteration 2017-05-20 09:57:40 +02:00
Chocobozzz
6a94a109b4 Server: add video preview 2016-11-16 20:29:26 +01:00
Chocobozzz
81ca2cd3f4 Fix gitignores 2016-11-06 21:25:21 +01:00
Chocobozzz
2c8a2f7fcc Add videos to gitignore 2016-11-01 18:15:35 +01:00
Chocobozzz
469526e14d Update gitignore 2016-10-21 13:13:51 +02:00
Chocobozzz
03dd4b9e34 Remove useless things from gitignore 2016-10-07 14:54:47 +02:00
Chocobozzz
a6375e6966 Merge branch 'master' into webseed-merged 2016-10-02 15:39:09 +02:00
Chocobozzz
4b931f7df9 gitignore: torrent/ -> torrents/ 2016-09-23 17:15:28 +02:00
Chocobozzz
eb4f957eca Client: add requests stats page 2016-09-23 17:09:38 +02:00
Chocobozzz
088a967fe0 Tags directory in gitignore 2016-08-28 11:34:52 +02:00
Chocobozzz
052937db8a First draft using only webseed for server 2016-07-27 21:21:57 +02:00