Commit Graph

7924 Commits

Author SHA1 Message Date
Jelle Besseling
0305db28c9
Add support for saving video files to object storage (#4290)
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
Chocobozzz
f88ae8f5bc
Disable openapi generation
Until https://github.com/OpenAPITools/openapi-generator/issues/8663 is
fixed
2021-08-11 14:58:37 +02:00
Chocobozzz
de34551da3
Typo 2021-08-11 14:43:03 +02:00
Chocobozzz
eb8bd9ab1a
Try to fix openapi script 2021-08-11 14:38:07 +02:00
Chocobozzz
5b075bc549
Fix search index tests 2021-08-11 11:49:01 +02:00
Chocobozzz
134006b0d4
Update client dependencies 2021-08-11 11:49:01 +02:00
Chocobozzz
31aa391d13
Update server dependencies 2021-08-11 11:49:00 +02:00
Mark Van den Borre
99c05b3dcc Update FAQ.md
s/franglais/english/
2021-08-11 10:46:14 +02:00
ROBERT MCDOWELL
863c1bc6dc
fixed typo (#4312)
* Fixed typo
2021-08-11 10:45:49 +02:00
Chocobozzz
18998c45c0
Correctly handle large HLS files for redundancy 2021-08-06 15:25:45 +02:00
Chocobozzz
a1c63fe1a2
Fix send rtmp command 2021-08-06 15:06:47 +02:00
Chocobozzz
7942e72e38
Fix cli build 2021-08-06 14:39:28 +02:00
Chocobozzz
679c12e69c
Improve target bitrate calculation 2021-08-06 14:13:26 +02:00
Chocobozzz
c826f34a45
Limit live bitrate 2021-08-06 10:39:40 +02:00
Chocobozzz
421ff4618d
Merge branch 'release/3.3.0' into develop 2021-08-05 15:17:18 +02:00
Chocobozzz
ff5f37e4f3
Fix sha fallback 2021-08-05 15:17:09 +02:00
Chocobozzz
e83d06a702
Remove deprecated transcoding job names 2021-08-05 14:29:44 +02:00
Chocobozzz
45570e9397
Fix upload without files 2021-08-05 14:24:03 +02:00
Chocobozzz
27db78400c
Fix backend channel name validator consistency 2021-08-05 13:54:35 +02:00
Chocobozzz
352819ef92
Do not index remote actors 2021-08-05 13:29:55 +02:00
ROBERT MCDOWELL
da948b75ca Standard output type syslog is obsolete in systemed
removing the setting
StandardOutput
StandardError
2021-08-05 10:58:05 +02:00
Chocobozzz
3e254de8be HLS v1 support 2021-08-05 10:57:49 +02:00
Chocobozzz
ff4de38385
Filter videos by live in custom markup 2021-08-02 17:10:07 +02:00
Chocobozzz
3da38d6e9f
Fetch things in bulk for the homepage 2021-08-02 15:40:09 +02:00
Chocobozzz
200eaf5152
Remove thumbnail flash for autoplay 2021-08-02 11:46:11 +02:00
Chocobozzz
4a7056b283
Update translations 2021-08-02 10:49:05 +02:00
Phongpanot
90ea6ef1f1 Translated using Weblate (Thai)
Currently translated at 52.8% (952 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/th/
2021-08-02 10:41:15 +02:00
Marc Strange
7f02b833ec Translated using Weblate (Catalan)
Currently translated at 36.8% (663 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Marc Strange
1fdea5f5d0 Translated using Weblate (Catalan)
Currently translated at 100.0% (118 of 118 strings)

Translation: PeerTube/player
Translate-URL: https://weblate.framasoft.org/projects/peertube/player/ca/
2021-08-02 10:41:15 +02:00
Marc Strange
6a95289c84 Translated using Weblate (Catalan)
Currently translated at 33.9% (611 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Grzesiek11
4c37cbfbe2 Translated using Weblate (Polish)
Currently translated at 99.7% (1795 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pl/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy
1becfc7cab Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Александр
e1220d60f2 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Laurent Ettouati
3ef9aceef9 Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
Александр
7aa6525309 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Simon Brosdetzko
a9a6ecf8be Translated using Weblate (German)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/de/
2021-08-02 10:41:15 +02:00
Racida S
3cfda8ea54 Translated using Weblate (Kabyle)
Currently translated at 95.5% (1720 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/kab/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy
706bc8aedc Translated using Weblate (Vietnamese)
Currently translated at 100.0% (246 of 246 strings)

Translation: PeerTube/server
Translate-URL: https://weblate.framasoft.org/projects/peertube/server/vi/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy
93a8ab2a74 Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Laurent Ettouati
2b7b9dd7bb Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
framabr
af85e02ab3 Translated using Weblate (Portuguese (Brazil))
Currently translated at 53.9% (971 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pt_BR/
2021-08-02 10:41:15 +02:00
Marc Strange
8ac253b598 Translated using Weblate (Catalan)
Currently translated at 32.6% (588 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Jeff Huang
2909b999fe Translated using Weblate (Chinese (Traditional))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/zh_Hant/
2021-08-02 10:41:15 +02:00
Александр
bdb7d35cf0 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
josé m
2f2b7b6b1b Translated using Weblate (Galician)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gl/
2021-08-02 10:41:15 +02:00
GunChleoc
a082d6fd1a Translated using Weblate (Gaelic)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Berto Te
26bafbb701 Translated using Weblate (Spanish)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/es/
2021-08-02 10:41:15 +02:00
GunChleoc
b68ec72ed6 Translated using Weblate (Gaelic)
Currently translated at 98.6% (1776 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Chocobozzz
d26836cd95
Refactor notifier 2021-08-02 10:39:51 +02:00
Chocobozzz
2bee9db56a
Update plugin doc 2021-07-30 13:58:18 +02:00