* Convert bool string comparisons to strconv.ParseBool for REST parameters
* Log failed bool conversions
* Rename errors, changed log levels
* drop strconv.ParseBool error handling
If the query string parameter is omitted, strconv.ParseBool returns an error for the empty strings, which spams the logs. Instead, just assume the default semantics of a `false` return value if an error occurs.
* allow randomized Client4 booleans
It's hard to test api4's handling of the various boolean input values
accepted. Extend Client4 with support for overriding how it builds those
strings, and pick a random value on test startup.
* gofmt -s
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Add auditing to server CLI.
Also:
- simplify auditing in API layer
- reduce number of AddMeta calls
- have models serialize themselves
- more consistent field naming
* New auditing API outputting to syslog via TLS
* New config section for specifying remote syslog server IP, port, and cert.
* Legacy audit API retained for access history feature
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM 10658 Change config fields to pointers (#8898)
* Change fields of config structs to pointers and set defaults
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix tests that go broken during switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Apply changes of current master while switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix new config pointer uses
* Fix app tests
* Fix mail test
* remove debugging statement
* fix TestUpdateConfig
* assign config consistently
* initialize AmazonS3Region in TestS3TestConnection
* initialize fields for TestEmailTest
* fix TestCheckMandatoryS3Fields
* MM-7633: Optimize memory utilization during file uploads
Refactored the file upload code to reduce redundant buffering and stream
directly to the file store. Added tests.
Benchmark results:
```
levs-mbp:mattermost-server levb$ go test -v -run nothing -bench Upload -benchmem ./app
...
BenchmarkUploadFile/random-5Mb-gif-raw-ish_DoUploadFile-4 10 122598031 ns/op 21211370 B/op 1008 allocs/op
BenchmarkUploadFile/random-5Mb-gif-raw_UploadFileTask-4 100 20211926 ns/op 5678750 B/op 126 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFiles-4 2 1037051184 ns/op 81806360 B/op 3705013 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFileTask-4 2 933644431 ns/op 67015868 B/op 3704410 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw-ish_DoUploadFile-4 100 13110509 ns/op 6032614 B/op 8052 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw_UploadFileTask-4 100 10729867 ns/op 1738303 B/op 125 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFiles-4 2 925274912 ns/op 70326352 B/op 3718856 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFileTask-4 2 995033336 ns/op 58113796 B/op 3710943 allocs/op
BenchmarkUploadFile/zero-10Mb-raw-ish_DoUploadFile-4 30 50777211 ns/op 54791929 B/op 2714 allocs/op
BenchmarkUploadFile/zero-10Mb-raw_UploadFileTask-4 50 36387339 ns/op 10503920 B/op 126 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFiles-4 30 48657678 ns/op 54791948 B/op 2719 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFileTask-4 50 37506467 ns/op 31492060 B/op 131 allocs/op
...
```
https://mattermost.atlassian.net/browse/MM-7633https://github.com/mattermost/mattermost-server/issues/7801
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]
- [x] Added or updated unit tests (required for all new features)
- [ ] Added API documentation (required for all new APIs)
- [ ] All new/modified APIs include changes to the drivers
*N/A*???
- [x] Includes text changes and localization file ([.../i18n/en.json](https://github.com/mattermost/mattermost-server/blob/master/i18n/en.json)) updates
Overview of changes:
- api4
- Replaced `uploadFile` handler with `uploadFileStream` that reduces
unnecessary buffering.
- Added/refactored tests for the new API.
- Refactored apitestlib/Check...Status functions.
- app
- Added App.UploadFileTask, a more efficient refactor of UploadFile.
- Consistently set `FileInfo.HasPreviewImage`
- Added benchmarks for the new and prior implementations
- Replaced passing around `*image.Image` with `image.Image` in the
existing code.
- model
- Added a more capable `client4.UploadFiles` API to match the new server
API’s capabilities.
- I18n
- Replaced `api.file.upload_file.bad_parse.app_error` with a more generic
`api.file.upload_file.read_request.app_error`
- plugin
- Removed type `plugin.multiPluginHookRunnerFunc` in favor of using
`func(hooks Hooks) bool` explicitly, to help with testing
- tests
- Added test files for testing images
Still remaining, but can be separate PRs - please let me know the preferred
course of action
- Investigate JS client API - how does it do multipart?
- Performance loss from old code on (small) image processing?
- Deprecate the old functions, change other API implementations to use
UploadFileTask
Definitely separate future PRs - should I file tickets foe these?
- Only invoke t.readAll() if there are indeed applicable plugins to run
- Find a way to leverage goexif buffer rather than re-reading
Suggested long-term improvements - should I file separate tickets for these?
- Actually allow uploading of large (GB-sized) files. This may require a
change in how the file is passed to plugins.
- (Many) api4 tests should probably be subtests and share a server setup -
will be much faster
- Performance improvements in image processing (goexif/thumbnail/preview)
(maybe use https://mattermost.atlassian.net/browse/MM-10188 for this)
Questions:
1. I am commiting MBs of test images, are there better alternatives? I can
probably create much less dense images that would take up considerably less
space, even at pretty large sizes
2. I18n: Do I need to do anything special for the string change? Or just wait
until it gets picked up and translated/updated?
3. The image dimensions are flipped in resulting FileInfo to match the actual
orientation. Is this by design? Should add a test for it, perhaps?
4. What to do in the case of partial success? S3 but not DB, some files but not
others? For now, just doing what the old code did, I think.
5. Make maxUploadDrainBytes configurable? Also, should this be the systemic
behavior of all APIs with non-empty body? Otherwise dropped altogether?
Check all other ioutil.ReadAll() from sockets. Find a way to set a total
byte limit on request Body?
* WIP - Fixed for GetPluginsEnvironment() changes
* WIP - PR feedback
1. Refactored checkHTTPStatus to improve failure messages
2. Use `var name []type` rather than `name := ([]type)(nil)`
3. Replaced single-letter `p` with a more intention-revealing `part`
4. Added tests for full image size, `HasPreviewImage`
* WIP - rebased (c.Session->c.App.Session)
* WIP - PR feedback: eliminated use of Request.MultipartReader
Instead of hacking the request object to use r.MultipartReader now have own
functions `parseMultipartRequestHeader` and `multipartReader` eliminating the
need to hack the request object to use Request.MultipartReader limitations.
* WIP - PR feedback: UploadFileX with functional options
* WIP - PR feedback: style
* WIP - PR feedback: errors cleanup
* WIP - clarified File Upload benchmarks
* WIP - PR feedback: display the value of erroneous formname
* WIP - PR feedback: fixed handling of multiple channel_ids
* WIP - rebased from master - fixed tests
* PR Feedback
* PR feedback - moved client4.UploadFiles to _test for now
I believe we don't need to force downloading of shared public files, because `writeFileResponse` method will compare its MIME type with `MEDIA_CONTENT_TYPES` and output it inline or as attachment automatically.
Usually we are share images and it is much more convenient to see them without downloading.
* factor out GetSubpathFromConfig
* mv web/subpath.go to utils/subpath.go
* serve up web, api and ws on /subpath if configured
* pass config to utils.RenderWeb(App)?Error
This allows the methods to extract the configured subpath and redirect
to the appropriate `/subpath/error` handler.
* ensure GetSubpathFromConfig returns trailing slashes deterministically
* fix error 404 handling
* redirect /subpath to /subpath/
This is necessary for the static handler to match, otherwise none of the
registered routes find anything. This also makes it no longer necessary
to add trailing slashes in the root router.
* Prepare file upload infrastructure for Data Retention.
This commit prepares the file upload infrastructure for the data
retention feature that is under construction. Changes are:
* Move file management code to utils to allow access to it from jobs.
* From now on, store all file uploads in a top level folder which is the
date of the day on which they were uploaded.
This commit is based on Harrison Healey's branch, but updated to work
with the latest master.
* Use NewAppError
* Revert "PLT-5765 Passed SiteURL to SendNotifications (#5705)"
This reverts commit a8e68bd890.
* Removed automatic configuration of SiteURL
* Reverted unintentional config changes
* Updated help text
* Added link to docs in Site URL warning
* Fixed merge conflict
