* MM-14246 - Plugin framework: support transactional semantics with KV Store
Rename old, new variable names
Moving New function to the bottom
* Made CompareAndUpdate sync, updated tests
* Removed going through channel in CompareAndSetPluginKey
* Inserting new key when oldValue is nil to KVCompareAndSet
* Updated error text to include CompareAndSet
* MM-14575 - Automatically serve static files for plugins
* Added static handler for plugin public files
* Added StaticFilesPath method to Environment for use by MainRouter
* Added "static_files" property to Manifest Server
* Added unit tests for these changes
* MM-14575: Adding comment for cache control value
* MM-14575: Moved Static Plugin Request handler to plugin_requests
* Updated testing
* MM-14575: Removing the StaticFiles from Manifest Server
* MM-14575: Removing static files from test
* MM-14575: Updating static files test
* MM14575: Removing cache directive from plugin static files
* MM14575: Moving plugin public directory to root
* MM-14575: Updating tests for changed public directory
* MM-14575: Moved compileGo to a common utils package for tests
* MM-14575: Moving plugins initialization to InitPlugins find in tests
* Update utils/test_files_compiler.go
Adding Copyright header
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* MM-14575: Consistent usage of static vs public name
* Removing spurious newline
* Comment typo
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* Removing spurious new line
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* MM14575: Adding a test to make sure only public files can be requested
* MM-14575 Adding a test for redirects on public files
* bots model, store and api (#9903)
* bots model, store and api
Fixes: MM-13100, MM-13101, MM-13103, MM-13105, MMM-13119
* uncomment tests incorrectly commented, and fix merge issues
* add etags support
* add missing licenses
* remove unused sqlbuilder.go (for now...)
* rejig permissions
* split out READ_BOTS into READ_BOTS and READ_OTHERS_BOTS, the latter
implicitly allowing the former
* make MANAGE_OTHERS_BOTS imply MANAGE_BOTS
* conform to general rest api pattern
* eliminate redundant http.StatusOK
* Update api4/bot.go
Co-Authored-By: lieut-data <jesse.hallam@gmail.com>
* s/model.UserFromBotModel/model.UserFromBot/g
* Update model/bot.go
Co-Authored-By: lieut-data <jesse.hallam@gmail.com>
* Update model/client4.go
Co-Authored-By: lieut-data <jesse.hallam@gmail.com>
* move sessionHasPermissionToManageBot to app/authorization.go
* use api.ApiSessionRequired for createBot
* introduce BOT_DESCRIPTION_MAX_RUNES constant
* MM-13512 Prevent getting a user by email based on privacy settings (#10021)
* MM-13512 Prevent getting a user by email based on privacy settings
* Add additional config settings to tests
* upgrade db to 5.7 (#10019)
* MM-13526 Add validation when setting a user's Locale field (#10022)
* Fix typos (#10024)
* Fixing first user being created with system admin privilages without being explicity specified. (#10014)
* Revert "Support for Embeded chat (#9129)" (#10017)
This reverts commit 3fcecd521a.
* s/DisableBot/UpdateBotActive
* add permissions on upgrade
* Update NOTICE.txt (#10054)
- add new dependency (text)
- handle switch to forked dependency (go-gomail -> go-mail)
- misc copyright owner updates
* avoid leaking bot knowledge without permission
* [GH-6798] added a new api endpoint to get the bulk reactions for posts (#10049)
* 6798 added a new api to get the bulk reactions for posts
* 6798 added the permsission check before getting the reactions
* GH-6798 added a new app function for the new endpoint
* 6798 added a store method to get reactions for multiple posts
* 6798 connected the app function with the new store function
* 6798 fixed the review comments
* MM-13559 Update model.post.is_valid.file_ids.app_error text per report (#10055)
Ticket: https://mattermost.atlassian.net/browse/MM-13559
Report: https://github.com/mattermost/mattermost-server/issues/10023
* Trigger Login Hooks with OAuth (#10061)
* make BotStore.GetAll deterministic even on duplicate CreateAt
* fix spurious TestMuteCommandSpecificChannel test failure
See
https://community-daily.mattermost.com/core/pl/px9p8s3dzbg1pf3ddrm5cr36uw
* fix race in TestExportUserChannels
* TestExportUserChannels: remove SaveMember call, as it is redundant and used to be silently failing anyway
* MM-13117: bot tokens (#10111)
* eliminate redundant Client/AdminClient declarations
* harden TestUpdateChannelScheme to API failures
* eliminate unnecessary config restoration
* minor cleanup
* make TestGenerateMfaSecret config dependency explicit
* TestCreateUserAccessToken for bots
* TestGetUserAccessToken* for bots
* leverage SessionHasPermissionToUserOrBot for user token APIs
* Test(Revoke|Disable|Enable)UserAccessToken
* make EnableUserAccessTokens explicit, so as to not rely on local config.json
* uncomment TestResetPassword, but still skip
* mark assert(Invalid)Token as helper
* fix whitespace issues
* fix mangled comments
* MM-13116: bot plugin api (#10113)
* MM-13117: expose bot API to plugins
This also changes the `CreatorId` column definition to allow for plugin
ids, as the default unless the plugin overrides is to use the plugin id
here. This branch hasn't hit master yet, so no migration needed.
* gofmt issues
* expunge use of BotList in plugin/client API
* introduce model.BotGetOptions
* use botUserId term for clarity
* MM-13129 Adding functionality to deal with orphaned bots (#10238)
* Add way to list orphaned bots.
* Add /assign route to modify ownership of bot accounts.
* Apply suggestions from code review
Co-Authored-By: crspeller <crspeller@gmail.com>
* MM-13120: add IsBot field to returned user objects (#10103)
* MM-13104: forbid bot login (#10251)
* MM-13104: disallow bot login
* fix shadowing
* MM-13136 Disable user bots when user is disabled. (#10293)
* Disable user bots when user is disabled.
* Grammer.
Co-Authored-By: crspeller <crspeller@gmail.com>
* Fixing bot branch for test changes.
* Don't use external dependancies in bot plugin tests.
* Rename bot CreatorId to OwnerId
* Adding ability to re-enable bots
* Fixing IsBot to not attempt to be saved to DB.
* Adding diagnostics and licencing counting for bot accounts.
* Modifying gorp to allow reading of '-' fields.
* Removing unnessisary nil values from UserCountOptions.
* Changing comment to GoDoc format
* Improving user count SQL
* Some improvments from feedback.
* Omit empty on User.IsBot
* Added support for PostActions in ephemeral posts
The general approach is that we take all the metadata that DoPostAction
needs to process client DoPostActionRequests, and store it in a
serialized, encrypted Cookie field, in the PostAction struct.
The client then must send it back, and it is then used to process
PostActions as a fallback top the metadata in the database.
This PR adds a new config setting, `ServiceSettings.ActionCookieSecret`.
In a cluster environment it must be the same for all instances.
- Added type PostActionCookie, and a Cookie string to PostAction.
- Added App.AddActionCookiesToPost.
- Use App.AddActionCookiesToPost in api4.createEphemeralPost,
App.SendEphemeralPost, App.UpdateEphemeralPost.
- Added App.DoPostActionWithCookie to process incoming requests with
cookies. For backward compatibility, it prefers the metadata in the
database; falls back to cookie.
- Added plugin.API.UpdateEphemeralPost and plugin.API.DeleteEphemeralPost.
- Added App.encryptActionCookie/App.decryptActionCookie.
* Style
* Fixed an unfortunate typo, tested with matterpoll
* minor PR feedback
* Fixed uninitialized Context
* Fixed another test failure
* Fixed permission check
* Added api test for DoPostActionWithCookie
* Replaced config.ActionCookieSecret with Server.PostActionCookieSecret
Modeled after AsymetricSigningKey
* style
* Set DeleteAt in DeleteEphemeralPost
* PR feedback
* Removed deadwood comment
* Added EXPERIMENTAL comment to the 2 APIs in question
* [MM-13828] Running tests from a new temp folder with all test resources
Possible fix for #10132
All packages which have a TestMain and use testlib.MainHelper will have a new current working directory which will have all the test
resources copied.
Note: default.json is copied as config.json as well to make sure tests don't have any impact due to changes in config by devs
* [MM-13828] Added TestMain to remaining packages to use testlib.MainHelper
This makes sure tests from all packages run with same test resources, setup in a new temp folder for each package
* Updated Jenkins file to not not config/default.json
This makes sure CI has same config files as a dev's machine
* [MM-13828] Changes requested from code review
Added accessor methods to testlib.MainHelper for accessing members
Fixed some broken tests due to change in cwd while tests run
Some other code refactoring and improvements
* [MM-13828] Added new factory method with options for creating test main helper and some code refactoring
testlib.NewMainHelperWithOptions supports options to turn on/off test dependencies and environment setup
Some other code refactoring
* Exporting members of testlib.MainHelper to make enterprise tests work
* Fixed gofmt error
* [MM-13828] removed unwanted dependency on plugins directory while setting up test resources
* [MM-13828] Fixed some tests failing due to them being running from temp folder
* [MM-13828] Some code changes suggested in PR review
* Fixed gofmt error
* Fix shadowed variables in cmd package
* Fix shadowed variables in plugin package
* Fix shadowed variables in store package
* Fix shadowed variables in web package
* Changes as requested
Signed-off-by: Hanzei <hanzei@mailbox.org>
* Fix build
* Remove unnessary statements
* Use require all the time
* Fix build
* Rename variables according to feedback
* Fix NPE
* Changes as requested
* Prevent clobbering of new fields by old plugins.
* Apply suggestions from code review
Spelling and copy paste error.
Co-Authored-By: crspeller <crspeller@gmail.com>
* MM-7633: Optimize memory utilization during file uploads
Refactored the file upload code to reduce redundant buffering and stream
directly to the file store. Added tests.
Benchmark results:
```
levs-mbp:mattermost-server levb$ go test -v -run nothing -bench Upload -benchmem ./app
...
BenchmarkUploadFile/random-5Mb-gif-raw-ish_DoUploadFile-4 10 122598031 ns/op 21211370 B/op 1008 allocs/op
BenchmarkUploadFile/random-5Mb-gif-raw_UploadFileTask-4 100 20211926 ns/op 5678750 B/op 126 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFiles-4 2 1037051184 ns/op 81806360 B/op 3705013 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFileTask-4 2 933644431 ns/op 67015868 B/op 3704410 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw-ish_DoUploadFile-4 100 13110509 ns/op 6032614 B/op 8052 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw_UploadFileTask-4 100 10729867 ns/op 1738303 B/op 125 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFiles-4 2 925274912 ns/op 70326352 B/op 3718856 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFileTask-4 2 995033336 ns/op 58113796 B/op 3710943 allocs/op
BenchmarkUploadFile/zero-10Mb-raw-ish_DoUploadFile-4 30 50777211 ns/op 54791929 B/op 2714 allocs/op
BenchmarkUploadFile/zero-10Mb-raw_UploadFileTask-4 50 36387339 ns/op 10503920 B/op 126 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFiles-4 30 48657678 ns/op 54791948 B/op 2719 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFileTask-4 50 37506467 ns/op 31492060 B/op 131 allocs/op
...
```
https://mattermost.atlassian.net/browse/MM-7633https://github.com/mattermost/mattermost-server/issues/7801
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]
- [x] Added or updated unit tests (required for all new features)
- [ ] Added API documentation (required for all new APIs)
- [ ] All new/modified APIs include changes to the drivers
*N/A*???
- [x] Includes text changes and localization file ([.../i18n/en.json](https://github.com/mattermost/mattermost-server/blob/master/i18n/en.json)) updates
Overview of changes:
- api4
- Replaced `uploadFile` handler with `uploadFileStream` that reduces
unnecessary buffering.
- Added/refactored tests for the new API.
- Refactored apitestlib/Check...Status functions.
- app
- Added App.UploadFileTask, a more efficient refactor of UploadFile.
- Consistently set `FileInfo.HasPreviewImage`
- Added benchmarks for the new and prior implementations
- Replaced passing around `*image.Image` with `image.Image` in the
existing code.
- model
- Added a more capable `client4.UploadFiles` API to match the new server
API’s capabilities.
- I18n
- Replaced `api.file.upload_file.bad_parse.app_error` with a more generic
`api.file.upload_file.read_request.app_error`
- plugin
- Removed type `plugin.multiPluginHookRunnerFunc` in favor of using
`func(hooks Hooks) bool` explicitly, to help with testing
- tests
- Added test files for testing images
Still remaining, but can be separate PRs - please let me know the preferred
course of action
- Investigate JS client API - how does it do multipart?
- Performance loss from old code on (small) image processing?
- Deprecate the old functions, change other API implementations to use
UploadFileTask
Definitely separate future PRs - should I file tickets foe these?
- Only invoke t.readAll() if there are indeed applicable plugins to run
- Find a way to leverage goexif buffer rather than re-reading
Suggested long-term improvements - should I file separate tickets for these?
- Actually allow uploading of large (GB-sized) files. This may require a
change in how the file is passed to plugins.
- (Many) api4 tests should probably be subtests and share a server setup -
will be much faster
- Performance improvements in image processing (goexif/thumbnail/preview)
(maybe use https://mattermost.atlassian.net/browse/MM-10188 for this)
Questions:
1. I am commiting MBs of test images, are there better alternatives? I can
probably create much less dense images that would take up considerably less
space, even at pretty large sizes
2. I18n: Do I need to do anything special for the string change? Or just wait
until it gets picked up and translated/updated?
3. The image dimensions are flipped in resulting FileInfo to match the actual
orientation. Is this by design? Should add a test for it, perhaps?
4. What to do in the case of partial success? S3 but not DB, some files but not
others? For now, just doing what the old code did, I think.
5. Make maxUploadDrainBytes configurable? Also, should this be the systemic
behavior of all APIs with non-empty body? Otherwise dropped altogether?
Check all other ioutil.ReadAll() from sockets. Find a way to set a total
byte limit on request Body?
* WIP - Fixed for GetPluginsEnvironment() changes
* WIP - PR feedback
1. Refactored checkHTTPStatus to improve failure messages
2. Use `var name []type` rather than `name := ([]type)(nil)`
3. Replaced single-letter `p` with a more intention-revealing `part`
4. Added tests for full image size, `HasPreviewImage`
* WIP - rebased (c.Session->c.App.Session)
* WIP - PR feedback: eliminated use of Request.MultipartReader
Instead of hacking the request object to use r.MultipartReader now have own
functions `parseMultipartRequestHeader` and `multipartReader` eliminating the
need to hack the request object to use Request.MultipartReader limitations.
* WIP - PR feedback: UploadFileX with functional options
* WIP - PR feedback: style
* WIP - PR feedback: errors cleanup
* WIP - clarified File Upload benchmarks
* WIP - PR feedback: display the value of erroneous formname
* WIP - PR feedback: fixed handling of multiple channel_ids
* WIP - rebased from master - fixed tests
* PR Feedback
* PR feedback - moved client4.UploadFiles to _test for now
* wip
* wip
* After running make plugin-mocks
* Add
TestUpdateUserActive and run make pluginapi
* Adding plugin_api_test.go
* Better formatting of code using gofmt
* Fix tests and run make pluginapi
* Specify the minimum server version on the comments
* Include more tests as per the CR
* Fix tests
* Checking err.Id intsead of err & Removed comments and trailing spaces
* wip
* Fix tests as per CR and spaces
* Make changes to tests as per CR
* refactor GetDirectChannel and CreateDirectChannel in one function
* remove CreateDirectChannel plugin api and update GetDirectChannel and GetGroupChannel plugin api
* update tests
* Add interactive dialogs
* Fix unit test
* Updates per feedback
* Fix typo
* Updates per feedback, add icon_url and error returns
* Updates per feedback
* Update per feedback
* [MM-11861] Design & implement a better way for plugins to update their own configuration
Added GetPluginConfig and SavePluginConfig plugin APIs.
Added test cases for testing new APIs.
* Fixed gofmt error
* Minor changes requested in PR
* Change GetTeamMembers() and GetPublicChannelsForTeam() arguments to page, perPage for plugin API
* Add test for GetPublicChannelsForTeam()
* Add test for GetTeamMembers()
* Changes as requested
* Change return from GetPublicChannelsForTeam() to []*model.Channel
* Add GetTeamsUnreadForUser to plugin api
* Remove teamIdToExclude from plugin method GetTeamsUnreadForUser
* Add minimum server version to plugin API doc of GetTeamsUnreadForUser
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
* adds GetUsersInChannelByStatus to plugin api with generated rpc code.
* fixed typo in comment with actual func name
* replaced Response model with AppError in output of GetUsersInChannelByStatus
* removed etag param from GetUsersInChannelByStatus since it is not used
* plugin api for GetUsersInChannelByStatus updated to take the limit, conforming to the app api.
* fixed an issue in my own logic on app/plugin integration.
* adds GetUsersInChannelByStatus to plugin api with generated rpc code.
* fixed typo in comment with actual func name
* replaced Response model with AppError in output of GetUsersInChannelByStatus
* removed etag param from GetUsersInChannelByStatus since it is not used
* plugin api for GetUsersInChannelByStatus updated to take the limit, conforming to the app api.
* fixed an issue in my own logic on app/plugin integration.
* GetUsersInChannelByStatus changed to more generic GetUsersInChannel which takes a sortBy parameter, allowing for more granular/extensible sorting functionality in the future
* GetUsersInChannel accepts sort parameter of 'username' and 'status'. Both values are consts in model pkg.
* Documents minimum server version for GetUsersInChannel.
* adds GetUsersInChannelByStatus to plugin api with generated rpc code.
* fixed typo in comment with actual func name
* replaced Response model with AppError in output of GetUsersInChannelByStatus
* removed etag param from GetUsersInChannelByStatus since it is not used
* plugin api for GetUsersInChannelByStatus updated to take the limit, conforming to the app api.
* fixed an issue in my own logic on app/plugin integration.
* adds GetUsersInChannelByStatus to plugin api with generated rpc code.
* Resolved conflict on rebase
* replaced Response model with AppError in output of GetUsersInChannelByStatus
* removed etag param from GetUsersInChannelByStatus since it is not used
* plugin api for GetUsersInChannelByStatus updated to take the limit, conforming to the app api.
* fixed an issue in my own logic on app/plugin integration.
* GetUsersInChannelByStatus changed to more generic GetUsersInChannel which takes a sortBy parameter, allowing for more granular/extensible sorting functionality in the future
* GetUsersInChannel accepts sort parameter of 'username' and 'status'. Both values are consts in model pkg.
* Documents minimum server version for GetUsersInChannel.
* replaces GetUsersInChannel from #9608 / #9643 with sortBy functionality
