This PR builds up on the pass-through DB driver to a fully functioning DB driver implementation via our RPC layer.
To keep things separate from the plugin RPC API, and have the ability to move fast with changes, a separate field Driver is added to MattermostPlugin. Typically the field which is required to be compatible are the API and Helpers. It would be well-documented that Driver is purely for internal use by Mattermost plugins.
A new Driver interface was created which would have a client and server implementation. Every object (connection, statement, etc.) is created and added to a map on the server side. On the client side, the wrapper structs hold the object id, and communicate via the RPC API using this id.
When the server gets the object id, it picks up the appropriate object from its map and performs the operation, and sends back the data.
Some things that need to be handled are errors. Typical error types like pq.Error and mysql.MySQLError are registered with encoding/gob. But for error variables like sql.ErrNoRows, a special integer is encoded with the ErrorString struct. And on the cilent side, the integer is checked, and the appropriate error variable is returned.
Some pending things:
- Context support. This is tricky. Since context.Context is an interface, it's not possible to marshal it. We have to find a way to get the timeout value from the context and pass it.
- RowsColumnScanType(rowsID string, index int) reflect.Type API. Again, reflect.Type is an interface.
- Master/Replica API support.
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* Replacing require.nil in app layer
* Fixing some tests
* Fixing tests
* Reverting the error correction, defering it for another PR
* Fixing golangci-lint errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
th.InitBasic initializes users, channels, and teams. But a lot of
tests don't use it at all, causing unncessary delays. A single InitBasic
call takes around 400ms. This PR removes 71 of them, approximately shaving
off 28 seconds. Arguably, this would be more because some tests run in loops.
* Pass device ID
* dont use device id as way of detecting
* fix spelling mistake
* update layers
* fix test
* fix linting
* save schema
* put columns in correct place
* fix linting
* update
* upgrade go change
* use props
* fix stuff
* update session tests
* address PR comments
* address PR comments
* MM-22706: pass along set_online flag in websocket response
To let the client know whether a user has created a post without
being online or not, we get the set_online query param and
pass it down to the websocket event being passed down to the client.
With this PR, the "data" field of the `posted` event will contain
a `set_online` boolean field set to true/false depending on the
query_param set_online value set in the createPost call.
* Setting to false for auto responder
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-16479: panic on io.Copy error in test
I'm unable to reproduce the original report on MM-16479, but in a bid to help track down a potential failure in the future, I've added a panic on err returned from `io.Copy` within the plugin, as this seems like the only path on which the hook could return successfully but fail to show the updated contents.
* assert number of copied bytes
* log n on io.Copy error too
* MM-16997: Simplified TestUploadFiles
Tickets:
https://mattermost.atlassian.net/browse/MM-16997https://mattermost.atlassian.net/browse/MM-16760
- The tests now fully buffer the data vefore uploading it. The prior
code was much more complex because it was intended to eventually
mature into generic client code; not a priority, definitely not
valuable for this test.
- Also improved error handling in TestHookFileWillBeUploaded for
MM-16760
* PR feedback: typo
* MM-14575 - Automatically serve static files for plugins
* Added static handler for plugin public files
* Added StaticFilesPath method to Environment for use by MainRouter
* Added "static_files" property to Manifest Server
* Added unit tests for these changes
* MM-14575: Adding comment for cache control value
* MM-14575: Moved Static Plugin Request handler to plugin_requests
* Updated testing
* MM-14575: Removing the StaticFiles from Manifest Server
* MM-14575: Removing static files from test
* MM-14575: Updating static files test
* MM14575: Removing cache directive from plugin static files
* MM14575: Moving plugin public directory to root
* MM-14575: Updating tests for changed public directory
* MM-14575: Moved compileGo to a common utils package for tests
* MM-14575: Moving plugins initialization to InitPlugins find in tests
* Update utils/test_files_compiler.go
Adding Copyright header
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* MM-14575: Consistent usage of static vs public name
* Removing spurious newline
* Comment typo
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* Removing spurious new line
Co-Authored-By: happygaijin <happygaijin@users.noreply.github.com>
* MM14575: Adding a test to make sure only public files can be requested
* MM-14575 Adding a test for redirects on public files
* [MM-13828] Running tests from a new temp folder with all test resources
Possible fix for #10132
All packages which have a TestMain and use testlib.MainHelper will have a new current working directory which will have all the test
resources copied.
Note: default.json is copied as config.json as well to make sure tests don't have any impact due to changes in config by devs
* [MM-13828] Added TestMain to remaining packages to use testlib.MainHelper
This makes sure tests from all packages run with same test resources, setup in a new temp folder for each package
* Updated Jenkins file to not not config/default.json
This makes sure CI has same config files as a dev's machine
* [MM-13828] Changes requested from code review
Added accessor methods to testlib.MainHelper for accessing members
Fixed some broken tests due to change in cwd while tests run
Some other code refactoring and improvements
* [MM-13828] Added new factory method with options for creating test main helper and some code refactoring
testlib.NewMainHelperWithOptions supports options to turn on/off test dependencies and environment setup
Some other code refactoring
* Exporting members of testlib.MainHelper to make enterprise tests work
* Fixed gofmt error
* [MM-13828] removed unwanted dependency on plugins directory while setting up test resources
* [MM-13828] Fixed some tests failing due to them being running from temp folder
* [MM-13828] Some code changes suggested in PR review
* Fixed gofmt error
* Rerouted the unit test logs through t.Log
* resolving merge confilvts
* Update testing.go
* Update helper_test.go
* Added godocs for NewTestingLogger
* Added go docs for NewTestingLogger
* Resolving conflicts
* MM-10856: deduplicate posts with the same pending post id
Leverage a fixed size cache with a window of 30 seconds to deduplicate posts received by a single app server. Clients that duplicate the same pending post id will see one request potentially delayed until the first finishes, after which the same payload should be returned by both. Duplicate posts outside the 30 second window will not be de-duplicated.
Note that the cache is not synchronized between app servers. In an HA cluster consisting of more than one app server, sticky load balancing (e.g. hashing by IP or Session ID) is required to route the users to the same app instance for multiple requests.
Other options considered for this feature:
* adding a column to the `Posts` table: rejected as being too heavyweight
* maintaining a `PendingPostIds` table: similarly rejected for the database impact
* using the pending post id as the post id and relying on the unique constraints on the Post table: rejected for being difficult to show that it's safe to use a client-provided value as the row identifier
* utils/lru: simplify to ttl internally and for new methods
* move seenPendingPostIdsCache to App.Server
* just fail concurrent post requests, vs. trying to wait
* add debug log when create post is deduplicated
* guard app plugins with mutex
Shutting down the app could race with a goroutine that uses the plugins environment, since we shut down the plugins first before cleaning up goroutines.
* fix go vet issues
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
Fix code issues in channel_test.go
Fix Channel Test Issues detected by Megacheck
Fix API Emoji Test Issues detected by Megacheck
Fixed API Issues Reported by Megacheck
Fixed App issues reported by megacheck
Remaining fixes
removed test added by mistake from old HEAD
gofmt
Store Fixes
simplified returns
Fix test for multi member channel delete
revert to delete unused function
* ensure plugin is always shutdown
Once we call `.client.Client()` the plugin has started, and must be shut
down. `newSupervisor` sometimes returned with an error (and without a
reference to the supervisor), leaving the client running indefinitely.
* Clarify the documentation to explain that plugin hooks will not trigger until `OnActivate` returns successfully, and will stop triggering just before `OnDeactivate` is called.
* test for plugin deadlock
* plugin/environment.go: switch to sync.Map
From: https://golang.org/pkg/sync/#Map
> If a goroutine holds a RWMutex for reading and another goroutine might call Lock, no goroutine should expect to be able to acquire a read lock until the initial read lock is released. In particular, this prohibits recursive read locking. This is to ensure that the lock eventually becomes available; a blocked Lock call excludes new readers from acquiring the lock.
The previous `RWMutex` was not safe given that we effectively acquired read locks recursively (hook -> api -> hook). This worked up until we activated or deactivated plugins, tried to acquire a write lock, and the plugin used the API to effectively trigger another hook.
Switching to sync.Map avoids this by divesting the need to lock at all, avoiding the potential for a recursive lock in the first place.
