Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* MM-33818: Add replica lag metric
We add two new metrics for monitoring replica lag:
- Monitor absolute lag based on binlog distance/transaction queue length.
- Monitor time taken for the replica to catch up.
To achieve this, we add a config setting to run a user defined SQL query
on the database.
We need to specify a separate datasource field as part of the config because
in some databases, querying the replica lag value requires elevated credentials
which are not needed for usual running of the application, and can even be a security risk.
Arguably, a peculiar part of the design is the requirement of the query output to be in a (node, value)
format. But since from the application, the SQL query is a black box and the user can set any query
they want, we cannot, in any way templatize this.
And as an extra note, freno also does it in a similar way.
The last bit is because we need to have a separate datasources, now we consume one extra connection
rather than sharing it with the pool. This is an unfortunate result of the design, and while one extra
connection doesn't make much of a difference in a single-tenant scenario. It does make so, in a multi-tenant scenario.
But in a multi-tenant scenario, the expectation would already be to use a connection pool. So this
is not a big concern.
https://mattermost.atlassian.net/browse/MM-33818
```release-note
Two new gauge metrics were added:
mattermost_db_replica_lag_abs and mattermost_db_replica_lag_time, both
containing a label of "node", signifying which db host is the metric from.
These metrics signify the replica lag in absolute terms and in the time dimension
capturing the whole picture of replica lag.
To use these metrics, a separate config section ReplicaLagSettings was added
under SqlSettings. This is an array of maps which contain three keys: DataSource,
QueryAbsoluteLag, and QueryTimeLag. Each map entry is for a single replica instance.
DataSource contains the DB credentials to connect to the replica instance.
QueryAbsoluteLag is a plain SQL query that must return a single row of which the first column
must be the node value of the Prometheus metric, and the second column must be the value of the lag.
QueryTimeLag is the same as above, but used to measure the time lag.
As an example, for AWS Aurora instances, the QueryAbsoluteLag can be:
select server_id, highest_lsn_rcvd-durable_lsn as bindiff from aurora_global_db_instance_status() where server_id=<>
and QueryTimeLag can be:
select server_id, visibility_lag_in_msec from aurora_global_db_instance_status() where server_id=<>
For MySQL Group Replication, the absolute lag can be measured from the number of pending transactions
in the applier queue:
select member_id, count_transaction_remote_in_applier_queue FROM performance_schema.replication_group_member_stats where member_id=<>
Overall, what query to choose is left to the administrator, and depending on the database and need, an appropriate
query can be chosen.
```
* Trigger CI
* Fix tests
* address review comments
* Remove t.Parallel
It was spawning too many connections,
and overloading the docker container.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34080: Removing sqlite entirely
The initial commit missed removing this blank import.
So the library still remained in our vendor directory.
Removing it for good now.
Bye bye sqlite.
https://mattermost.atlassian.net/browse/MM-34080
* fix go.mod
This is a deadlock due to reversed locking order of the SidebarChannels
and SidebarCategories table.
I could not find the exact culprit query from the deadlock output
because it only shows the last query a transaction is running. And from looking
at the code, the only query that runs "UPDATE SidebarCategories SET DisplayName = ?, Sorting = ? WHERE Id = ?"
is UpdateSidebarCategories. But for the deadlock to happen, it has to lock
SidebarChannels _first_, and then _then_ lock SidebarCategories.
Looking a bit more throughly, I found that DeleteSidebarCategory does indeed
lock the tables in an inverse way and if DeleteSidebarCategory runs concurrently with
UpdateSidebarCategories, they will deadlock.
Here's how it will happen.
```
tx1
DELETE FROM SidebarChannels WHERE CategoryId = 'xx';
tx2
UPDATE SidebarCategories SET DisplayName='dn' WHERE Id='xx';
tx2
DELETE FROM SidebarChannels WHERE (ChannelId IN ('yy') AND CategoryId = 'xx');
tx1
DELETE FROM SidebarCategories WHERE Id = 'xx';
```
And then we see:
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
To fix this, we simply reorder the Delete query to lock the SidebarCategories first,
and then SidebarChannels.
In fact, any transaction updating/deleting rows from those two tables should always operate on that order
if possible.
https://mattermost.atlassian.net/browse/MM-31396
```release-note
Fixed a database deadlock that can happen if a sidebar category is updated and deleted at the same time.
```
* MM-33789: Revert fallback to master for GetAllProfilesInChannel
This fixes a regression introduced in https://github.com/mattermost/mattermost-server/pull/16911.
It was causing problems with too many invalidations and overloading the writer instance for big installations.
Reverting this does not affect correctness at all because it was done out of abundance of caution and the
idea at that point was it was to be done for all caches.
https://mattermost.atlassian.net/browse/MM-33789
```release-note
NONE
```
* fix gofmt issues
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529b.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Create basic make commands for configuring golang-migrate
* Showcase full flow with new migrations
* Apply PR suggestions
* Migrate over team members
* Update mocks
* Fix specs
* Move columns that added after table creation onto separate stmts
* Put back gorp table definitions
* Fix issues with golang-migrate that not tracks underlying db driver
* Help prompt after new migration and consistent checksum for bindata
* Put gorp mapping back
* Apply PR suggestiong
* Close migrations after they run
* Add migration file to bindata check
* Updates needed
* Reset store_test
* Add copyright
* Apply PR suggestions
* Fix new circleci check
* Put back upgrade step for backwards comp
* Add store test to test migration directions
* Apply PR suggestions
* Add go-bindata to tools
* Apply PR suggestios
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Update services/searchengine/bleveengine/bleve.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Adding tests requested in the PR review
* fixing tests
* Adding a feature flag to avoid indexing files before the feature is released
* Fixing i18n
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-32893: Refactor the getting of DB handle
We add a method in *SqlStore that returns the correct DB handle
depending on the context.
https://mattermost.atlassian.net/browse/MM-32893
```release-note
NONE
```
* fix receiver name
* Migrate channel to waitgroup for access the store in parallel
* Addressing PR review comments
* Using a cleanest way of group errors
* Reverting go.mod change
* MM-30882: Fix read-after-write issue for demoting user
In (*App).DemoteUserToGuest, we would demote a user, and then immediately
read it back to do future operations from the user. This reading back
of the user had the effect of sticking the old value into the cache
after which it would never be updated.
There was another issue along with this, which was when the invalidation
message would broadcast across the cluster, it would hit the cache invalidation
problem where an unrelated store call would miss the cache because
it was invalidated, and then again read from replica and stick the old value.
To fix all these, we return the new value directly from the store method
to avoid having the app to read it again.
And we add a map in the localcache layer which tracks invalidations made,
and then switch to use master if it's true.
The core change is fairly limited, but due to changing the store method signatures,
a lot of code needed to be updated to pass "context.Background". Therefore the PR
just "appears" to be big, but the main changes are limited to app/user.go,
sqlstore/user_store.go and user_layer.go
https://mattermost.atlassian.net/browse/MM-30882
```release-note
Fix an issue where demoting a user to guest would not take effect in
an environment with read replicas.
```
* Fix concurrent map access
* Fixing mistakes
* fix tests
* Optimise creation of dm
* Handle direct channels with the same user
* Cover GetMany with specs and add it on tha cache layer as well
* Fix specs by handling user dming themselves
* Apply PR suggestions
* Apply PR suggestions
* Use require.NoError instead of require.Nil on userstore test
* Improve readability of GetOrCreateDirectChannel
* Apply PR suggestions
* Update layers
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Replacing require.Nil with require.NoError
* More replacements
* More Nil/NotNill to NoError/Error
* Other detected errors
* renaming apperr to err
* Removed not needed line
* Rename old appErr variables that are no longer model.AppError values
* Fixing tiny typo
* Reverting changes outside the store (accidentally added)
* Apply suggestions from code review
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
