Commit Graph

54 Commits

Author SHA1 Message Date
Hossein
daec0b3201 All Sections to SubSections (#16917)
* initial

* Revert "initial"

This reverts commit 3d631aeecd.

* [MM-32352] Add Experimental Subsections BACKEND (#16887)

Automatic Merge

* update appiface

* Fix app layers

* Ancillary Permissions on backend (#17061)

Automatic Merge

* [MM-32799] Add About Section (#17015)

* Add About Section

* add mock key

* Update role.go

* Update role.go

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* [MM-33437] Fix config access tags for experimental settings (#17111)

Automatic Merge

* [MM-32794] Reporting Sub Section (#17035)

* test

* revert

* add permissions

* add new permission stuff

* add store mock

* fix bad merge

* gofmt fix

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* [MM-32343] Environment SubSection (#17054)

* pre-checkout commit

* fix permission for testSiteURL

* pre-merge commit

* increase size of Permissions column in Roles table

* add entry for ENVIRONMENT to testlib/store.go

* use TEXT for Permissions column in Roles table

* use environment subsection permissions for API endpoints

* use subsections permissions for /config/environment

* add suggestions from hahmadia

* update tests to use subsection permissions

* add permissions column back in

* comment out code in upgradeDatabaseToVersion534

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* MM-32351: Add Compliance Subsections (#17023)

* add subsections for compliance sectin

* add to mock functions

* updates for read job

* fixes

* fix test

* update tests

* update tests

* another test fix

* some cleanup

* update mlog

* fix linting

* Fix bad merges

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Hossein <hahmadia@users.noreply.github.com>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>

* MM-32347 Site Subsections (#17095)

* Init

* Added migration key in testlib store

* Fix syntax error

* fix bad merge

* fix lint

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* MM-32350 Integrations (#17097)

* implement server subsections

* fix tests

* update test

* go fmt

Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>

* patch forgotten endpoints

* Adding subsection permissions for Authentication (#17087)

* adding new permissions, migrations to do

* permission migrations and ancilary permissions

* running make app-layers

* fixing tests and lint

* adding permissions to saml

* ldap write permissions

* running make app-layers

* fixing conflict

* making app layers

* clean up and fix tests

* change job type

* fix js error, if site url not returned

Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Update permissions_migrations.go

* gofmt

* upgrade to 535

* gofmt

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Anurag Shivarathri <anurag6713@gmail.com>
Co-authored-by: Ben Cooke <benkcooke@gmail.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
2021-04-06 10:39:48 -04:00
Doug Lauder
02196e04fa MM-27493 Shared channels (MVP) (#17301)
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
2021-04-01 13:44:56 -04:00
Jesús Espino
95b0809850 Extracting html templates into a library (#16946)
* Extracting html templates into a library

* Moving tests to the right place

* Fixing tests

* Addressing PR review comments

* Addressing PR review comments

* Replacing attomic with RWMutex

* Returning errors as channel for Templates watcher

* Address PR review comments

* Other small fixes

* Simplifying NewWithWatcher

* Addressing PR review comments

* Making error handling on rendering templates more robust

* Fixing tests

* Changing how we return errors

* Fixing shadow variables

* Addressing PR review comments

* Logging errors from the outside of sendNotificationEmail

* Fixing lock in shutdown

* Fixing the resource copy for commands tests temporary directories

* Removing unused import

* A couple of tiny fixes
2021-03-12 18:46:43 +01:00
Martin Raymond Kraft
58dce5930e [MM-31094] Replication Lag (#16888)
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.

* MM-31094: Removes mysql-read-replica from default docker services.

* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.

* MM-31094: Updates (store.UsersStore).Get to use context.

* MM-31094: Updates (store.PostStore).Get to use context.

* MM-31094: Removes feature flag and config setting.

* MM-31094: Rolls back some master reads.

* MM-31094: Rolls a non-cache read.

* MM-31094: Removes feature flag from the store.

* MM-31094: Removes unused constant and struct field.

* MM-31094: Removes some old feature flag references.

* MM-31094: Fixes some tests.

* MM-31094: App layers fix.

* MM-31094: Fixes mocks.

* MM-31094: Don't reparse flag.

* MM-31094: No reparse.

* MM-31094: Removed unused FeatureFlags field.

* MM-31094: Removes unnecessary feature flags variable declarations.

* MM-31094: Fixes copy-paste error.

* MM-31094: Fixes logical error.

* MM-30194: Removes test method from store.

* Revert "MM-30194: Removes test method from store."

This reverts commit d5a6e8529b.

* MM-31094: Conforming to make's strange syntax.

* MM-31094: Configures helper for read replica with option.

* MM-31094: Adds some missing ctx's.

* MM-31094: WIP

* MM-31094: Updates test names.

* MM-31094: WIP

* MM-31094: Removes unnecessary master reads.

* MM-31094: ID case changes out of scope.

* MM-31094: Removes unused context.

* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.

* MM-31094: Removes SQLITE db driver ref.

* MM-31094: Layer generate fix.

* MM-31094: Removes unnecessary changes.

* MM-31094: Moves test method.

* MM-31094: Re-add previous fix.

* MM-31094: Removes make command for dev.

* MM-31094: Fix for login.

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2021-03-12 12:37:30 -05:00
Jesús Espino
9cc5089af8 Moving mlog to corelibs (#16915)
* Moving mlog to corelibs

* Regenerating app layers

* Fix golangci-lint problem

* Fixing golangci-lint errors

* Renaming from corelibs to shared

* Renaming from corelibs to shared

* Fixing import

* Fixing merge problems

* Fixing build
2021-03-05 09:18:37 +01:00
Jesús Espino
78355ae2a7 Moving filesstore services into shared folder (#16940)
* Moving filesstore services into shared folder

* Fixing app-layers generation

* Renaming from filesstore to filestore
2021-03-02 14:37:21 +01:00
Jesús Espino
85293fcf41 Add the API for search files (#15605)
* Adding search files api

* Fixing golangci-lint

* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch

* Add content extraction config migration

* Fixing a problem with document extraction

* Unapplying certain changes moved to other PR

* Fixing tests

* Making extract content app migration a private method

* Addressing PR review comments

* Addressing PR review comments

* Adding feature flag

* Removing debug string

* Fixing imports

* Fixing linting errors

* Do not migrate the config if the feature flag is not enabled

* Fix tests

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2021-02-26 07:41:05 +01:00
Jesús Espino
a3de71fba4 Reducing the filestore dependencies from the rest of the source code (#16816)
* Reducing the filestore dependencies from the rest of the source code

* Making more generic config conversion to FileBackendSettings

* Fixing usage of the NewFileBackend function

* Fixing more usages of the NewFileBackend function

* Fix some linter errors

* Fix more linter errors

* Fixing some unit tests

* Fixing linter problem

* Addressing PR review comments

* Simplifing the CopyFile for tests

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2021-02-15 10:09:28 +01:00
Hossein
01f264cd62 [MM-31790] Support Packet Generation BACKEND (#16667)
* init commit

* clean up the code

* make mocks

* fix translations

* mocks and lint fixes

* add tests

* little fixes

* Update i18n/en.json

Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Update i18n/en.json

Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Update i18n/en.json

Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Update i18n/en.json

Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Update i18n/en.json

Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>

* Address Comments

* fix i18n

* update api endpoint

* add enable file and file level for conditional show of banner

* Address Comments

* Make it more clear about returns

* Create zip file utility function

* update en.json

* address comments

* write tests

* check for data in test

* remove warning string

* Correct expected and actual

* set database through environment variables

* reset environment variable at end of test

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
2021-02-01 15:18:52 -05:00
Hossein
4d5ca92a04 [MM-28000] Fix system manager can download export files without right permissions (#16569)
Automatic Merge
2021-01-12 20:45:17 +01:00
Agniva De Sarker
e89b26e8f3 goimports (#16640)
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`

* added goimports lint check to .golangci.yml

* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case

* make app-layers, *-mocks and store-layers for ci check

Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2021-01-07 22:42:43 +05:30
Ibrahim Serdar Acikgoz
8036ba86e7 [MM-31398] upgrade error level to critical in case of panic or os.Exit (#16584)
* upgrade error level to critical in case of panic or os.Exit

* reflect revivew comments

* fix error capitalization
2020-12-23 13:11:35 +03:00
Hossein
d666bb8b3d Add Billing Permissions (#16364)
* add billing permissions

* update store.go

* Update permission.go

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-12-08 12:41:28 -05:00
Agniva De Sarker
d137cf0aa5 MM-31082: Use a base path for test assets (#16478)
* MM-31082: Use a base path for test assets

Tests can be run from other tools using a different base path.
We need to respect the MM_SERVER_PATH variable for that.

https://mattermost.atlassian.net/browse/MM-31082

```release-note
NONE
```

* do not append mattermost-server if env var is set

* fix typo
2020-12-04 20:38:07 +05:30
Jesús Espino
a74fe05695 Removing supplier concept from the sql store (#16355)
* Removing supplier concept from the sql store

* Removing other metions to supplier

* Fixing gofmt

* Fixing gofmt

* Renaming NewSqlStore to New

* Fixing tests

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-12-02 12:06:23 +01:00
Ibrahim Serdar Acikgoz
213345fa2c add manage_remote_clusters permission (#16252) 2020-11-18 18:40:12 +03:00
Agniva De Sarker
c82c37a36e MM-28067: Optimize app server startup in tests (#16263)
* MM-28067: Optimize app server startup in tests

For every test, we would wipe out the database and then start
the server again. This would run through all the migrations
and create new rows in Systems and Roles table every time.
As a result, this was consuming a lot of setup time for every test.

We optimize this by preloading the DB with dummy data in the roles
and systems table so that the server code just skips over those migrations.
This is completely forward compatible and adding new migrations does not
need to generate the sql files again. Only in case of schema changes to the
roles or systems table, this would need to be done.

It is unlikely the Systems table schema will get changed. The Roles table
might change in future but it's a comparatively rare event. Given the reduction
in CI runtime we are seeing, it's a worthy optimization.

We also apply some more optimizations:
- Coalesce multiple UpdateConfig calls into a single one. Each UpdateConfig call
has to do a json marshal which would take precious CPU cycles. It's much more
efficient to do everything in a single call.
- Remove unnecessary debug.FreeOSMemory in reload config. This was an artifact
from old days and is no longer required.

Numbers:
Results show a full **2 minutes** shaved off the test runtime. Earlier, tests
would take around 16 minutes. Now they take 14 minutes.

```release-note
NONE
```

* fix app package
2020-11-16 13:55:32 +01:00
Farhan Munshi
c9a4a475d3 [MM-28717] Refactor applyMultiRoleFilters to use sq builder (#15500)
* Refactor apply multi role filters and add role filters to get all profiles

* Add some tests

* Fix tests

* Fix lint

* Trigger CI

* Rename param to make more sense

* Tie get filtered user stats to usermanagement read users

* Dont filter out other system roles when searching for team members or team admins only filter out system admins

* add new permissions

* add migration

* fix test

* remove system roles as default permissions

* implement changes discussed with dennis

* add read only and fix i18n

* use model consts instead of strings

* turn the permissions into pseudo constants

* Update read only default permissions

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>
2020-11-13 10:57:57 -05:00
Agniva De Sarker
6c71fbaebd MM-29478: Fix race in TestNotifyClusterEvent (#16240)
The messages slice was accessed in a race manner. We protect it
with a mutex

```release-note
NONE
```

https://mattermost.atlassian.net/browse/MM-29478
2020-11-09 19:25:51 +05:30
Agniva De Sarker
72432ab3ff MM-27909: Add manage_shared_channels permission (#15601)
* MM-27909: Add manage_shared_channels permission

We add a new permission to manage shared channels.

It's a channel scoped permission and only the system admin
has that by default.

https://mattermost.atlassian.net/browse/MM-27909

* change to system scoped

* Trigger CI

* Trigger CI

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-10-26 15:54:50 +05:30
Farhan Munshi
2baf95df1e [MM-27274] Add permission to convert public channel to private (#15370)
* Add permission to convert public channel to private

* Rename to PERMISSION_CONVERT_PUBLIC_CHANNEL_TO_PRIVATE

* Fix tests

* Update definitions for roles that have manage_team

* Fix tests

* Add convert private channel to public permission

* Add to channel scoped perms

* Update permission checks on channel privacy endpoint

* Trigger CI
2020-09-03 13:41:07 -04:00
Miguel de la Cruz
49a75059c5 [MM-27919] Allow i18n and resources to be loaded using an environment variable (#15341)
* [MM-27919] Allow i18n and resources to be loaded using an environment variable

* Improve error message

* Modify environment variable name to be less ambiguous

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-08-25 17:29:40 +02:00
Martin Kraft
8354206e5c MM-25543: New Admin Roles (#14960)
* MM-23832: Initial set of changes

* MM-23832: further iteration

* MM-23832: further iteration

* MM-23832: further iteration

* MM-23832: Fixes merge.

* create migration for new Roles

* MM-23832: Renames some roles.

* MM-23832: Adds ability to see logs.

* MM-23832: Removes manage roles from restricted admin.

* MM-23832: Make authentication section read-only for restricted admin.

* MM-23832: Allow restricted admin to purge caches.

* MM-23832: Adds ability to recycle DB connections.

* MM-23832: Adds ability to purge indexes.

* MM-23832: Adds ability to test email and S3 config.

* MM-23832: Adds abilituy to read job status.

* MM-23832: Adds ability to read plugin statuses.

* MM-23832: Renames Restricted Admin to System Manager.

* MM-23832: Adds manage team roles to system_user_manager.

* MM-23832: Updates some permissions.

* MM-23832: Allow get all channels and get moderations.

* MM-23832: Adds some permissions to User Manager.

* MM-23832: Remove write users from user manager.

* MM-23832: Changes permissions for the usermanagement > users sysconsole section.

* MM-23832: Removes read_settings and write_settings permissions. Ensures the usermanagement parent permissions encompass the sub-permissions.

* MM-23832: Updates permissions.

* MM-23832: Changes some permissions checks, adds new permissions to roles.

* MM-23832: Adds ability to update a role.

* MM-23832: Permissions updates.

* MM-23832: Removes write access to plugins for system manager.

* MM-23832: Removes read compliance from new roles.

* MM-23832: Adds mock for new roles creation migration.

* MM-23832: Changes to variadic param.

* MM-23832: Removes some duplication in the permissions model. Renames some permissions constants.

* MM-23832: Updates some migrations.

* MM-23832: Removes some unnecessary constants.

* MM-23832: Changes back to old app method name.

* MM-23832: Fixes incorrect permission check.

* MM-23832: Changes write to read permission check.

* MM-23832: Removes the authentication permission from link/unlink group.

* MM-23832: Enable testing LDAP with read permissions.

* MM-23832: Make testing elasticsearch a read permission.

* MM-23832: Warn metrics are associated to any system console read permissions.

* MM-23832: Updates some permissions checks.

* MM-23832: Removes non-systemconsole permissions from roles.

* MM-23832: Update default permission assignment of sysadmin.

* MM-23832: Fixes incorrect permission check. Removes some unused stuff.

* MM-23832: Update permission to check.

* MM-23832: Switches to struct tags.

* MM-23832: Adds some docs for the permissions tag.

* MM-23832: Removes whitespace.

* MM-23832: Combines system admin restricted access with other acess-control tag.

* MM-23832: Fixes some tests.

* MM-23832: Clarifies docs, does not assume prior permission check in '-' access value case.

* MM-23832: Updates to correct access tag value.

* MM-23832: Adds test of the config settings tag access.

* MM-23832: Undoes whitespace change.

* MM-23832: Removes comment.

* MM-23832: Adds the permissions to the new roles rather than using OR conditions on the permissions checks.

* MM-23832: Removes or condition on permission check.

* MM-23832: Updates mapping.

* MM-23832: Typo fix.

* MM-23832: Adds new 'read_jobs' permission.

* MM-23832: Add read_jobs to all roles with manage_jobs.

* MM-23832: Adds new permission read_other_users_teams.

* MM-23832: Adds read filtering of config.

* MM-23932: Change tag value.

* MM-23832: Fixes some tests. Adds test for read config access tag.

* MM-23832: Adds permissions to list teams.

* MM-23832: Removes the '-' tag value. Adds a new permission read_channel_groups. Updates a permission check.

* MM-23832: Removes unnecessary parent permission for user_management. Fixes permission check change error.

* MM-23832: Removes unused parameter to filter/merge function.

* MM-23832: Renames migration name.

* MM-23832: Fix for godoc.

* MM-23832: Fixes tests.

* MM-23832: Only makes a map once rather than every function call. Doesn't require access tag on config field structs. Reverts one test update and fixes another.

* MM-23832: Removes all of the unnecessary uses of (*App).SessionHasPermissionToAny since removing the user_management parent permission.

* MM-23832: Updates constant type.

* MM-23832: Removes unnecessary comment.

* MM-23832: Renames permissions.

* MM-23832: Fix for permission name changes.

* MM-23832: Adds missing config access tags. Adds some requirec ancillary permissions for write_usermanagement_teams.

* MM-23832: Adds local API endpoint for getting config.

* MM-23832: If tag value is blank or restrict_sys_admin_write then don't do the permission check.

* MM-23832: nil check for strings prior to dereferencing.

* MM-23832: Fix for config display logic.

* MM-23832: Updates godoc.

* MM-23832: Delays the unrestricted check for parity with other permissions checks if the channel id does not exist.

* MM-23832: Removes tautology.

* MM-23832: Re-adds status code check.

* MM-23832: Adds new permission to edit brand image.

* MM-23832: Exports variable for use by mmctl.

* MM-23832: Initialize exported map for use by mmctl.

* MM-23832: Accept deprecated permissions as valid.

* MM-23832: Adds missing permissions to archive a channel.

* MM-23832: Adds missing permissions for managing team.

* MM-23832: Properly filters config values in patch and update API responses.

* MM-23832: Fixes license viewing and writing permissions.

* MM-23832: Require license to assign 'new system roles'.

* MM-23832: Adds translation keys.

* MM-23832: Updates translation order.

* MM-27529: Splits read_channel_groups into read_public_channel_groups and read_private_channel_groups.

* MM-23832: Prevent read-only permissions from editing site url test parameter.

* MM-23832: Prevent read permissions from sniffing ports and elastic password.

* MM-23832: Adds missing permission required for write user management channels.

* MM-23832: Allows new roles to search for channels.

* MM-23832: Adds ability for system_manager to manage jobs.

* MM-23832: Cluster status access by sysconsole permission, not manage_system.

* MM-23832: Adds 'add_user_to_team' permission to sysconsole write usermanagement teams.

* MM-23832: Fixes lint.

* MM-23832: Test fix.

* MM-23832: Test fix.

Co-authored-by: Catalin Tomai <catalin.tomai@mattermost.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-08-21 16:49:31 -04:00
Jesús Espino
7fe6c94eda Adding Upgrade to Enterprise version feature (#14539)
* Adding Upgrade to Enterprise version feature

* Addressing PR review comments, and adding some minor improvements

* Add tests file

* Addressing PR comments

* fix linter checks

* Storing and exposing the upgraded from TE info

* Fix showing errors on mac

* A more appropiate status code for not-supported upgrade

* Fixing tests

* Handling permissions errors

* More server logging around upgrade failures

* Apply text changes suggested from code review

Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>

* Address PR review comments

* Only allow to restart the system after an upgrade

* Verify file signature before upgrade

* Adding limit to the downloaded file

* Simplifying the upgrade binary process with backup in memory

* Fixing backup/restore mechanism for the binary file

* Improve file permissions handling

* Askin the permissions for the right place (the parent directory)

* Fixing tests

* Addressing PR review comments

* Fix license headers

* Fixing retry layer

* Making it work on windows builds

* Adding license header

* Fixing 2 tests

* Fixing tests that need UpgradeFromTE System key mock

* Extracting i18n translation

* Apply suggestions from code review

Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>

* Improving how the errors are written

* Fixing another error text

* Removing unneeded translation

* Fixing upgrade status strings

* Update i18n/en.json

Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>

* Fixing tests

Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-08-21 20:23:04 +02:00
Rodrigo Villablanca
20e44399c7 SystemStore migration to return plain errors (#14835)
* SystemStore migration to return plain errors

* Fix nilness

* Fix translations

* Fix merge

* Fix layers

* Fix merge errors

* Lint: remove unnecessary use of sprint

* Fix merge errors

* Fix i18n

Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
2020-08-13 20:32:57 +05:30
Agniva De Sarker
3cf05efdd9 MM-25404: Add GetHealthScore metric for the cluster (#14767)
* MM-25404: Add GetHealthScore metric for the cluster

This PR adds the necessary function to the cluster interface
so that they can be called from App code.

* Add mocks

* Remove fakeapp

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-06-24 18:46:33 +05:30
Miguel de la Cruz
4fde004a5b [MM-23535] Add disable database search flag and return empty posts results if set (#14245)
* [MM-23535] Add disable database search flag and return empty posts results if set

* Add UpdateConfig function for the SearchStore and hook it into the app lifecycle

* Add the config listener in the server instance instead of using FakeApp

* Instantiate searchlayer as a pointer to avoid passing around copies of it
2020-05-13 14:00:57 +02:00
Eli Yukelzon
7800116429 MM-23093 Implement Server Setup telemetry - server configuration (#14374)
* added advanced first day diagnostics reporting

* typo

* config corrected

* defaults

* moved from config to system db table

* missing file

* added error handling

* tests

* typos

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-04-30 18:18:12 +03:00
Jesse Hallam
636d168b84 MM-23926: avoid setting last viewed for bots (#14253)
The current code path for `CreatePostAsUser` tries to update the `LastViewedAt` for bots, which logs a warning message if the bot isn't actually in the channel.

This pull request changes the semantics of bot posting to not update the bot's `LastViewedAt` timestamp, avoiding this log altogether. It matches the semantics of `from_webhook`, but notably makes bots slightly less like users in that they no longer "read" channels when they post. This seems reasonable, but I'm both looking for validation of this semantic change in addition to the code review.

Fixes: https://mattermost.atlassian.net/browse/MM-23926

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-04-22 12:28:31 -03:00
Farhan Munshi
ab338e8417 [MM-23019] Create use_group_mentions permission and migrate existing roles (#13987)
* MM-22962 Create use_group_mentions permission and give to all non guest roles that can create post

* Add use_group_mentions to team admin role for test

* Trigger CI

* MM-22962 Remove old migration keys
2020-03-30 23:08:48 -04:00
Jesse Hallam
f149ada16a MM-23261 plugin stderr debug logs (#14166)
* explicitly assert panic as error log

* Revert "[MM-18150] plugin panic trace should not be lost (#13559)"

This reverts commit 5d928b4f94, while leaving the unit tests intact
and now asserting debug logs instead.

* missing license header
2020-03-30 15:00:45 -03:00
Miguel de la Cruz
4fe25b1cdd [MM-21551] Add search tests structure to test the search engines (#14031)
* WIP

* Adding bleve to go modules

* WIP

* Adding missing files from searchengine implementation

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* User and channel indexing and searches implemented

* Make bleve tests run with in-memory indexes

* Implement post index and deletion tests

* Initial commits for the search layer

* Removing unnecesary indexing

* WIP

* WIP

* More fixes for tests

* Adding the search layer

* Finishing the migration of searchers to the layer

* Removing unnecesary code

* Allowing multiple engines active at the same time

* WIP

* Add simple post search

* Print information when using bleve

* Adding some debugging to understand better how the searches are working

* Making more dynamic config of search engines

* Add post search basics

* Adding the Purge API endpoint

* Fixing bleve config updates

* Adding missed file

* Regenerating search engine mocks

* Adding missed v5 to modules imports

* fixing i18n

* Fixing some test around search engine

* Removing all bleve traces

* Cleaning up the vendors directory and go.mod/go.sum files

* Regenerating timer layer

* Adding properly the license

* Fixing govet shadow error

* Fixing some tests

* Fixing TestSearchPostsFromUser

* Fixing another test

* Fixing more tests

* Fixing more tests

* Removing SearchEngine redundant text from searchengine module code

* Fixing some reindexing problems in members updates

* Fixing tests

* Addressing PR comments

* Reverting go.mod and go.sum

* Addressing PR comments

* Fixing tests compilation

* Fixing govet

* Adding search engine stop method

* Being more explicit on where we use includeDeleted

* Adding GetSqlSupplier test helper method

* Mocking elasticsearch start function

* Fixing tests

* Search tests

* Fix tests

* Fix mod

* Fixing searchEngine for test helpers with store mocks

* Remove loglines

* Fix i18n strings

* Migrate search posts tests

* Fix linter

* Do not run search tests if -short flag is enabled

* Migrate back store tests that didn't belong to the searchlayer

* Fix scopelint issues

Co-authored-by: Jesús Espino <jespinog@gmail.com>
2020-03-30 19:17:40 +02:00
Martin Kraft
4d99aa22ba MM-22212 & MM-22208: Read from the higher-scoped scheme if the permission is non-moderated. (#13813)
* MM-22212: Read non-moderated permissions from higher-scoped scheme.

* MM-2212: Corrects test count in comment.

* MM-22212: Adds godoc comment.

* MM-2212: Switches to the channel roles check in a few more places.

* MM-22212: Refactors and fixes.

* MM-22212: Reverts change, no longer required.

* MM-22212: Removes translation.

* MM-22212: Un-comments merged new permission.

* MM-22212: Un-comments merged new permission.

* MM-22212: Performance tweak.

* MM-22212: Fixes some fmting.

* MM-22212: Add unit test for newly-added store methods.

* MM-22212: Renames app method.

* MM-22212: Re-uses existing function to find string in slice.

* MM-22212: Keeps 'higher-scoped' terminology for consistency.

* MM-22212: Refactors based on PR feedback.

* MM-22212: Fix for some bad merging.

* MM-22212: Renamed some things.

* MM-22212: Use an 'else' instead of a 'continue' for readability.

* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.

* MM-22212: Adds mock to new cache store.

* MM-22212: Adds missing open tracing app layer methods.

* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.

* MM-22212: Migrates team schemes. Removes unused AppError.

* MM-22212: Fix for for if.

* MM-22212: Fixes iterator.

* MM-22212: Updates open tracing generated methods.

* MM-22212: Fix mocks.

* MM-22212: Change migration key name.

* MM-22212: Switched to data structure from other branch.

* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.

* MM-22212: Adds tracking of channel moderation.

* Revert "MM-22212: Adds tracking of channel moderation."

This reverts commit 23689efa22.

* MM-22212: Switch some functions to methods and vice versa.

* MM-22212: Fix for refactor bug not notifiying websocket about changed role.

* MM-22212: Adds test for public/private 'manage_members' handling.

* MM-22122 Fix manage channel members edge case for public and private channels (#14049)

* MM-22212: Adds moderated permission to team_admin.

* MM-22212: Updates migration.

* MM-22212: Revert unnecessary update to default roles.

* Add channel scheme updated event when channel scheme is deleted or created (#14057)

* MM-22212: Adds newline.

* MM-22212: Migration fix.

* MM-22212: Fix for migration.

* MM-22212: Test fix.

Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
2020-03-23 13:44:20 -04:00
Jesús Espino
c66e182b08 Adding the new search engine abstraction (#13304)
* WIP

* Adding bleve to go modules

* WIP

* Adding missing files from searchengine implementation

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* User and channel indexing and searches implemented

* Make bleve tests run with in-memory indexes

* Implement post index and deletion tests

* Initial commits for the search layer

* Removing unnecesary indexing

* WIP

* WIP

* More fixes for tests

* Adding the search layer

* Finishing the migration of searchers to the layer

* Removing unnecesary code

* Allowing multiple engines active at the same time

* WIP

* Add simple post search

* Print information when using bleve

* Adding some debugging to understand better how the searches are working

* Making more dynamic config of search engines

* Add post search basics

* Adding the Purge API endpoint

* Fixing bleve config updates

* Adding missed file

* Regenerating search engine mocks

* Adding missed v5 to modules imports

* fixing i18n

* Fixing some test around search engine

* Removing all bleve traces

* Cleaning up the vendors directory and go.mod/go.sum files

* Regenerating timer layer

* Adding properly the license

* Fixing govet shadow error

* Fixing some tests

* Fixing TestSearchPostsFromUser

* Fixing another test

* Fixing more tests

* Fixing more tests

* Removing SearchEngine redundant text from searchengine module code

* Fixing some reindexing problems in members updates

* Fixing tests

* Addressing PR comments

* Reverting go.mod and go.sum

* Addressing PR comments

* Fixing tests compilation

* Fixing govet

* Adding search engine stop method

* Being more explicit on where we use includeDeleted

* Adding GetSqlSupplier test helper method

* Mocking elasticsearch start function

* Fixing tests

Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-03-13 15:33:18 +01:00
Jesús Espino
7035e09fe9 Adding changes to separate unit tests and integration tests (#13670)
* Introducing unit (not integration) tests for the app layer

* Initial support for unit tests at the API

* Adding unit tests support to the store layer

* Add unit tests support in commands

* Adding last tests needed for run unit tests properly

* Fixing govet

* Removing some duplication

* Fixing tests

* Fixing tests

* Not compiling test helpers with the main module for api

* Revert "Not compiling test helpers with the main module for api"

This reverts commit 36a199bbe0.

* Fixing tests

* Fixing unit tests

* More consistency between api4/apiteslib.go and app/helper_test.go

* Renaming things to make more obvious the new Setup functions purpose

* Reverting change in go.sum

* Start with empty mock for app layer

* Start with empty mock for api layer

* Start with empty mock for web layer

* Renaming SetupWithStoreMockConfig to SetupConfigWithStoreMock

* Fixing tests on web package

* Removing unnecesary function
2020-03-02 08:13:39 -08:00
JtheBAB
cd36c9f041 MM-22355 - Fix typos detected by goreportcad.com (#13842)
* Update webhook_test.go

* Update user_test.go

* Update filesstore_test.go

* Update plugin_requests.go

* Update syncables.go

* Update helper.go

* Update html_entities.go

* Update user.go

* Update team.go

* Update notification.go

* Update notification_test.go

* Update plugin_api_test.go

* Update post_metadata.go

* Update channel_test.go

* Update database.go

* Update channel.go

* Update user_store.go

* Update team_test.go

* revert andd

* Revert back to infintie

Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-03-02 17:10:40 +01:00
Jesús Espino
596f986755 Switch to Postgres when running tests and turn off the fsync setting (#13678)
* Run all tests fasts in postgres

* Moving postgres config to a file

* Addressing PR review comments

* Testing against mysql in the CI to keep things less changed

* Revert "Testing against mysql in the CI to keep things less changed"

This reverts commit fc940c111d.

* Fixing a test broken in postgres but working in mysql

* Fixing some tests

* Fixing some config tests with postgres

* Fixing the rest of config tests

* Fixing govet error

* Fixing search tests for postgres
2020-02-06 15:15:18 +01:00
Andy Librian
3865bc501e golint fix (#13613) 2020-01-20 12:32:20 +01:00
Jesús Espino
a63684fcb5 Consistent license message for all the go files (#13235)
* Consistent license message for all the go files

* Fixing the last set of unconsistencies with the license headers

* Addressing PR review comments

* Fixing busy.go and busy_test.go license header
2019-11-29 12:59:40 +01:00
Miguel de la Cruz
2259b7f2a8 [MM-19948] Set version on module file and internal paths (#13186)
* [MM-19948] Set version on module file and internal paths

* Fixes after merge

* Fix i18n checker error
2019-11-28 14:39:38 +01:00
Jesse Hallam
6a906e91ad enable non-GOPATH testing with different mattermost-server folder name (#12930)
* enable non-GOPATH testing

Tweak the testlib package to support finding the root without assuming the folder name of the repository.

Fixes: MM-19729

* fix web tests
2019-11-06 14:46:51 -05:00
Jesús Espino
8febdcf2b8 Removing unnecesary unnused old layered store (#12999) 2019-11-06 12:43:41 +01:00
Ben Schumacher
7a665aacdd Run gosimple against codebase (#12928) 2019-10-29 07:45:09 +01:00
Carlos Tadeu Panato Junior
c7b583ccdd use mkdirall instead of mkdir (#12032)
* use mkdirall instead of mkdir

* revert

* create the folder here
2019-09-18 18:39:01 +02:00
Jesús Espino
28cc7e7e36 Migrating roles and schemes to new Cache Layer (#11936)
* Migrating roles and schemes to new Cache Layer

* Adding missed license headers

* Updating cache tests

* Adding the cache layer to the testlib helper

* Fixing cyclic dependency

* fixing a bit of not-idiomatic error handling

* Another small fix arrount idiomatic error handling
2019-09-12 18:52:45 +02:00
Ali Farooq
e2ef5d7149 Log panic when exiting (if any) (#11726) 2019-07-30 09:37:07 +02:00
Jesse Hallam
98ff5fab32 MM-16261: Synchronize plugins in HA (#11657)
* MM-16272 - Synchronize plugins across cluster (#11611)

* MM-16272 - Synchronize plugins across cluster

* Adding a test

* MM-16272 - Fixed tests

* MM-16272 - PR feedback

* MM-16270 - Plugin Sync (#11615)

* Initial implementation for plugin synch with file store. WIP

* Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory

* Refactored plugin filestore operations out of main install/remove plugin

* Fixing error handling details

* Changes to use structured logging

* More logging fixes

* Wording and comments improvements

* Error handling and control flow improvements

* Changed managed flag check to use os.stat

* Added file store plugin dir and filename consts

* Replaced FileRead to use a the FileReader in PluginSync

* Minor styling and PR feedback changes

* Minor error handling improvements

* Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins

* PR Feedback improvements

* Minor err handling fix

* Removing FileStorePath from PluginEventData (#11644)

* Fix plugin path (#11654)

* tweak path, logging

Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity.

* Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist

* PR Feedback

* Install prepackaged plugins locally only (#11656)

* s/uninstall/remove

* Updated ClusterMessage comment

* Updated PluginSync to test against s3 + local storage
2019-07-18 15:05:53 -03:00
Jesse Hallam
25f3bf4a2b MM-16506: conditionally build go.mod for plugins (#11430)
* MM-16506: conditionally build go.mod for plugins

Unless `GO111MODULE=off`, generate a `go.mod` that points at the local copy of `mattermost-server` to ensure plugin tests that compile source code on demand always test with the local copy of mattermost-server.  This also fixes an issue with early adopters of `GO111MODULE=on` on the server failing to find the right version of go-i18n dependencies.

* plugin: enable testlib resource management

* customize fileutils.Find* for testlib
2019-07-05 12:34:49 -03:00
Eli Yukelzon
6a42ad2af5 [MM-14400] Programmatically generate default.json (#10551)
* create/update config.json using go generate

* added default config generator
added config-reset to Jenkins and make package, updated defaults to consider 'isNew' flag

* corrections after code review

* fixed Config.isValid to handle empty encryption keys

* fixed Config.isValid to handle empty encryption keys

* fixed Config.isValid to handle empty encryption keys

* isUpdate now only checks for nil

* Addressed review comments, added unit testing for default config generator

* err shadowing

* license

* provide output file for config generator via ENV variable, since go generate doesn't support arguments and we need two output paths (config-reset and package)

* cleanup

* proper defaults for PushNotificationServer and SendPushNotifications

* corrected generating defaults for TrustedProxyIPHeader to be consistent with default.json in master

* Check for empty SiteURL as well as nil

* corrected SiteURL settings and checking

* crazy typos fixed

* corrected tests to newly expected values

* relaxed the checks

* fixed formatting
2019-06-06 13:57:01 -04:00
Jesse Hallam
72448d12a9 deprecate timezones.json (#10311) 2019-02-25 12:07:45 -04:00