* Adding search files api
* Fixing golangci-lint
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Add content extraction config migration
* Fixing a problem with document extraction
* Unapplying certain changes moved to other PR
* Fixing tests
* Making extract content app migration a private method
* Addressing PR review comments
* Addressing PR review comments
* Adding feature flag
* Removing debug string
* Fixing imports
* Fixing linting errors
* Do not migrate the config if the feature flag is not enabled
* Fix tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-28067: Optimize app server startup in tests
For every test, we would wipe out the database and then start
the server again. This would run through all the migrations
and create new rows in Systems and Roles table every time.
As a result, this was consuming a lot of setup time for every test.
We optimize this by preloading the DB with dummy data in the roles
and systems table so that the server code just skips over those migrations.
This is completely forward compatible and adding new migrations does not
need to generate the sql files again. Only in case of schema changes to the
roles or systems table, this would need to be done.
It is unlikely the Systems table schema will get changed. The Roles table
might change in future but it's a comparatively rare event. Given the reduction
in CI runtime we are seeing, it's a worthy optimization.
We also apply some more optimizations:
- Coalesce multiple UpdateConfig calls into a single one. Each UpdateConfig call
has to do a json marshal which would take precious CPU cycles. It's much more
efficient to do everything in a single call.
- Remove unnecessary debug.FreeOSMemory in reload config. This was an artifact
from old days and is no longer required.
Numbers:
Results show a full **2 minutes** shaved off the test runtime. Earlier, tests
would take around 16 minutes. Now they take 14 minutes.
```release-note
NONE
```
* fix app package
* Refactor apply multi role filters and add role filters to get all profiles
* Add some tests
* Fix tests
* Fix lint
* Trigger CI
* Rename param to make more sense
* Tie get filtered user stats to usermanagement read users
* Dont filter out other system roles when searching for team members or team admins only filter out system admins
* add new permissions
* add migration
* fix test
* remove system roles as default permissions
* implement changes discussed with dennis
* add read only and fix i18n
* use model consts instead of strings
* turn the permissions into pseudo constants
* Update read only default permissions
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>
* MM-27909: Add manage_shared_channels permission
We add a new permission to manage shared channels.
It's a channel scoped permission and only the system admin
has that by default.
https://mattermost.atlassian.net/browse/MM-27909
* change to system scoped
* Trigger CI
* Trigger CI
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add permission to convert public channel to private
* Rename to PERMISSION_CONVERT_PUBLIC_CHANNEL_TO_PRIVATE
* Fix tests
* Update definitions for roles that have manage_team
* Fix tests
* Add convert private channel to public permission
* Add to channel scoped perms
* Update permission checks on channel privacy endpoint
* Trigger CI
* MM-23832: Initial set of changes
* MM-23832: further iteration
* MM-23832: further iteration
* MM-23832: further iteration
* MM-23832: Fixes merge.
* create migration for new Roles
* MM-23832: Renames some roles.
* MM-23832: Adds ability to see logs.
* MM-23832: Removes manage roles from restricted admin.
* MM-23832: Make authentication section read-only for restricted admin.
* MM-23832: Allow restricted admin to purge caches.
* MM-23832: Adds ability to recycle DB connections.
* MM-23832: Adds ability to purge indexes.
* MM-23832: Adds ability to test email and S3 config.
* MM-23832: Adds abilituy to read job status.
* MM-23832: Adds ability to read plugin statuses.
* MM-23832: Renames Restricted Admin to System Manager.
* MM-23832: Adds manage team roles to system_user_manager.
* MM-23832: Updates some permissions.
* MM-23832: Allow get all channels and get moderations.
* MM-23832: Adds some permissions to User Manager.
* MM-23832: Remove write users from user manager.
* MM-23832: Changes permissions for the usermanagement > users sysconsole section.
* MM-23832: Removes read_settings and write_settings permissions. Ensures the usermanagement parent permissions encompass the sub-permissions.
* MM-23832: Updates permissions.
* MM-23832: Changes some permissions checks, adds new permissions to roles.
* MM-23832: Adds ability to update a role.
* MM-23832: Permissions updates.
* MM-23832: Removes write access to plugins for system manager.
* MM-23832: Removes read compliance from new roles.
* MM-23832: Adds mock for new roles creation migration.
* MM-23832: Changes to variadic param.
* MM-23832: Removes some duplication in the permissions model. Renames some permissions constants.
* MM-23832: Updates some migrations.
* MM-23832: Removes some unnecessary constants.
* MM-23832: Changes back to old app method name.
* MM-23832: Fixes incorrect permission check.
* MM-23832: Changes write to read permission check.
* MM-23832: Removes the authentication permission from link/unlink group.
* MM-23832: Enable testing LDAP with read permissions.
* MM-23832: Make testing elasticsearch a read permission.
* MM-23832: Warn metrics are associated to any system console read permissions.
* MM-23832: Updates some permissions checks.
* MM-23832: Removes non-systemconsole permissions from roles.
* MM-23832: Update default permission assignment of sysadmin.
* MM-23832: Fixes incorrect permission check. Removes some unused stuff.
* MM-23832: Update permission to check.
* MM-23832: Switches to struct tags.
* MM-23832: Adds some docs for the permissions tag.
* MM-23832: Removes whitespace.
* MM-23832: Combines system admin restricted access with other acess-control tag.
* MM-23832: Fixes some tests.
* MM-23832: Clarifies docs, does not assume prior permission check in '-' access value case.
* MM-23832: Updates to correct access tag value.
* MM-23832: Adds test of the config settings tag access.
* MM-23832: Undoes whitespace change.
* MM-23832: Removes comment.
* MM-23832: Adds the permissions to the new roles rather than using OR conditions on the permissions checks.
* MM-23832: Removes or condition on permission check.
* MM-23832: Updates mapping.
* MM-23832: Typo fix.
* MM-23832: Adds new 'read_jobs' permission.
* MM-23832: Add read_jobs to all roles with manage_jobs.
* MM-23832: Adds new permission read_other_users_teams.
* MM-23832: Adds read filtering of config.
* MM-23932: Change tag value.
* MM-23832: Fixes some tests. Adds test for read config access tag.
* MM-23832: Adds permissions to list teams.
* MM-23832: Removes the '-' tag value. Adds a new permission read_channel_groups. Updates a permission check.
* MM-23832: Removes unnecessary parent permission for user_management. Fixes permission check change error.
* MM-23832: Removes unused parameter to filter/merge function.
* MM-23832: Renames migration name.
* MM-23832: Fix for godoc.
* MM-23832: Fixes tests.
* MM-23832: Only makes a map once rather than every function call. Doesn't require access tag on config field structs. Reverts one test update and fixes another.
* MM-23832: Removes all of the unnecessary uses of (*App).SessionHasPermissionToAny since removing the user_management parent permission.
* MM-23832: Updates constant type.
* MM-23832: Removes unnecessary comment.
* MM-23832: Renames permissions.
* MM-23832: Fix for permission name changes.
* MM-23832: Adds missing config access tags. Adds some requirec ancillary permissions for write_usermanagement_teams.
* MM-23832: Adds local API endpoint for getting config.
* MM-23832: If tag value is blank or restrict_sys_admin_write then don't do the permission check.
* MM-23832: nil check for strings prior to dereferencing.
* MM-23832: Fix for config display logic.
* MM-23832: Updates godoc.
* MM-23832: Delays the unrestricted check for parity with other permissions checks if the channel id does not exist.
* MM-23832: Removes tautology.
* MM-23832: Re-adds status code check.
* MM-23832: Adds new permission to edit brand image.
* MM-23832: Exports variable for use by mmctl.
* MM-23832: Initialize exported map for use by mmctl.
* MM-23832: Accept deprecated permissions as valid.
* MM-23832: Adds missing permissions to archive a channel.
* MM-23832: Adds missing permissions for managing team.
* MM-23832: Properly filters config values in patch and update API responses.
* MM-23832: Fixes license viewing and writing permissions.
* MM-23832: Require license to assign 'new system roles'.
* MM-23832: Adds translation keys.
* MM-23832: Updates translation order.
* MM-27529: Splits read_channel_groups into read_public_channel_groups and read_private_channel_groups.
* MM-23832: Prevent read-only permissions from editing site url test parameter.
* MM-23832: Prevent read permissions from sniffing ports and elastic password.
* MM-23832: Adds missing permission required for write user management channels.
* MM-23832: Allows new roles to search for channels.
* MM-23832: Adds ability for system_manager to manage jobs.
* MM-23832: Cluster status access by sysconsole permission, not manage_system.
* MM-23832: Adds 'add_user_to_team' permission to sysconsole write usermanagement teams.
* MM-23832: Fixes lint.
* MM-23832: Test fix.
* MM-23832: Test fix.
Co-authored-by: Catalin Tomai <catalin.tomai@mattermost.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding Upgrade to Enterprise version feature
* Addressing PR review comments, and adding some minor improvements
* Add tests file
* Addressing PR comments
* fix linter checks
* Storing and exposing the upgraded from TE info
* Fix showing errors on mac
* A more appropiate status code for not-supported upgrade
* Fixing tests
* Handling permissions errors
* More server logging around upgrade failures
* Apply text changes suggested from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Address PR review comments
* Only allow to restart the system after an upgrade
* Verify file signature before upgrade
* Adding limit to the downloaded file
* Simplifying the upgrade binary process with backup in memory
* Fixing backup/restore mechanism for the binary file
* Improve file permissions handling
* Askin the permissions for the right place (the parent directory)
* Fixing tests
* Addressing PR review comments
* Fix license headers
* Fixing retry layer
* Making it work on windows builds
* Adding license header
* Fixing 2 tests
* Fixing tests that need UpgradeFromTE System key mock
* Extracting i18n translation
* Apply suggestions from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Improving how the errors are written
* Fixing another error text
* Removing unneeded translation
* Fixing upgrade status strings
* Update i18n/en.json
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Fixing tests
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-22962 Create use_group_mentions permission and give to all non guest roles that can create post
* Add use_group_mentions to team admin role for test
* Trigger CI
* MM-22962 Remove old migration keys
* MM-22212: Read non-moderated permissions from higher-scoped scheme.
* MM-2212: Corrects test count in comment.
* MM-22212: Adds godoc comment.
* MM-2212: Switches to the channel roles check in a few more places.
* MM-22212: Refactors and fixes.
* MM-22212: Reverts change, no longer required.
* MM-22212: Removes translation.
* MM-22212: Un-comments merged new permission.
* MM-22212: Un-comments merged new permission.
* MM-22212: Performance tweak.
* MM-22212: Fixes some fmting.
* MM-22212: Add unit test for newly-added store methods.
* MM-22212: Renames app method.
* MM-22212: Re-uses existing function to find string in slice.
* MM-22212: Keeps 'higher-scoped' terminology for consistency.
* MM-22212: Refactors based on PR feedback.
* MM-22212: Fix for some bad merging.
* MM-22212: Renamed some things.
* MM-22212: Use an 'else' instead of a 'continue' for readability.
* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.
* MM-22212: Adds mock to new cache store.
* MM-22212: Adds missing open tracing app layer methods.
* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.
* MM-22212: Migrates team schemes. Removes unused AppError.
* MM-22212: Fix for for if.
* MM-22212: Fixes iterator.
* MM-22212: Updates open tracing generated methods.
* MM-22212: Fix mocks.
* MM-22212: Change migration key name.
* MM-22212: Switched to data structure from other branch.
* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.
* MM-22212: Adds tracking of channel moderation.
* Revert "MM-22212: Adds tracking of channel moderation."
This reverts commit 23689efa22.
* MM-22212: Switch some functions to methods and vice versa.
* MM-22212: Fix for refactor bug not notifiying websocket about changed role.
* MM-22212: Adds test for public/private 'manage_members' handling.
* MM-22122 Fix manage channel members edge case for public and private channels (#14049)
* MM-22212: Adds moderated permission to team_admin.
* MM-22212: Updates migration.
* MM-22212: Revert unnecessary update to default roles.
* Add channel scheme updated event when channel scheme is deleted or created (#14057)
* MM-22212: Adds newline.
* MM-22212: Migration fix.
* MM-22212: Fix for migration.
* MM-22212: Test fix.
Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
* Introducing unit (not integration) tests for the app layer
* Initial support for unit tests at the API
* Adding unit tests support to the store layer
* Add unit tests support in commands
* Adding last tests needed for run unit tests properly
* Fixing govet
* Removing some duplication
* Fixing tests
* Fixing tests
* Not compiling test helpers with the main module for api
* Revert "Not compiling test helpers with the main module for api"
This reverts commit 36a199bbe0.
* Fixing tests
* Fixing unit tests
* More consistency between api4/apiteslib.go and app/helper_test.go
* Renaming things to make more obvious the new Setup functions purpose
* Reverting change in go.sum
* Start with empty mock for app layer
* Start with empty mock for api layer
* Start with empty mock for web layer
* Renaming SetupWithStoreMockConfig to SetupConfigWithStoreMock
* Fixing tests on web package
* Removing unnecesary function
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-12083: unittest using externally managed database
* cherry-pick Makefile changes from @cpanato
* Jenkins changes (#9915)
* add docker compose
* udpate
* when using minio dont need to set the region
* update
* add wait for it script
* using old minio
* add new jenkins file
* update makefile
* add dockerfile
* rename the docker-compose proj to avoid colision (#9917)
* rename the docker-compose proj to avoid colision
* enable debug
* enable debug to double checkt the branchs and fix docker-compose name (#9919)
* add ee hash to check (#9920)
* fix name (#9921)
* update jenkins file to push from branch and prs
* if a new push comes in stop the running build
* split mysql and postgres variables
* add script to run jenkins-like env in local dev env
* update docker-compose project name to use uuid to make it more randon
* fix DCNAME definition
* update elasticsearch docker image
* revert test
* tidy up stages, and wait for mysql differently
* update docker image and add check for postgres
* checking if is ready
* update docker compose to have a wait for deps
* add readme and rename dockerfile
* fix -unittest setup
* using mm docker image
* restore parallel unit tests at the package level
Spin up a dedicated database for each package under test to avoid races in accessing the same tables.
Simplify the interface for configuring the test database to just a DSN instead of multiple exports for each field.
* try to work around root mysql access in CI
* update local-test-env.sh too
* MYSQL_ROOT_HOST: %
* fix missing quotes
* setting some memory limits for mysql
* revert memory docker compose does not support
* fix env name for postgres
* expose errors in app/export_test.go
* fix test label, better error checking on teardown
* increase query timeout for tests
* fix export_test
* update local dev script
* add configurable mysql root passwd
* testlib: introduce and leverage
This doesn't yet factor out the individual test helpers: many packages
still rely on `api4` directly to do this, but now wire up the test store
setup through this package. `app` and `store`, in particular, don't use
`testlib` because of circular dependencies at the moment.
* cmd: command_test.go: use api4 testlib
* cmd: plugin_test.go: remove dependence on test-config.json
* cmd: config_test.go use configured database settings
* ensure test-(te|ee) exit with status code
* test-server: run all tests, deprecating test-te/test-ee
* cmd/mattermost/commands: fix unit tests
Instead of relying on (and modifying) a config.json found in the current path, explicitly create a temporary one from defaults for each test. This was likely the source of various bugs over time, but specifically allows us to override the SqlSettings to point at the configured test database for all tests simultaneously.
* wrap run/check into a test helper
It was insufficient to set a config for each invocation of CheckCommand or RunCommand: some tests relied on the config having changed in a subsequent assertion. Instead, create a new test helper embedding api4.TestHelper. This has the nice advantage of cleaning up all the teardown.
* additional TestConfigGet granularity
* customized config path to avoid default location
* be explicit if the storetest initialization fails
* generate safe coverprofile names in the presence of subtests
* additional TestConfigShow granularity
* fix permission_test.go typo
* fix webhook tests
* actually flag.Parse() to skip database setup on os.Execed tests
* fix recent regression in #9962, not caught by unit tests
