Commit Graph

57 Commits

Author SHA1 Message Date
Hector
62b57143c8 [MM 19840] Create a Cache Interface abstraction to support multiple cache backends (#13384)
* Refactor to use structured logging

* Properly formatted with gofmt

* created interface Cache, but construction of cache is still coupled.

* Implementing cache factories to build caches

* Simple redis implementation without error handling. Keys and values by default are string

* refactor NewLocalCacheLayer to inject cache factory

* Removed redis impl to focus on cache abstraction

* CacheFactory injected on sqlsupplier and saved in Store struct

* remove useless private method

* replace concrete declaration of lru cache to cache abstraction

* discard spaces

* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)

* refactor to include all caches in the same package cache and subpackages

* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)

* closing cacheprovider and releasing resources while closing sql store

* fixed merge conflict fail

* gofmt files

* remove unused property from post_store

* naming refactor to avoid stutter. Added godocs on interface

* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer

* fixed broken test

* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer

* move statusCache to server to initialize it with the new cache provider

* update terms_service and channel_layer to have new cacheProvider

* gofmt

* Add Connect method to the cache provider

* mock cacheprovider in user_layer_test

Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-01-09 09:57:28 +01:00
Doug Lauder
420c411595 MM-8607: Add ability to turn off non-critical services when under load; cluster support (#13267)
* MM-8607: Add cluster support for server busy status

MM-8607: Busy.Cluster should be non-exported

MM-8607: rename logging field seconds -> expires_sec

MM-8607: each node clears its own busy flag

MM-8607: ensure unit tests are not sensitive to test machine speed

* MM-8607: chg comment to force CI rebuild
2019-12-03 18:56:39 +01:00
Jesús Espino
a63684fcb5 Consistent license message for all the go files (#13235)
* Consistent license message for all the go files

* Fixing the last set of unconsistencies with the license headers

* Addressing PR review comments

* Fixing busy.go and busy_test.go license header
2019-11-29 12:59:40 +01:00
Miguel de la Cruz
2259b7f2a8 [MM-19948] Set version on module file and internal paths (#13186)
* [MM-19948] Set version on module file and internal paths

* Fixes after merge

* Fix i18n checker error
2019-11-28 14:39:38 +01:00
threepwood-mm
f9d9dd8b60 [MM-14521] New endpoint to validate site urls (#11944) 2019-09-06 12:52:14 +02:00
Miguel de la Cruz
8a13c9d1d5 [MM-15661] Create SendNotificationEmail and move config option check to it (#11478) 2019-07-04 14:16:41 +01:00
Jesús Espino
c8920588a0 Adding new "VIEW_MEMBERS" permissions restrict the scope of users visibility (#10487)
* MM-14138: Adding new "VIEW_MEMBERS" permissions restrict the scope of users visibility

* Fixing gofmt

* Fixing broken tests

* Addressing PR review comments from Miguel de la Cruz

* Removed hack

* A bit nicer and cleaner code in the UserBelongsToChannels function

* Adding cluster cache invalidation for user team ids

* Checking in the correct order permissions to not leek existency information

* Adding restrictions to TeamMembers and User status requests

* Fixing tests

* Fixing status endpoint permissions checks

* Adding more tests

* Fixing tests

* More tests and making the restrictions query based only on joins

* Adding more tests

* Adding more tests

* fixing merge problems

* Reverting status changes to avoid performance issues

* Adding more tests

* Fixing test

* i18n extract

* Adding extra method for get restrictions for a team

* Add the new elasticsearch functions to search users with restrictions

* Add missing translation string

* Rename restrictedChannelIds to restrictedToChannels

* Remove ToDo

* Adding the permission to the SystemAdmin role during permissions migrations
2019-04-29 16:56:56 +02:00
Eli Yukelzon
6cc36ab176 [MM-14890] Logs endpoint always returns an empty line (#10559)
* fixed new line being returned in logs, improved the test

* improved the EOL check

* Fixed var shadowning
2019-04-18 10:16:49 +01:00
Jesse Hallam
9ef8c1e8b1 MM-14439: experimental restrict system admin (#10414)
* api4: break out license and config from system

* app: move some config functions from admin.go to config.go

* add ExperimentalSettings.RestrictSystemAdmin

* forbid various actions to restricted system admin

* update default.json

* fix function names in errors
2019-03-08 13:15:28 -05:00
Jesse Hallam
1e462da2d4 MM-14143 config cleanup final (#10374)
* TestGetLicenseFileFromDisk: avoid using fileutils.FindConfigFile

* config: abstract config-related file access, extend memory store

* simplify config validate to avoid file knowledge

* fix relative file tests

* cluster: fix ConfigChanged event

The old and new configurations were swapped when notifying the enterprise code of configuration changes, creating needless instability in propagating config updates across a cluster.

* config/database: ignore duplicates

* test cleanup

* remove unnecessary Save() in test
2019-03-06 15:06:45 -05:00
Christopher Poile
8bd182c38f MM-14145: The config store Set will now Save automatically (#10377)
* MM-14145: The config store Set will now Save automatically

When UpdateConfig (and configStore.Set) is called in admin.go and
config.go, commonStore.Set now takes a store-specific persist function.
It uses that persist function to save the configuration automatically.

Removed: Now callers do not have to call configStore.Save or
app.PersistConfig, and those functions have been removed.

Possible downside: this means a "failed to persist config" error can now
be thrown during a app.UpdateConfig or commonStore.Set call. But
considering application code never really sets a config without saving
it (except in the test cases, which were testing that -- see below), it
seems fine to group these responsibilities.

Also removed: tests for 'set without save'. Since that can not happen
anymore, the tests are not needed.

* Removed Save completely, cleaned up formatting, joined save test with
set tests.

* fixed shadowed variable error
2019-02-28 10:51:42 -05:00
Jesse Hallam
285b646d67 MM-13893: introduce file store (#10243)
* config file store

Introduce an interface and concrete implementation for accessing the config.

This mostly maps 1:1 with the exiting usage in `App`, except for internalizing the watcher. A future change will likely eliminate `App.PersistConfig()` and make this implicit on `Set` or `Patch`

* experimental file test changes

* emoji: move file driver checks from api4 to app

It is no longer possible to app.UpdateConfig and provide an invalid configuration, making it hard to test this case. This check doesn't really belong in the api anyway, since it's a configuration validity check and not a permissions check. Either way, the check now occurs at the App level.

* api4: generate valid public link salts for test

* TestStartServerRateLimiterCriticalError: use mock store to test invalid config

* remove config_test.go

* remove needsSave, and have Load() save to the backing store as necessary

* restore README.md

* move ldap UserFilter check to model isValid checks

* remove databaseStore until ready

* remove unimplemented Patch

* simplify unlockOnce implementation

* revert forgetting to set s.Ldap

* config/file.go: rename ReadOnlyConfigurationError to ErrReadOnlyConfiguration

* config: export FileStore

* add TestFileStoreSave

* improved config/utils test coverage

* restore config/README.md copy

* tweaks

* file store: acquire a write lock on Save/Close to safely close watcher

* fix unmarshal_test.go
2019-02-12 10:19:01 -08:00
Jesse Hallam
3a71709103 MM-13893: refactor config (#10230)
* refactor utils/config* to config/

* pull validateLdapFilter into app

* clean up Config/GetConfig/GetSanitizedConfig usage

Eliminate app.GetConfig() in favour of just using app.Config() directly,
but expose app.GetSanitizedConfig() for when the old behaviour was
required.

* web: isolate config setup

* TestInvitePeopleProvider: make config explicit

* regenerateClientConfig: avoid racey map access

* integrate watch flag into app.ConfigFile option

* make app.Option return an error

* release.mk: only cp static files from config/

* release.mk: fix cp static files from config/

* api4: TestPlugin cleanup

* s/c/cfg/ for clarity

* fix merge conflict

* testlib: allow customization of testlib driver name
2019-02-12 08:37:54 -05:00
Joram Wilander
2ca222033c MM-10658 Change config fields to pointers (#9033)
* MM 10658 Change config fields to pointers (#8898)

* Change fields of config structs to pointers and set defaults

MM-10658 https://github.com/mattermost/mattermost-server/issues/8841

* Fix tests that go broken during switching config structs to pointers

MM-10658 https://github.com/mattermost/mattermost-server/issues/8841

* Apply changes of current master while switching config structs to pointers

MM-10658 https://github.com/mattermost/mattermost-server/issues/8841

* Fix new config pointer uses

* Fix app tests

* Fix mail test

* remove debugging statement

* fix TestUpdateConfig

* assign config consistently

* initialize AmazonS3Region in TestS3TestConnection

* initialize fields for TestEmailTest

* fix TestCheckMandatoryS3Fields
2019-01-31 08:12:01 -05:00
Christopher Speller
da265fbaf7 Moving app from singular to being created for every request (#9889)
* Moving app from singular to being created for every request.

* Automatic refactor

* Adding license header

* Feedback fixes
2018-11-28 10:56:21 -08:00
Christopher Speller
ecade2f1ec MM-12849 Moving all non request scoped items to Server struct (#9806)
* Moving goroutine pool

* Auto refactor

* Moving plugins.

* Auto refactor

* Moving fields to server

* Auto refactor

* Removing siteurl duplication.

* Moving reset of app fields

* Auto refactor

* Formatting

* Moving niling of Server to after last use

* Fixing unit tests.
2018-11-07 10:20:07 -08:00
Harshil Sharma
af275fe924 #MM-12130 changes for custom service terms (#9450)
* #MM-12130 changes for custom service terms

* Fixed styling

* Added getServiceTerms API

* removed unnecessary panic

* removed custom service terms text from flat config

* reverted user sql store as those changes are no longer needed

* added tests

* Updated a config key to be more standard

* Added copyright info

* Loading service terms only if the feature is enabled

* Loading service terms only if the feature is enabled

* removed unused index

* added createservice termns API

* made a param to bool instead of string

* added createservice termns API

* review fixes

* fixed styling

* Minor refactoring

* removed saveConfig and loadConfig magic

* added empty service terms text check to createServiceTerms API

* refactoed some urls to be terms_of_service instead of service_terms

* removed check for support settings

* changed URLs in tests

* removed unused code

* fixed a bug

* added service termd id in conif

* fixed a test

* review fixes

* minor fixes

* Fixed TestCreateServiceTerms
2018-09-26 16:49:22 -04:00
Jesús Espino
7636650a25 Migrate to idiomatic error handling app/a*.go and app/b*.go (#9455) 2018-09-25 08:42:06 -04:00
Jesús Espino
a08df883b4 Move file backend to its own service (#9435)
* Move file backend to its own service

* Moving utils/inbucket to mailservice package
2018-09-20 10:07:03 -07:00
Christopher Speller
686c2fbab7 Structured logging (#8673)
* Implementing structured logging

* Changes to en.json to allow refactor to run.

* Fixing global logger

* Structured logger initalization.

* Add caller.

* Do some log redirection.

* Auto refactor

* Cleaning up l4g reference and removing dependancy.

* Removing junk.

* Copyright headers.

* Fixing tests

* Revert "Changes to en.json to allow refactor to run."

This reverts commit fd8249e99b.

* Fixing some auto refactor strangeness and typo.

* Making keys more human readable.
2018-04-27 12:49:45 -07:00
Joram Wilander
191e1ad556 MM-8180 Improve error message for testing smtp connections (#8608)
* Improve error message for testing smtp connections

* Update error message
2018-04-12 11:36:56 -04:00
Harrison Healey
0a6b96cb40 MM-9849 Added tracking of which settings are set through environment variables (#8586)
* MM-9849 Added tracking of which settings are set through environment variables

* Removed old version of viper

* Added forked version of viper

* Fixed unit tests

* Fixed more unit tests

* Removed copy from App.GetEnvironmentConfig
2018-04-09 18:16:11 +02:00
Joram Wilander
4a1802c039 MM-9664 Add invalidation metrics for store caches (#8340)
* Add invalidation metrics for store caches

* Increment session invalidation metric

* Fix tests
2018-03-05 10:35:26 -05:00
Chris
a6309aaf48 Remove license globals entirely (#8229)
* remove license globals entirely

* fix infinite recursion

* test fix
2018-02-09 10:04:48 -06:00
Chris
7e5ce97668 Remove global cfg vars (#8099)
* remove global cfg vars

* enterprise update
2018-01-12 09:02:11 -05:00
Chris
1d9efd0e39 Remove global config watcher (#8080)
* remove global config watcher

* keep config watcher disabled for tests

* compile fix

* fix resource leak
2018-01-11 13:23:41 -08:00
Yusuke Nemoto
ddd99f7476 PLT-6896 per-paging for logs (#7903)
* PLT-6896 Read logs from last

* Getting rid of file.Stats

* remove deprecated value

* Make non-reassigned value constant
2017-12-13 13:04:55 -06:00
Chris
ce2b2be5de Refactoring cfg refs and load / save functions (#7749)
* refactoring cfg refs and load / save functions

* improve error output
2017-10-31 10:39:31 -04:00
Chris
7ed011745a Remove more global refs / state (#7723)
* remove more global refs / state

* fix job enterprise initialization

* fix api4 test compilation

* saml api endpoints fix
2017-10-26 15:21:22 -04:00
Chris
8e19ba029f Reduce utils.Cfg references (#7650)
* app.UpdateConfig method

* test fix

* another test fix

* the config override option as-was is just error prone, remove it for now

* derp
2017-10-18 15:36:43 -07:00
Chris
bff2b5e735 Miscellaneous app cleanup (#7594)
* app cleanup

* whoops, forgot a file

* some minor cleanup

* longer container deadline

* defensive checks
2017-10-09 14:59:48 -07:00
Chris
0f66b6e726 store/sqlstore cleanup and postgres tests (#7595)
* sqlstore cleanup / postgres tests

* remove stopped containers

* cmd/platform compile fix

* remove test-postgres target from makefile
2017-10-09 13:16:14 -04:00
Chris
4e79d2d4d0 remove jobs.Srv and other jobs-related globals (#7535) 2017-09-29 10:29:29 +01:00
Chris
8c80cdde38 remove remaining Global() calls (outside of tests) (#7521) 2017-09-28 00:52:34 +08:00
Chris
49fe5fbf3d Move sql store code into store/sqlstore package (#7502)
* move sql store code into store/sqlstore package

* move non-sql constants back up to store

* fix api test

* derp
2017-09-25 10:11:25 -04:00
Chris
266ff86702 remove more global refs (#7480) 2017-09-21 10:13:34 +01:00
Chris
ac74066f0e remove einterface gets (#7455) 2017-09-19 18:31:35 -05:00
Christopher Speller
29fca51821 Renaming repo 2017-09-06 23:11:59 -07:00
Chris
1adfd0e9be app type transition (#7167) 2017-09-06 17:12:54 -05:00
Saturnino Abril
f968c56890 Revert "[PLT-7475] Add S3 region to system console and add S3 validation (#7373)" (#7387)
This reverts commit 8d680cf64e.

Move to 4.3.
Will discuss more if validation is necessary.
2017-09-06 11:02:42 -05:00
Saturnino Abril
8d680cf64e [PLT-7475] Add S3 region to system console and add S3 validation (#7373)
* add S3 region to system console and add S3 validation

* update translation message

* add bool as return value to Validate* functions

* update Validate* functions to be pure
2017-09-06 03:42:18 +08:00
Corey Hulen
7843dc3cfa Fixing leader election (#7267) 2017-09-05 07:58:30 -07:00
George Goldberg
e85b5fb988 App: NewLocAppError -> NewAppError (#7327)
* App: NewLocAppError -> NewAppError

* Remove statuscode that got missed.
2017-09-01 11:42:02 -04:00
Carlos Tadeu Panato Junior
12a9180807 [PLT-1015] Make all Mattermost system emails Multi-Part, with both a HTML and Plain Text version. (#5698)
* Implementation to Make all Mattermost system emails Multi-Part, with both a HTML and Plain Text version

* update lib

* update code per review

* update to use the mattermost repo
2017-08-21 11:31:13 -07:00
Chris
ff0811e4a4 lint all the things (#7154)
* lint all the things

* remove extra echo
2017-08-09 13:35:49 -07:00
George Goldberg
0495a51949 PLT-6916: Redesign the jobs package and Jobserver. (#6733)
This commit redesigns the jobserver to be based around an architecture
of "workers", which carry out jobs of a particular type, and "jobs"
which are a unit of work carried by a particular worker. It also
introduces "schedulers" which are responsible for scheduling jobs of a
particular type automatically (jobs can also be scheduled manually when
apropriate).

Workers may be run many times, either in instances of the platform
binary, or the standalone jobserver binary. In any mattermost cluster,
only one instance of platform OR jobserver must run the schedulers. At
the moment this is controlled by a config variable, but in future will
be controlled through the cluster leader election process.
2017-07-07 15:21:02 +01:00
Christopher Speller
9659a6da06 Stage 1 of caching layer. Framework (#6693) 2017-06-27 08:02:08 -07:00
Corey Hulen
b5d2c12d22 Setting per-paging for logs to load 10K initially like the old apis (#6697)
* Setting per-paging for logs to load 10K initially like the old apis

* Fixing unit test
2017-06-20 13:13:02 -07:00
Corey Hulen
36f216cb7c PLT-6080 moving clustering to memberlist (#6499)
* PLT-6080 adding cluster discovery service

* Adding memberlist lib

* Adding memberlist lib

* WIP

* WIP

* WIP

* WIP

* Rolling back config changes

* Fixing make file

* Fixing config for cluster

* WIP

* Fixing system console for clustering

* Fixing default config

* Fixing config

* Fixing system console for clustering

* Tweaking hub setting

* Bumping up time

* merging vendor dir

* Updating vendor dir

* Fixing unit test

* Fixing bad merge

* Remove some testing code

* Moving comment

* PLT-6868 adding db ping retry

* Removing unused loc strings

* Adding defer to cancel
2017-06-19 08:44:04 -07:00
Corey Hulen
acc991dea1 PLT-6317 disable file watcher while running from makefile (#6103)
* PLT-6317 disable file watcher while running from make file

* Removing debug stmt
2017-04-17 10:55:03 -04:00