Commit Graph

36 Commits

Author SHA1 Message Date
Harrison Healey
4e59a27293 Move HTTPService and ConfigService into services package (#9422)
* Move HTTPService and ConfigService into utils package

* Re-add StaticConfigService

* Move config and http services into their own packages
2018-09-26 12:42:51 -04:00
George Goldberg
8766690c81 MM-10502: Only cluster master should run job schedulers. (#9174)
* MM-10502: Only cluster master should run job schedulers.

* Use sync.Map for thread safety.

* Fix tests.
2018-07-31 07:40:23 -07:00
George Goldberg
60359b4bae MM-10269: Make jobserver work without restart. (#9005) 2018-06-26 13:30:33 -04:00
George Goldberg
51bd710ecd MM-9728: Online migration for advanced permissions phase 2 (#8744)
* MM-9728: Online migration for advanced permissions phase 2

* Add unit tests for new store functions.

* Move migration specific code to own file.

* Add migration state function test.

* Style fixes.

* Add i18n strings.

* Fix mocks.

* Add TestMain to migrations package tests.

* Fix typo.

* Fix review comments.

* Fix up the "Check if migration is done" check to actually work.
2018-05-14 15:59:04 +01:00
Christopher Speller
686c2fbab7 Structured logging (#8673)
* Implementing structured logging

* Changes to en.json to allow refactor to run.

* Fixing global logger

* Structured logger initalization.

* Add caller.

* Do some log redirection.

* Auto refactor

* Cleaning up l4g reference and removing dependancy.

* Removing junk.

* Copyright headers.

* Fixing tests

* Revert "Changes to en.json to allow refactor to run."

This reverts commit fd8249e99b.

* Fixing some auto refactor strangeness and typo.

* Making keys more human readable.
2018-04-27 12:49:45 -07:00
Martin Kraft
ffa24fbcb2 MM-8834: Changes separator between jobs message and details. (#8589) 2018-04-09 09:39:57 -04:00
Jesús Espino
014a3b6a60 Fixing misspell errors (#8544) 2018-03-29 10:04:54 -04:00
Chris
e8943936c5 general cleanup (#8387) 2018-03-07 12:36:40 -06:00
Pierre de La Morinerie
07fd7aeeb8 Add tests for the platform server command (#8231)
* Cleanup app state on initialization error

When returning an initialization error, the app state was not cleaned
up. This is especially visible during tests, as `appCount` is not
decremented, and makes the new app initialization fail.

* Test the `platform server` command

As the `platform server` command only exits when interrupted by
a signal, it is not possible to test it as the other cobra
commands. Instead we directly test the actual command function.

The internal command handler is slighly refactored to take
a channel in argument, and registers it as the signal handler.
Nothing very different—except than controlling this channel
from the outside allows the test to send the system signal
itself, thus preventing the server to run forever.
2018-02-12 08:46:32 -08:00
Chris
a6309aaf48 Remove license globals entirely (#8229)
* remove license globals entirely

* fix infinite recursion

* test fix
2018-02-09 10:04:48 -06:00
Chris
4e6cc846a6 Finally remove utils.Cfg (#8113)
* finally remove utils.Cfg

* fix compile error

* another test compilation fix
2018-01-17 13:38:37 -05:00
Chris
7e5ce97668 Remove global cfg vars (#8099)
* remove global cfg vars

* enterprise update
2018-01-12 09:02:11 -05:00
Jesús Espino
dd9ad10d70 [PLT-6936] Translate AppError.Message automatically by default (#8063) 2018-01-08 13:13:24 -05:00
Jonathan
375c0632fa PLT-7503: Create Message Export Scheduled Task and CLI Command (#7612)
* Created message export scheduled task

* Added CLI command to immediately kick off an export job

* Added email addresses for users joining and leaving the channel to the export

* Added support for both MySQL and PostgreSQL

* Fixing gofmt error

* Added a new ChannelMemberHistory store and associated tests

* Updating the ChannelMemberHistory channel as users create/join/leave channels

* Added user email to the message export object so it can be included in the actiance export xml

* Don't fail to log a leave event if a corresponding join event wasn't logged

* Adding copyright notices

* Adding message export settings to daily diagnostics report

* Added System Console integration for message export

* Cleaned up TODOs

* Made batch size configurable

* Added export from timestamp to CLI command

* Made ChannelMemberHistory table updates best effort

* Added a context-based timeout option to the message export CLI

* Minor PR updates/improvements

* Removed unnecessary fields from MessageExport object to reduce query overhead

* Removed JSON functions from the message export query in an effort to optimize performance

* Changed the way that channel member history queries and purges work to better account for edge cases

* Fixing a test I missed with the last refactor

* Added file copy functionality to file backend, improved config validation, added default config values

* Fixed file copy tests

* More concise use of the testing libraries

* Fixed context leak error

* Changed default export path to correctly place an 'export' directory under the 'data' directory

* Can't delete records from a read replica

* Fixed copy file tests

* Start job workers when license is applied, if configured to do so

* Suggestions from the PR

* Moar unit tests

* Fixed test imports
2017-11-30 09:07:04 -05:00
Chris
10c5a927cb more global config ref cleanup (#7802) 2017-11-09 14:46:20 -06:00
Corey Hulen
b3b46a01aa PLT-5813 support SAML sync via LDAP (#7668)
* PLT-5813 support SAML sync via LDAP

* Cleaning up based on review
2017-10-31 08:37:34 -07:00
Chris
c5e8cb25ca simplify things (#7735) 2017-10-30 11:57:24 -05:00
Chris
7ed011745a Remove more global refs / state (#7723)
* remove more global refs / state

* fix job enterprise initialization

* fix api4 test compilation

* saml api endpoints fix
2017-10-26 15:21:22 -04:00
Chris
6f6005c617 Store mocks (#7724)
* store mocks

* add example
2017-10-26 11:36:54 -05:00
George Goldberg
76bd1bb212 PLT-7705: API to get data retention policy. (#7539)
* PLT-7705: API to get data retention policy.

* Fix review comments.
2017-10-02 12:43:21 +01:00
Chris
4e79d2d4d0 remove jobs.Srv and other jobs-related globals (#7535) 2017-09-29 10:29:29 +01:00
George Goldberg
a06830b2f8 PLT-7644: Improve job scheduler architecture. (#7532) 2017-09-28 09:11:13 -07:00
Corey Hulen
81c18a01bd PLT-7542 Converting LDAP sync to the job server (#7452)
* PLT-7542 Converting LDAP sync to the job server

* Fixing minor issues

* Fixing build failure

* Translate error message

* Translate error message

* Translate error message

* Translate error message

* Fixing merge

* Fixing bad merge
2017-09-25 13:30:33 -07:00
George Goldberg
7243aa6751 PLT-6558: Basic data retention job scheduler/worker implementation. (#7449)
* PLT-7639: Batch delete methods for data retention.

* PLT-6558: Basic data retention job worker/scheduler implementation.
2017-09-15 09:53:48 -07:00
Christopher Speller
29fca51821 Renaming repo 2017-09-06 23:11:59 -07:00
Chris
1adfd0e9be app type transition (#7167) 2017-09-06 17:12:54 -05:00
George Goldberg
22459ee17a PLT-7302: Aggregate Elasticsearch indexes over a certain age. (#7224)
* PLT-7302: Aggregate Elasticsearch indexes over a certain age.

This is done by a scheduled daily job, in order to keep the shard count
to a sensible level in Elasticsearch.

* Use map[string]string instead of StringMap
2017-08-17 10:05:17 -04:00
Chris
ff0811e4a4 lint all the things (#7154)
* lint all the things

* remove extra echo
2017-08-09 13:35:49 -07:00
Christopher Speller
0331409488 Add jobserver command to platform binary (#7067)
* Add jobserver command to platform binary

* Fixing jobserver config. Adding jobserver flags.

* Fixing govet in makefile
2017-08-01 08:53:42 -04:00
George Goldberg
00e8b6621a PLT-6595 (Client): Elasticsearch indexing system console UI (#6991)
* PLT-6595: System Console for Elasticsearch Job Management.

* Fixing UI issues

* Fixing colors

* ESLint Fixes.

* Update test snapshots.

* Fixing cancel button

* Fix review comments.

* Config capitalisation.

* Review fixes.
2017-08-01 13:11:29 +01:00
George Goldberg
ff0a790516 PLT-7040: Fix capitalisation of Elasticsearch in Config. (#7020) 2017-07-27 08:48:02 +01:00
George Goldberg
6c6f2a1138 PLT-6595-Server: Job Management APIs. (#6931)
* PLT-6595-Server: Job Management APIs.

* MANAGE_JOBS Permission

* Fix test.
2017-07-20 08:25:35 -07:00
George Goldberg
83d53ea98c PLT-6475: Elasticsearch Indexing Worker. (#6879) 2017-07-11 09:09:15 +01:00
George Goldberg
0495a51949 PLT-6916: Redesign the jobs package and Jobserver. (#6733)
This commit redesigns the jobserver to be based around an architecture
of "workers", which carry out jobs of a particular type, and "jobs"
which are a unit of work carried by a particular worker. It also
introduces "schedulers" which are responsible for scheduling jobs of a
particular type automatically (jobs can also be scheduled manually when
apropriate).

Workers may be run many times, either in instances of the platform
binary, or the standalone jobserver binary. In any mattermost cluster,
only one instance of platform OR jobserver must run the schedulers. At
the moment this is controlled by a config variable, but in future will
be controlled through the cluster leader election process.
2017-07-07 15:21:02 +01:00
Christopher Speller
9659a6da06 Stage 1 of caching layer. Framework (#6693) 2017-06-27 08:02:08 -07:00
Harrison Healey
577ed27f1b PLT-6408 Framework for job server (#6404)
* Added initial job server

* Added job server to be ran as part of platform

* Added test job to the enterprise repo

* Fixed job server not loading license

* Renamed job package to jobs

* Fixed TE not being buildable

* Added JobStatus table to database

* Changed fields used by JobStatus

* Added APIs to query job status

* Added config change listener to server

* Added option to run job server from Makefile

* Added ability to enable/disable jobs from config

* Commented out placeholder for search indexing job

* Fixed govet

* Removed debug messages and fixed job api init message
2017-05-18 15:05:57 -04:00