Commit Graph

14 Commits

Author SHA1 Message Date
Jonathan
375c0632fa PLT-7503: Create Message Export Scheduled Task and CLI Command (#7612)
* Created message export scheduled task

* Added CLI command to immediately kick off an export job

* Added email addresses for users joining and leaving the channel to the export

* Added support for both MySQL and PostgreSQL

* Fixing gofmt error

* Added a new ChannelMemberHistory store and associated tests

* Updating the ChannelMemberHistory channel as users create/join/leave channels

* Added user email to the message export object so it can be included in the actiance export xml

* Don't fail to log a leave event if a corresponding join event wasn't logged

* Adding copyright notices

* Adding message export settings to daily diagnostics report

* Added System Console integration for message export

* Cleaned up TODOs

* Made batch size configurable

* Added export from timestamp to CLI command

* Made ChannelMemberHistory table updates best effort

* Added a context-based timeout option to the message export CLI

* Minor PR updates/improvements

* Removed unnecessary fields from MessageExport object to reduce query overhead

* Removed JSON functions from the message export query in an effort to optimize performance

* Changed the way that channel member history queries and purges work to better account for edge cases

* Fixing a test I missed with the last refactor

* Added file copy functionality to file backend, improved config validation, added default config values

* Fixed file copy tests

* More concise use of the testing libraries

* Fixed context leak error

* Changed default export path to correctly place an 'export' directory under the 'data' directory

* Can't delete records from a read replica

* Fixed copy file tests

* Start job workers when license is applied, if configured to do so

* Suggestions from the PR

* Moar unit tests

* Fixed test imports
2017-11-30 09:07:04 -05:00
George Goldberg
a06830b2f8 PLT-7644: Improve job scheduler architecture. (#7532) 2017-09-28 09:11:13 -07:00
Corey Hulen
81c18a01bd PLT-7542 Converting LDAP sync to the job server (#7452)
* PLT-7542 Converting LDAP sync to the job server

* Fixing minor issues

* Fixing build failure

* Translate error message

* Translate error message

* Translate error message

* Translate error message

* Fixing merge

* Fixing bad merge
2017-09-25 13:30:33 -07:00
George Goldberg
22459ee17a PLT-7302: Aggregate Elasticsearch indexes over a certain age. (#7224)
* PLT-7302: Aggregate Elasticsearch indexes over a certain age.

This is done by a scheduled daily job, in order to keep the shard count
to a sensible level in Elasticsearch.

* Use map[string]string instead of StringMap
2017-08-17 10:05:17 -04:00
George Goldberg
00e8b6621a PLT-6595 (Client): Elasticsearch indexing system console UI (#6991)
* PLT-6595: System Console for Elasticsearch Job Management.

* Fixing UI issues

* Fixing colors

* ESLint Fixes.

* Update test snapshots.

* Fixing cancel button

* Fix review comments.

* Config capitalisation.

* Review fixes.
2017-08-01 13:11:29 +01:00
George Goldberg
6c6f2a1138 PLT-6595-Server: Job Management APIs. (#6931)
* PLT-6595-Server: Job Management APIs.

* MANAGE_JOBS Permission

* Fix test.
2017-07-20 08:25:35 -07:00
George Goldberg
0495a51949 PLT-6916: Redesign the jobs package and Jobserver. (#6733)
This commit redesigns the jobserver to be based around an architecture
of "workers", which carry out jobs of a particular type, and "jobs"
which are a unit of work carried by a particular worker. It also
introduces "schedulers" which are responsible for scheduling jobs of a
particular type automatically (jobs can also be scheduled manually when
apropriate).

Workers may be run many times, either in instances of the platform
binary, or the standalone jobserver binary. In any mattermost cluster,
only one instance of platform OR jobserver must run the schedulers. At
the moment this is controlled by a config variable, but in future will
be controlled through the cluster leader election process.
2017-07-07 15:21:02 +01:00
Harrison Healey
577ed27f1b PLT-6408 Framework for job server (#6404)
* Added initial job server

* Added job server to be ran as part of platform

* Added test job to the enterprise repo

* Fixed job server not loading license

* Renamed job package to jobs

* Fixed TE not being buildable

* Added JobStatus table to database

* Changed fields used by JobStatus

* Added APIs to query job status

* Added config change listener to server

* Added option to run job server from Makefile

* Added ability to enable/disable jobs from config

* Commented out placeholder for search indexing job

* Fixed govet

* Removed debug messages and fixed job api init message
2017-05-18 15:05:57 -04:00
Harrison Healey
fb6f2a123c PLT-5860 Updated copyright date (#6058)
* PLT-5860 Updated copyright date in about modal

* PLT-5860 Updated copyright notice in JSX files

* PLT-5860 Updated copyright notice in go files

* Fixed misc copyright dates

* Fixed component snapshots
2017-04-12 08:27:57 -04:00
Corey Hulen
6bf080393d Fixing race conditions in the code base (#5966)
* Adding initial race detector

* Remove setting of config twice

* Fixing config file watch and config reload on license save

* Fixing config file watch and config reload on license save

* Fixing build error

* Fixing locking issue

* Fixing makefile

* Fixing race in config

* Fixing race in status unit test

* Adding EE race tests

* Fixing race in cluster info

* Removing code that's isn't needed

* Fixing some more races

* Fixing govet issue
2017-04-04 14:42:07 -04:00
Corey Hulen
1359f7f391 Adding go vet from hack-a-thon (#5328)
* Adding go vet to the api package

* Adding go vet to app package

* Adding go vet to manualtesting package

* Adding go vet to the model package

* Adding go vet to the store dir

* Adding go vet to utils package

* Adding missing dirs with go files

* Fixing up makefile

* Fixing up makefile

* Removing root dir
2017-02-09 21:39:15 +00:00
Christopher Speller
f5375254f9 Adding migration support to LDAP from other account types (#3655) 2016-07-26 17:39:51 -04:00
Christopher Speller
8005911ff6 Syncronization interval now set on config save. (#3421) 2016-06-28 15:10:50 -04:00
Christopher Speller
ea3342aa6c Adding LDAP Syncronization (#3219) 2016-06-03 09:33:59 -04:00