Commit Graph

70 Commits

Author SHA1 Message Date
Jonathan
375c0632fa PLT-7503: Create Message Export Scheduled Task and CLI Command (#7612)
* Created message export scheduled task

* Added CLI command to immediately kick off an export job

* Added email addresses for users joining and leaving the channel to the export

* Added support for both MySQL and PostgreSQL

* Fixing gofmt error

* Added a new ChannelMemberHistory store and associated tests

* Updating the ChannelMemberHistory channel as users create/join/leave channels

* Added user email to the message export object so it can be included in the actiance export xml

* Don't fail to log a leave event if a corresponding join event wasn't logged

* Adding copyright notices

* Adding message export settings to daily diagnostics report

* Added System Console integration for message export

* Cleaned up TODOs

* Made batch size configurable

* Added export from timestamp to CLI command

* Made ChannelMemberHistory table updates best effort

* Added a context-based timeout option to the message export CLI

* Minor PR updates/improvements

* Removed unnecessary fields from MessageExport object to reduce query overhead

* Removed JSON functions from the message export query in an effort to optimize performance

* Changed the way that channel member history queries and purges work to better account for edge cases

* Fixing a test I missed with the last refactor

* Added file copy functionality to file backend, improved config validation, added default config values

* Fixed file copy tests

* More concise use of the testing libraries

* Fixed context leak error

* Changed default export path to correctly place an 'export' directory under the 'data' directory

* Can't delete records from a read replica

* Fixed copy file tests

* Start job workers when license is applied, if configured to do so

* Suggestions from the PR

* Moar unit tests

* Fixed test imports
2017-11-30 09:07:04 -05:00
Chris
eb1a00ef5f Reorganize file util functionality (#7848)
* reorganize file util functionality

* fix api test compilation

* fix rebase issue
2017-11-16 16:04:27 -05:00
Corey Hulen
b3b46a01aa PLT-5813 support SAML sync via LDAP (#7668)
* PLT-5813 support SAML sync via LDAP

* Cleaning up based on review
2017-10-31 08:37:34 -07:00
George Goldberg
76bd1bb212 PLT-7705: API to get data retention policy. (#7539)
* PLT-7705: API to get data retention policy.

* Fix review comments.
2017-10-02 12:43:21 +01:00
Chris
4e79d2d4d0 remove jobs.Srv and other jobs-related globals (#7535) 2017-09-29 10:29:29 +01:00
George Goldberg
e8ca3d64f4 PLT-7470: Add metrics for searches. (#7507) 2017-09-25 13:44:44 -07:00
Corey Hulen
81c18a01bd PLT-7542 Converting LDAP sync to the job server (#7452)
* PLT-7542 Converting LDAP sync to the job server

* Fixing minor issues

* Fixing build failure

* Translate error message

* Translate error message

* Translate error message

* Translate error message

* Fixing merge

* Fixing bad merge
2017-09-25 13:30:33 -07:00
George Goldberg
265392fa48 PLT-7667: Clean up elasticsearch indexes when running data retention jobs. (#7481) 2017-09-21 17:54:31 -04:00
Chris
266ff86702 remove more global refs (#7480) 2017-09-21 10:13:34 +01:00
Christopher Speller
29fca51821 Renaming repo 2017-09-06 23:11:59 -07:00
Corey Hulen
7843dc3cfa Fixing leader election (#7267) 2017-09-05 07:58:30 -07:00
Joram Wilander
e2042c4b65 Add built-in plugin for getting LDAP attributes (#7317) 2017-09-01 14:28:15 -04:00
George Goldberg
22459ee17a PLT-7302: Aggregate Elasticsearch indexes over a certain age. (#7224)
* PLT-7302: Aggregate Elasticsearch indexes over a certain age.

This is done by a scheduled daily job, in order to keep the shard count
to a sensible level in Elasticsearch.

* Use map[string]string instead of StringMap
2017-08-17 10:05:17 -04:00
Corey Hulen
29a7e182a6 PLT-6079 Adding leader election (#7105) 2017-08-16 10:29:45 -07:00
George Goldberg
f30c0b5d38 PLT-6596: Elasticsearch interface changes. (#6993) 2017-07-21 08:38:44 +08:00
George Goldberg
fe368a7456 PLT-6595: API to purge Elasticsearch indexes. (#6971) 2017-07-19 09:43:05 +01:00
George Goldberg
83d53ea98c PLT-6475: Elasticsearch Indexing Worker. (#6879) 2017-07-11 09:09:15 +01:00
George Goldberg
39b071bc0e Add optional config parameter to test elasticsearch endpoint. (#6877) 2017-07-07 12:12:14 -04:00
George Goldberg
db2f6cf076 PLT-6976: Elasticsearch capitalisation and tests. (#6839)
* Fixes Elasticsearch to have consistent capitalisation everywhere
  across the code and UI (except the config file unfortunately).
* Adds basic unit tests for Elastichsearch.
* Adds a Elasticsearch docker image to the Makefile to enable testing
  the Elasticsearch feature.
2017-07-07 15:36:17 +01:00
George Goldberg
0495a51949 PLT-6916: Redesign the jobs package and Jobserver. (#6733)
This commit redesigns the jobserver to be based around an architecture
of "workers", which carry out jobs of a particular type, and "jobs"
which are a unit of work carried by a particular worker. It also
introduces "schedulers" which are responsible for scheduling jobs of a
particular type automatically (jobs can also be scheduled manually when
apropriate).

Workers may be run many times, either in instances of the platform
binary, or the standalone jobserver binary. In any mattermost cluster,
only one instance of platform OR jobserver must run the schedulers. At
the moment this is controlled by a config variable, but in future will
be controlled through the cluster leader election process.
2017-07-07 15:21:02 +01:00
George Goldberg
f2898927f1 PLT-6474: Server: Add elasticsearch/test endpoint to API. (#6792) 2017-06-29 17:40:14 -04:00
Corey Hulen
36f216cb7c PLT-6080 moving clustering to memberlist (#6499)
* PLT-6080 adding cluster discovery service

* Adding memberlist lib

* Adding memberlist lib

* WIP

* WIP

* WIP

* WIP

* Rolling back config changes

* Fixing make file

* Fixing config for cluster

* WIP

* Fixing system console for clustering

* Fixing default config

* Fixing config

* Fixing system console for clustering

* Tweaking hub setting

* Bumping up time

* merging vendor dir

* Updating vendor dir

* Fixing unit test

* Fixing bad merge

* Remove some testing code

* Moving comment

* PLT-6868 adding db ping retry

* Removing unused loc strings

* Adding defer to cancel
2017-06-19 08:44:04 -07:00
Christopher Speller
04088009f8 Adding force flag to migrate auth command (#6593) 2017-06-06 14:42:53 -04:00
Harrison Healey
577ed27f1b PLT-6408 Framework for job server (#6404)
* Added initial job server

* Added job server to be ran as part of platform

* Added test job to the enterprise repo

* Fixed job server not loading license

* Renamed job package to jobs

* Fixed TE not being buildable

* Added JobStatus table to database

* Changed fields used by JobStatus

* Added APIs to query job status

* Added config change listener to server

* Added option to run job server from Makefile

* Added ability to enable/disable jobs from config

* Commented out placeholder for search indexing job

* Fixed govet

* Removed debug messages and fixed job api init message
2017-05-18 15:05:57 -04:00
George Goldberg
0db5e3922f PLT-6472: Basic Elastic Search implementation. (#6382)
* PLT-6472: Basic Elastic Search implementation.

This currently supports indexing of posts at create/update/delete time.
It does not support batch indexing or reindexing, and does not support
any entities other than posts yet. The purpose is to more-or-less
replicate the existing full-text search feature but with some of the
immediate benefits of using elastic search.

* Alter settings for AWS compatability.

* Remove unneeded i18n strings.
2017-05-18 11:26:52 -04:00
Corey Hulen
e62afeace0 Adding slow pump detection to web_conn and better metrics (#6114)
* Adding slow pump detection to web_conn and better metrics

* Fixing bad merge

* Fixing typo
2017-04-21 12:38:26 -04:00
Harrison Healey
fb6f2a123c PLT-5860 Updated copyright date (#6058)
* PLT-5860 Updated copyright date in about modal

* PLT-5860 Updated copyright notice in JSX files

* PLT-5860 Updated copyright notice in go files

* Fixed misc copyright dates

* Fixed component snapshots
2017-04-12 08:27:57 -04:00
Harrison Healey
c7bdce8a66 PLT-5900 Removed automatic configuration of SiteURL (#5952)
* Revert "PLT-5765 Passed SiteURL to SendNotifications (#5705)"

This reverts commit a8e68bd890.

* Removed automatic configuration of SiteURL

* Reverted unintentional config changes

* Updated help text

* Added link to docs in Site URL warning

* Fixed merge conflict
2017-04-04 11:54:52 -04:00
George Goldberg
7f266c1951 Merge branch 'release-3.7' into master. 2017-03-17 16:29:26 +00:00
Joram Wilander
0bc3e46082 Implement GET /logs endpoint for APIv4 (#5778) 2017-03-16 11:59:44 -07:00
Harrison Healey
a8e68bd890 PLT-5765 Passed SiteURL to SendNotifications (#5705) 2017-03-10 10:18:18 +00:00
Joram Wilander
83291e4c8f Merge 3.7 changes into master (#5679)
* Save schema version for 3.7 (#5675)

* Adding index and cache to reactinos store (#5654)

* Fix badge count for push notifications (#5672)
2017-03-07 16:05:01 -05:00
Christopher Speller
33b8b72a02 Adding index and cache to reactinos store (#5654) 2017-03-07 09:37:00 -05:00
Corey Hulen
847c716ea2 Example of how to call metrics (#5621) 2017-03-06 10:15:37 -05:00
Christopher Speller
83c113595a Revert "Adding caching to get channel member (#5518)"
This reverts commit ba028ed74b.
2017-02-27 17:43:23 -05:00
Christopher Speller
ba028ed74b Adding caching to get channel member (#5518) 2017-02-24 09:15:36 -05:00
enahum
748a416961 PLT-3193 Add channel notification preferences for push and email noti… (#5500)
* PLT-3193 Add channel notification preferences for push and email notifications

* unit tests, model validation and localization

* Feedback review

* Adding back allowFromCache check

* Setting push and email to use default settings

* Move props as constants

* address feedback
2017-02-23 11:08:48 -03:00
Christopher Speller
2a9c1afd44 Adding webhook metrics (#5348) 2017-02-10 07:05:12 -08:00
Joram Wilander
58dcf6984b Add cluster request time logging for master (#5347) 2017-02-10 07:04:14 -08:00
Christopher Speller
9934954d18 Adding websocket event metrics (#5342) 2017-02-10 07:01:08 -08:00
Joram Wilander
097289f8e4 Merge 3.6.2 into master (#5211)
* Add webhook cache

* Add channel by name cache

* Fxing profiles in channels cache

* Fix merge
2017-01-27 14:07:34 -05:00
Joram Wilander
d3a285e64d Migrate functions to app package (#5106)
* Refactor and move session logic into app package

* Refactor email functions into the app package

* Refactor password update into app package

* Migrate user functions to app package

* Move team functions into app package

* Migrate channel functions into app package

* Pass SiteURL through to app functions

* Update based on feedback
2017-01-19 09:00:13 -05:00
Christopher Speller
96b4e0ed87 Adding metrics to cache for profile by ids api (#4925) 2017-01-03 11:57:38 -03:00
Christopher Speller
f96173528f Adding metrics for caching mechanisms (#4828) 2016-12-19 10:16:22 -05:00
Christopher Speller
86fb0d87a3 Adding caching layer to some posts calls (#4779) 2016-12-14 00:24:24 -03:00
Corey Hulen
c4974374d9 PLT-4853 Adding cache purging to the server (server) (#4735)
* PLT-4853 Adding caching invalidation to HA

* PLT-4853 Adding cach purging to the server
2016-12-08 10:18:15 -05:00
Corey Hulen
0383c58d01 PLT-4357 Adding stats to HA (#4638)
* PLT-4357 adding stats to HA

* PLT-4357 adding stats to HA

* Fixing getting stats from the current server
2016-11-28 10:40:02 -05:00
Corey Hulen
7961599b2e PLT-4357 adding performance monitoring (#4622)
* WIP

* WIP

* Adding metrics collection

* updating vendor packages

* Adding metrics to config

* Adding admin console page for perf monitoring

* Updating glide

* switching to tylerb/graceful
2016-11-22 14:05:54 -05:00
enahum
772b57386f PLT-4550 move webrtc to TE (#4440)
* PLT-4550 move webrtc to TE

* remove webrtc from makefile enterprise tests
2016-11-04 12:05:32 -04:00
Joram Wilander
0234f793f2 EE: PLT-4512 Show secret in addition to QR code when activating MFA (#4427)
* EE: Update MFA to display secret for manual entry

* Width adjustments for secret (#4423)

* Add unit test
2016-11-03 10:41:11 -04:00