* Add psd preview support
* Adding required vendor files
* Adding oov/psd license to NOTICE.txt file
* Adding the license notice for gopherjs to the NOTICE.txt
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Adds Advanced Logging to server. Advanced Logging is an optional logging capability that allows customers to send log records to any number of destinations.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
Allows developers to specify discrete log levels as well as the standard trace, debug, info, ... panic. Existing code and logging API usage is unchanged.
Log records are emitted asynchronously to reduce latency to the caller. Supports hot-reloading of logger config, including adding removing targets.
Advanced Logging is configured within config.json via "LogSettings.AdvancedLoggingConfig" which can contain a filespec to another config file, a database DSN, or JSON.
* MM-25710: Use an efficient cache serialization algorithm
We investigate 3 packages for selecting a suitable replacement
for gob encoding. The algorithm chosen was msgpack which gives
a decent boost over the standard gob encoding.
Any external schema dependent algorithms like protobuf, flatbuffers, avro,
capn'proto were not considered as that would entail converting the model structs
into separate schema objects and then code generating the Go structs.
It could be done theoretically at a later stage specifically for structs
which are in the hot path. This is a general solution for now.
The packages considered were:
- github.com/tinylib/msgp
- github.com/ugorji/go/codec
- github.com/vmihailenco/msgpack/v5
msgp uses code generation to generate encoding/decoding code without the reflection overhead.
Theoretically, therefore this is supposed to give the fastest performance. However, a major
flaw in that package is that it works only at a file/directory level, not at a package level.
Therefore, for structs which are spread across multiple files, it becomes near to impossible
to chase down all transitive dependencies to generate the code. Even if that's done, it fails
on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611)
Therefore, we are left with 2 choices. Both of them use the same underlying algorithm.
But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples
comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance.
msgpack/v5 came out to be the fastest by a small margin.
benchstat master.txt ugorji.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9)
LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10)
LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8)
LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10)
11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10)
LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8)
LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9)
LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10)
In general, we can see that the time to marshal/unmarshal pays off as the size of the struct
increases.
We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory.
Since we are interested in fastest speed, we choose msgpack/v5.
As a future optimization, we can use a mix of msgpack and msgp for hot structs.
To do that, we would need to shuffle around some code so that for the hot struct,
all its dependencies are in the same file.
Let's use this in production for some time, watch grafana graphs for the hottest caches
and come back to optimizing this more once we have more data.
Side note: we have to do with micro-benchmarks for the time being, because all the caches
aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests
and do comparisons.
* Bring back missing import
* Fix tests
* Enable gossip encryption
* Fix order
* Auto-generate key
* Update gorp fork to include BeginTx
* Add a test for InsertIfExists
And point gorp to a custom branch for now
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* [MM-20979] Add first implementation of the Bleve search engine
* Fix i18n
* Migrate searchengine utils tests
* Fix linter
* Don't add allTermsQ if both termQueries and notTermQueries are empty
* Fix test that should work if user is system admin
* Modify naming according to review comments
* Abstract getIndexDir function
* Extracting bleve engine name as a constant
* Merge both Indexer interfaces into one
* Add worker stopped message
* Allow worker to be started/stopped with config change
* Use constants for index names
* Modify test order
* Fix linter
* Trying to unlock the CI
Add auditing to server CLI.
Also:
- simplify auditing in API layer
- reduce number of AddMeta calls
- have models serialize themselves
- more consistent field naming
* MM-23222 add file target (with rotation) to audit
* MM-23222 mirror syslog audits to local filesystem
* provides config options for file name, max size, max age
* rotates files based on max size and max age; delete as needed based on max backups
* include cluster id in log records
* sort meta data fields
* New auditing API outputting to syslog via TLS
* New config section for specifying remote syslog server IP, port, and cert.
* Legacy audit API retained for access history feature
* MM-22282 Add Ephemeral response when using mentions without permission and add new prop to disable mention highlights on client
* MM-22622 Make test name test actually what it does and fix comment style
* MM-22622 Check ephemeral post created when post create with mentions on API
* MM-22622 More tests for App>CreatePost
* MM-22622 Make DisableMentionHighlights more concise and rename ephemeral post
* MM-22622 Dont send ephemeral message for system message created by user
* Trigger build
* Remove unnecessary struct2interface dependency
Running go mod tidy on the repo removes it.
This was also preventing 1.14beta from running the repo,
because now it verifies modules.txt with go.mod, which had a mismatch.
* fixing CI
* Fix test-server target
* Fix some discrepancies in vendor
* Removing the Makefile target for now
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Add MLOG for sql store
* Update printf function
* Update store/sqlstore/supplier.go
Co-Authored-By: Martin Kraft <martin@upspin.org>
* Update go mod
* update go mod and tidy it up
* update vendor folder
This change is being made to address an issue where the go-i18n
translation library would result in partial-translations when a
given language dictionary was missing a given plural keyword. The
improvement made here leads the translation library to try an
'other' keyword lookup if the first plural keyword fails to have
a value.
This change was not accepted upstream due to concern regarding
changing the behavior, so we are using a fork at this time to
address the issue.
