Fix an issue where context fields logged by server on behalf of plugins didn't contain correct name/value pairs:
- bump Logr version to add key/value methods to sugar logger
- expose factory args when configuring logging with custom target types (needed for FocalBoard to create log target adapter that converts typed fields into slices of interface{} as per plugin logging API)
* Making doc extraction more reliable
* fix error message
* Adding test doc file
* Removing test that not works on CI because the lack of a binary
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34080: Removing sqlite entirely
The initial commit missed removing this blank import.
So the library still remained in our vendor directory.
Removing it for good now.
Bye bye sqlite.
https://mattermost.atlassian.net/browse/MM-34080
* fix go.mod
* Create basic make commands for configuring golang-migrate
* Showcase full flow with new migrations
* Apply PR suggestions
* Migrate over team members
* Update mocks
* Fix specs
* Move columns that added after table creation onto separate stmts
* Put back gorp table definitions
* Fix issues with golang-migrate that not tracks underlying db driver
* Help prompt after new migration and consistent checksum for bindata
* Put gorp mapping back
* Apply PR suggestiong
* Close migrations after they run
* Add migration file to bindata check
* Updates needed
* Reset store_test
* Add copyright
* Apply PR suggestions
* Fix new circleci check
* Put back upgrade step for backwards comp
* Add store test to test migration directions
* Apply PR suggestions
* Add go-bindata to tools
* Apply PR suggestios
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Migrate channel to waitgroup for access the store in parallel
* Addressing PR review comments
* Using a cleanest way of group errors
* Reverting go.mod change
* MM-21012: Revamp websocket implementation
We replace the old gorilla/websocket implementation with the
gobwas/ws library. The gorilla library was in maintenance mode
and had a high level API due to which we cannot use that for
situations where a large number of concurrent connections needs
to be supported.
The ws library is a very low-level library that allows us
to work with raw net.Conns. We make several improvements:
- We completely remove the reader goroutines, and instead
replace them with a manual epoll implementation which sends off
messages to be read when it receives any data on the connection.
This lets us scale to a much larger number of connections.
- The reader buffer is eliminated, because we directly read
from the connection now.
https://mattermost.atlassian.net/browse/MM-21012
```release-notes
Improved the websocket implementation by using epoll manually
to read from a websocket. As a result, the number of goroutines
is expected to go down by half.
```
* fix tests
* fix shadowing errors
* final changes
* windows support!
* Remove pointer to waitgroup
* Fix edge case
* Trigger CI
* Trigger CI
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
There is a race between go mod tidy and go mod vendor
due to https://github.com/golang/go/issues/37376.
However, if we give priority to go mod vendor, then
gopls complains of inconsistent vendoring. We make go mod tidy happy
as it's a more commonly used command than go mod vendor,
and is a common problem faced by other devs too
The package paths were changed. Therefore, go mod tidy
on master failed with:
```
go: finding module for package github.com/hmhealey/go-opengraph/opengraph
go: found github.com/hmhealey/go-opengraph/opengraph in github.com/hmhealey/go-opengraph v0.0.0-20201209151214-f2d823730dba
go: github.com/hmhealey/go-opengraph@v0.0.0-20201209151214-f2d823730dba used for two different module paths (github.com/dyatlov/go-opengraph and github.com/hmhealey/go-opengraph)
```
https://mattermost.atlassian.net/browse/MM-31511
```release-note
NONE
```
* Document extractor service
* Fixing vendor modules
* Addressing PR Review comments
* Some small simplifications
* Fixing a linter complain
* simplifying a bit the code using package variables
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
It looks like maintainers aren't very active on the repo
and https://github.com/nytimes/gziphandler/pull/107 has come to a standstill.
I don't want to wait forever for this to go in. Let's use a replace directive
to point to the fork. When the PR gets merged, we can just bump the dependency
and remove the replace directive.
Re: the PR, copying the text from the PR description
This gives 40-50% improvements in CPU with very minor increase
in memory, just as a drop-in replacement. I think that is a very reasonable tradeoff.
The library is mature and safe to be used in production.
```
name old time/op new time/op delta
GzipHandler_S2k-8 74.9µs ± 2% 34.4µs ± 2% -54.07% (p=0.000 n=10+9)
GzipHandler_S20k-8 379µs ± 1% 226µs ± 3% -40.42% (p=0.000 n=9+10)
GzipHandler_S100k-8 1.95ms ± 2% 1.15ms ± 1% -41.27% (p=0.000 n=9+9)
GzipHandler_P2k-8 24.3µs ±25% 10.7µs ±25% -55.80% (p=0.000 n=10+10)
GzipHandler_P20k-8 132µs ± 2% 75µs ± 1% -42.95% (p=0.000 n=9+10)
GzipHandler_P100k-8 658µs ± 2% 371µs ± 3% -43.68% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
GzipHandler_S2k-8 7.71kB ± 5% 9.13kB ± 7% +18.33% (p=0.000 n=10+10)
GzipHandler_S20k-8 65.1kB ± 3% 70.3kB ± 3% +8.05% (p=0.000 n=10+10)
GzipHandler_S100k-8 348kB ± 4% 382kB ± 2% +9.85% (p=0.000 n=10+10)
GzipHandler_P2k-8 7.60kB ± 1% 7.93kB ± 2% +4.33% (p=0.000 n=10+10)
GzipHandler_P20k-8 64.4kB ± 1% 66.3kB ± 2% +2.92% (p=0.000 n=10+10)
GzipHandler_P100k-8 304kB ± 1% 309kB ± 1% +1.67% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
GzipHandler_S2k-8 21.0 ± 0% 21.0 ± 0% ~ (all equal)
GzipHandler_S20k-8 24.0 ± 0% 24.0 ± 0% ~ (all equal)
GzipHandler_S100k-8 27.0 ± 0% 27.0 ± 0% ~ (all equal)
GzipHandler_P2k-8 21.0 ± 0% 21.0 ± 0% ~ (all equal)
GzipHandler_P20k-8 24.0 ± 0% 24.0 ± 0% ~ (all equal)
GzipHandler_P100k-8 26.0 ± 0% 26.0 ± 0% ~ (all equal)
```
https://mattermost.atlassian.net/browse/MM-28491
MM-27744 disable Zap for unit tests.
Zap has no concept of shutdown or close. Zap is only shutdown when the app exits. Not a problem for console logging, but when creating a new Zap logger that outputs to files on every unit test, that leaves no easy way to clean up until process exit. Depending on what else is running this can exhaust all file handles and cause unit tests to fail.
Zap is now disabled unit tests and uses Logr instead, regardless of config settings. `make test-server` peak file handle usage dropped from ~5K to less than 100.
* Add functionality to update password with password hash
This can be used to update a user's password with a direct password
hash instead of providing the password in plaintext.
* Use test helper for local mode
* Add a config for MM User Limit
* Adding graceful errors for if an administrator invites people passed their user limit
* Including changed vendor files
* Adding unit test
* Fix a bug
* Push up working tests (Thanks Joram)
* Add more cases, clean up logs in code
* One more case
* Refactoring based on PR comments
* Updating i18n
* Some changes based on PR review
* Remove a comment
* Bring back some translations that were somehow removed
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add psd preview support
* Adding required vendor files
* Adding oov/psd license to NOTICE.txt file
* Adding the license notice for gopherjs to the NOTICE.txt
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Adds Advanced Logging to server. Advanced Logging is an optional logging capability that allows customers to send log records to any number of destinations.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
Allows developers to specify discrete log levels as well as the standard trace, debug, info, ... panic. Existing code and logging API usage is unchanged.
Log records are emitted asynchronously to reduce latency to the caller. Supports hot-reloading of logger config, including adding removing targets.
Advanced Logging is configured within config.json via "LogSettings.AdvancedLoggingConfig" which can contain a filespec to another config file, a database DSN, or JSON.
* MM-25710: Use an efficient cache serialization algorithm
We investigate 3 packages for selecting a suitable replacement
for gob encoding. The algorithm chosen was msgpack which gives
a decent boost over the standard gob encoding.
Any external schema dependent algorithms like protobuf, flatbuffers, avro,
capn'proto were not considered as that would entail converting the model structs
into separate schema objects and then code generating the Go structs.
It could be done theoretically at a later stage specifically for structs
which are in the hot path. This is a general solution for now.
The packages considered were:
- github.com/tinylib/msgp
- github.com/ugorji/go/codec
- github.com/vmihailenco/msgpack/v5
msgp uses code generation to generate encoding/decoding code without the reflection overhead.
Theoretically, therefore this is supposed to give the fastest performance. However, a major
flaw in that package is that it works only at a file/directory level, not at a package level.
Therefore, for structs which are spread across multiple files, it becomes near to impossible
to chase down all transitive dependencies to generate the code. Even if that's done, it fails
on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611)
Therefore, we are left with 2 choices. Both of them use the same underlying algorithm.
But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples
comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance.
msgpack/v5 came out to be the fastest by a small margin.
benchstat master.txt ugorji.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9)
LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10)
LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8)
LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10)
11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10)
LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8)
LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9)
LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10)
In general, we can see that the time to marshal/unmarshal pays off as the size of the struct
increases.
We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory.
Since we are interested in fastest speed, we choose msgpack/v5.
As a future optimization, we can use a mix of msgpack and msgp for hot structs.
To do that, we would need to shuffle around some code so that for the hot struct,
all its dependencies are in the same file.
Let's use this in production for some time, watch grafana graphs for the hottest caches
and come back to optimizing this more once we have more data.
Side note: we have to do with micro-benchmarks for the time being, because all the caches
aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests
and do comparisons.
* Bring back missing import
* Fix tests
