* Implement LDAP Certificate
* add diagnostics and translations
* update from code review
* pass pointer to update pict function
* pass object to first function
* remove debug log messages
* update test to add localmode test
* update lint errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
MM-27744 disable Zap for unit tests.
Zap has no concept of shutdown or close. Zap is only shutdown when the app exits. Not a problem for console logging, but when creating a new Zap logger that outputs to files on every unit test, that leaves no easy way to clean up until process exit. Depending on what else is running this can exhaust all file handles and cause unit tests to fail.
Zap is now disabled unit tests and uses Logr instead, regardless of config settings. `make test-server` peak file handle usage dropped from ~5K to less than 100.
* Moving diagnostics into a service
* Fixing golint checks
* Fixing tests
* Renaming from diagnostics to telemetry
* Adding missing files
* Initializing telemetry earlier in the server startup
* Fixing tests
* Adding a log for the telemetryID initialization error
* Addressing PR review comments
* Fixing merge problem
* Removing some extra Diagnostics mentions
* Making tests pass
* Adding Upgrade to Enterprise version feature
* Addressing PR review comments, and adding some minor improvements
* Add tests file
* Addressing PR comments
* fix linter checks
* Storing and exposing the upgraded from TE info
* Fix showing errors on mac
* A more appropiate status code for not-supported upgrade
* Fixing tests
* Handling permissions errors
* More server logging around upgrade failures
* Apply text changes suggested from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Address PR review comments
* Only allow to restart the system after an upgrade
* Verify file signature before upgrade
* Adding limit to the downloaded file
* Simplifying the upgrade binary process with backup in memory
* Fixing backup/restore mechanism for the binary file
* Improve file permissions handling
* Askin the permissions for the right place (the parent directory)
* Fixing tests
* Addressing PR review comments
* Fix license headers
* Fixing retry layer
* Making it work on windows builds
* Adding license header
* Fixing 2 tests
* Fixing tests that need UpgradeFromTE System key mock
* Extracting i18n translation
* Apply suggestions from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Improving how the errors are written
* Fixing another error text
* Removing unneeded translation
* Fixing upgrade status strings
* Update i18n/en.json
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Fixing tests
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Implemented AmazonS3PathPrefix
* Remove unecessary method
* fix for test
* fix for test which are failing
* fix for test which are failing
* fix for test
Co-authored-by: Dusan Panic <dusan@salestrekker.com>
* [MM-26397] Take query size and order into account for Bleve
* Add a test to check post search pagination
* Add tests for checking limit when searching users
* Make pagination an independent test to discriminate DB engines
* Two missing methods to add in the channel layer
* Added delete user/channel posts methods
- Created in both search engines but only implemented in ES
- Add those methods in the search layer
- Included the PermanentDeleteByUser/Channel methods
* Two new delete documents are included in the bleve code with this
change:
- DeleteChannelPosts
- DeleteUserPosts
These two new functions delete post documents from the index-based
in the filed value provided
* MM-25710: Use an efficient cache serialization algorithm
We investigate 3 packages for selecting a suitable replacement
for gob encoding. The algorithm chosen was msgpack which gives
a decent boost over the standard gob encoding.
Any external schema dependent algorithms like protobuf, flatbuffers, avro,
capn'proto were not considered as that would entail converting the model structs
into separate schema objects and then code generating the Go structs.
It could be done theoretically at a later stage specifically for structs
which are in the hot path. This is a general solution for now.
The packages considered were:
- github.com/tinylib/msgp
- github.com/ugorji/go/codec
- github.com/vmihailenco/msgpack/v5
msgp uses code generation to generate encoding/decoding code without the reflection overhead.
Theoretically, therefore this is supposed to give the fastest performance. However, a major
flaw in that package is that it works only at a file/directory level, not at a package level.
Therefore, for structs which are spread across multiple files, it becomes near to impossible
to chase down all transitive dependencies to generate the code. Even if that's done, it fails
on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611)
Therefore, we are left with 2 choices. Both of them use the same underlying algorithm.
But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples
comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance.
msgpack/v5 came out to be the fastest by a small margin.
benchstat master.txt ugorji.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9)
LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10)
LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8)
LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10)
11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10)
LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8)
LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9)
LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10)
In general, we can see that the time to marshal/unmarshal pays off as the size of the struct
increases.
We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory.
Since we are interested in fastest speed, we choose msgpack/v5.
As a future optimization, we can use a mix of msgpack and msgp for hot structs.
To do that, we would need to shuffle around some code so that for the hot struct,
all its dependencies are in the same file.
Let's use this in production for some time, watch grafana graphs for the hottest caches
and come back to optimizing this more once we have more data.
Side note: we have to do with micro-benchmarks for the time being, because all the caches
aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests
and do comparisons.
* Bring back missing import
* Fix tests
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* provide and implement a new cache provider
* make interface more flexible
* let Connect and Close return error
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Go 1.14 introduces usage of alternate go.mod files to track
other dependencies which are not related to the main app.
We use this to track all tool dependencies so that everybody uses
the same version of all tools, including CI.
This will prevent version conflicts due to everybody using different
versions of the tools.
And it will not try to upgrade the tool version, every single time,
one runs the tool command.
While here, we also re-generate some filestore mocks which weren't up to date.
Fixes#13088
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-25006: Fix occurences of unclosed http bodies
And while here, we also use ioutil.Discard to read the remaining
body instead of reading with ioutil.ReadAll which allocates a separate
byte buffer.
* Fix mistake
* Address review comments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* [MM-20979] Add first implementation of the Bleve search engine
* Fix i18n
* Migrate searchengine utils tests
* Fix linter
* Don't add allTermsQ if both termQueries and notTermQueries are empty
* Fix test that should work if user is system admin
* Modify naming according to review comments
* Abstract getIndexDir function
* Extracting bleve engine name as a constant
* Merge both Indexer interfaces into one
* Add worker stopped message
* Allow worker to be started/stopped with config change
* Use constants for index names
* Modify test order
* Fix linter
* Trying to unlock the CI
* MM-24395: Set response header to be attachment for SVG images.
We check the file extension and appropriately set the response header.
SVG images without a file extension aren't proxied at all. So there's no
problem with that.
* Add test
* Improve test a bit
* Capture content type
* incorporate comments
* Set attachment mode in case of error too
* MM-23666: rewrite TestHTTPClient
The existing `TestHTTPClient` actually tries to reach google.com, making it brittle to network hiccups (as unlikely as that might be). Rewrite the test to rely on a local server instead.
I found the existing implementation of `TestHTTPClient` extremely hard to understand, so I've rewritten it more explicitly, expanded coverage somewhat. In doing so, I found myself somewhat surprised by the current behaviour. If `allowHost` returns `true`, we ignore `allowIP` altogether. If `allowHost` returns `false`, we check with `allowIP`. There's a lack of symmetry here that the current filter names don't make clear. Perhaps `allowHost` should be renamed `skipIPCheckForHost`?
Fixes: https://mattermost.atlassian.net/browse/MM-23666
* more compact assertions
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
* Search tests
* Fix tests
* Fix mod
* Fixing searchEngine for test helpers with store mocks
* Remove loglines
* Fix i18n strings
* Migrate search posts tests
* Fix linter
* Do not run search tests if -short flag is enabled
* Migrate back store tests that didn't belong to the searchlayer
* Fix scopelint issues
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* Improving mfa tests coverage and format
* Adding assertions for the specific error types
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
This reverts commit 0361e8b97e.
The new cache ultimately did not give much benefit because the hit rate
was pretty low and it did not lead to any reduction in store timings.
So the increase in cluster messages was not worth it.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
