* Two missing methods to add in the channel layer
* Added delete user/channel posts methods
- Created in both search engines but only implemented in ES
- Add those methods in the search layer
- Included the PermanentDeleteByUser/Channel methods
* Two new delete documents are included in the bleve code with this
change:
- DeleteChannelPosts
- DeleteUserPosts
These two new functions delete post documents from the index-based
in the filed value provided
* MM-25710: Use an efficient cache serialization algorithm
We investigate 3 packages for selecting a suitable replacement
for gob encoding. The algorithm chosen was msgpack which gives
a decent boost over the standard gob encoding.
Any external schema dependent algorithms like protobuf, flatbuffers, avro,
capn'proto were not considered as that would entail converting the model structs
into separate schema objects and then code generating the Go structs.
It could be done theoretically at a later stage specifically for structs
which are in the hot path. This is a general solution for now.
The packages considered were:
- github.com/tinylib/msgp
- github.com/ugorji/go/codec
- github.com/vmihailenco/msgpack/v5
msgp uses code generation to generate encoding/decoding code without the reflection overhead.
Theoretically, therefore this is supposed to give the fastest performance. However, a major
flaw in that package is that it works only at a file/directory level, not at a package level.
Therefore, for structs which are spread across multiple files, it becomes near to impossible
to chase down all transitive dependencies to generate the code. Even if that's done, it fails
on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611)
Therefore, we are left with 2 choices. Both of them use the same underlying algorithm.
But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples
comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance.
msgpack/v5 came out to be the fastest by a small margin.
benchstat master.txt ugorji.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9)
LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10)
LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8)
LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10)
11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10)
LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8)
LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9)
LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10)
In general, we can see that the time to marshal/unmarshal pays off as the size of the struct
increases.
We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory.
Since we are interested in fastest speed, we choose msgpack/v5.
As a future optimization, we can use a mix of msgpack and msgp for hot structs.
To do that, we would need to shuffle around some code so that for the hot struct,
all its dependencies are in the same file.
Let's use this in production for some time, watch grafana graphs for the hottest caches
and come back to optimizing this more once we have more data.
Side note: we have to do with micro-benchmarks for the time being, because all the caches
aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests
and do comparisons.
* Bring back missing import
* Fix tests
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* provide and implement a new cache provider
* make interface more flexible
* let Connect and Close return error
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Go 1.14 introduces usage of alternate go.mod files to track
other dependencies which are not related to the main app.
We use this to track all tool dependencies so that everybody uses
the same version of all tools, including CI.
This will prevent version conflicts due to everybody using different
versions of the tools.
And it will not try to upgrade the tool version, every single time,
one runs the tool command.
While here, we also re-generate some filestore mocks which weren't up to date.
Fixes#13088
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-25006: Fix occurences of unclosed http bodies
And while here, we also use ioutil.Discard to read the remaining
body instead of reading with ioutil.ReadAll which allocates a separate
byte buffer.
* Fix mistake
* Address review comments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* [MM-20979] Add first implementation of the Bleve search engine
* Fix i18n
* Migrate searchengine utils tests
* Fix linter
* Don't add allTermsQ if both termQueries and notTermQueries are empty
* Fix test that should work if user is system admin
* Modify naming according to review comments
* Abstract getIndexDir function
* Extracting bleve engine name as a constant
* Merge both Indexer interfaces into one
* Add worker stopped message
* Allow worker to be started/stopped with config change
* Use constants for index names
* Modify test order
* Fix linter
* Trying to unlock the CI
* MM-24395: Set response header to be attachment for SVG images.
We check the file extension and appropriately set the response header.
SVG images without a file extension aren't proxied at all. So there's no
problem with that.
* Add test
* Improve test a bit
* Capture content type
* incorporate comments
* Set attachment mode in case of error too
* MM-23666: rewrite TestHTTPClient
The existing `TestHTTPClient` actually tries to reach google.com, making it brittle to network hiccups (as unlikely as that might be). Rewrite the test to rely on a local server instead.
I found the existing implementation of `TestHTTPClient` extremely hard to understand, so I've rewritten it more explicitly, expanded coverage somewhat. In doing so, I found myself somewhat surprised by the current behaviour. If `allowHost` returns `true`, we ignore `allowIP` altogether. If `allowHost` returns `false`, we check with `allowIP`. There's a lack of symmetry here that the current filter names don't make clear. Perhaps `allowHost` should be renamed `skipIPCheckForHost`?
Fixes: https://mattermost.atlassian.net/browse/MM-23666
* more compact assertions
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
* Search tests
* Fix tests
* Fix mod
* Fixing searchEngine for test helpers with store mocks
* Remove loglines
* Fix i18n strings
* Migrate search posts tests
* Fix linter
* Do not run search tests if -short flag is enabled
* Migrate back store tests that didn't belong to the searchlayer
* Fix scopelint issues
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* Improving mfa tests coverage and format
* Adding assertions for the specific error types
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
This reverts commit 0361e8b97e.
The new cache ultimately did not give much benefit because the hit rate
was pretty low and it did not lead to any reduction in store timings.
So the increase in cluster messages was not worth it.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-17149 - Extend config.json for marketplace settings (#11933)
* MM-17149 - Extend config.json for marketplace settings
* Renamed MarketplaceUrl, tracking default marketplace url
* Added EnableMarketplace to the client config
* Revert "Added EnableMarketplace to the client config"
This reverts commit 0f982c4c66.
* MM-17149 - Added EnableMarketplace to the client config (#11958)
* Added EnableMarketplace to the client config
* Moved EnableMarketplace setting out of limited client configuration
* MM-17150, MM-17545, MM-18100 - Implement GET /api/v4/plugins/m… (#11977)
* MM-17150 - Implement GET /api/v4/plugins/marketplace proxying upstream
MM-17545 - Merge locally installed plugins into GET /api/v4/plugins/marketplace
* Replaced MarketplacePluginState with Installed
* Setting InstalledVersion instead of Installed
* marketplace client setting per_page if non zero
* Creating insecure client for marketplace url
* Fixed trailing slash for default marketplace url
* Adding filtering
* Fixed function names
* Renamed Manifest() to GetManifest(), added godoc for BaseMarketplacePlugin
* Handling plugin.ErrNotFound correctly
* Checking err == nil instead when a plugin is installed
* MM-18450 - Local-only plugin search (#12152)
* MM-17846: plugin icons (#12157)
* MM-17846: add support for plugin icons
Extend the model definitions to support plugin icons from the marketplace.
* s/IconURL/IconData
* MM-18475 - Converge on snake_case responses from the marketplace (#12179)
* MM-18520 - MM-Server should forward server version to marketplace server (#12181)
* Renamed request to filter client4.GetMarketplacePlugins
* Renamed request to filter
* Guarding against bad marketplace server response
* replace dockerhost with localhost
* remove uneeded setup-max build step (no more dockerhost)
* changes as recommended by @cpanato
* make clean-docker with docker-compose
* added ports to docker-compose.yml (needed for osx). ignore error for ldapadd (when already exists)
* add clean-old-docker to legacy.mk
* docker-compose stop instead of down for `make stop-docker`
* MM-16272 - Synchronize plugins across cluster (#11611)
* MM-16272 - Synchronize plugins across cluster
* Adding a test
* MM-16272 - Fixed tests
* MM-16272 - PR feedback
* MM-16270 - Plugin Sync (#11615)
* Initial implementation for plugin synch with file store. WIP
* Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory
* Refactored plugin filestore operations out of main install/remove plugin
* Fixing error handling details
* Changes to use structured logging
* More logging fixes
* Wording and comments improvements
* Error handling and control flow improvements
* Changed managed flag check to use os.stat
* Added file store plugin dir and filename consts
* Replaced FileRead to use a the FileReader in PluginSync
* Minor styling and PR feedback changes
* Minor error handling improvements
* Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins
* PR Feedback improvements
* Minor err handling fix
* Removing FileStorePath from PluginEventData (#11644)
* Fix plugin path (#11654)
* tweak path, logging
Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity.
* Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist
* PR Feedback
* Install prepackaged plugins locally only (#11656)
* s/uninstall/remove
* Updated ClusterMessage comment
* Updated PluginSync to test against s3 + local storage
