* MM-10856: deduplicate posts with the same pending post id
Leverage a fixed size cache with a window of 30 seconds to deduplicate posts received by a single app server. Clients that duplicate the same pending post id will see one request potentially delayed until the first finishes, after which the same payload should be returned by both. Duplicate posts outside the 30 second window will not be de-duplicated.
Note that the cache is not synchronized between app servers. In an HA cluster consisting of more than one app server, sticky load balancing (e.g. hashing by IP or Session ID) is required to route the users to the same app instance for multiple requests.
Other options considered for this feature:
* adding a column to the `Posts` table: rejected as being too heavyweight
* maintaining a `PendingPostIds` table: similarly rejected for the database impact
* using the pending post id as the post id and relying on the unique constraints on the Post table: rejected for being difficult to show that it's safe to use a client-provided value as the row identifier
* utils/lru: simplify to ttl internally and for new methods
* move seenPendingPostIdsCache to App.Server
* just fail concurrent post requests, vs. trying to wait
* add debug log when create post is deduplicated
* refactor GetDirectChannel and CreateDirectChannel in one function
* remove CreateDirectChannel plugin api and update GetDirectChannel and GetGroupChannel plugin api
* update tests
* MM-11434 Only call PreparePostForClient once when creating a post
* Have PreparePostForClient provide new metadata when a post already has it and update tests
* guard app plugins with mutex
Shutting down the app could race with a goroutine that uses the plugins environment, since we shut down the plugins first before cleaning up goroutines.
* fix go vet issues
* Add interactive dialogs
* Fix unit test
* Updates per feedback
* Fix typo
* Updates per feedback, add icon_url and error returns
* Updates per feedback
* Update per feedback
* Remove unused error return value from PreparePostForClient
* Remove unused error return value from PreparePostListForClient
* MM-11434 Parallelize PreparePostListForClient
* MM-11434 Skip looking reactions and files on post whenever possible
* Add note about the use of deprecated fields
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
* Move OpenGraph code into its own file
* Move OpenGraph image proxying to app layer
* Move test file code out of api4 package
* MM-11272 Add OpenGraph and image dimension metadata to posts
* MM-11272 Added app.PreparePostForClient
* MM-11272 Added app.PreparePostListForClient
* MM-11272 Added EmojiStore.GetMultipleByName
* MM-11272 Added emojis to PreparePostForClient
* MM-11272 Added unit tests for getting reaction counts
* MM-11272 Added unit tests for TestPreparePostForClient
* MM-11272 Added emojis from reactions to Post.Emojis
* MM-11272 Always update post.UpdateAt when reactions change to bust cache
* Fixed merge conflicts
* Moved post metadata-related code into its own file
* Update store mocks
* Fixed typo
* Add missing license headers
* Updated post metadata tests when custom emojis are disabled
* Fix unreliable unit tests
* Fix inconsistent casing in SQL statements
* Fix blank line
* Invalidate store cache after making changes
* Clear post cache synchronously with reactions
Previously, mattermost-server would always request with the default
user-agent of Go's net/http package that is `Go-http-client/1.1` or
something similar.
This has several disadvantages, one is that the default user-agent
made it pretty hard to distinguish mattermost requests from other
service requests in a network log for example.
Now a user-agent of the form `mattermost-<current-version>` is set in
the client.
- [x] Added or updated unit tests (required for all new features)
* MM-7188: Cleaning push notification on every read, not only on channel switch
* Removed unnecesary goroutine
* Fixing tests
* Applying suggestion from PR
* Allow connetions to /plugins for interactive message buttons.
* Adding siteurl to exclusions for AllowedUntrustedInternalConnections
* Adding subpath support for allowing interactive message buttons plugin connections.
* initial implementation of after, before, on search flags allowing to restrict the search to a specific day or a date range
* missed setting beforeDate in SearchParams in one place
* fixed condition when only flags are used for search without any plain terms
* changed date format used for after/before/on flags to be in ISO8601 format as suggested in PR comments, added a helper function to pad month and day with zeroes allowing the user user either format, with or without leading zeroes
* corrected expected compare to date setting for the TestParseDateFilterToTimeISO8601 test
* fixed a bug for the scenario when you only have the date flags without any terms, added a couple of tests for that scenario
* updated the date filter logic to use parameters to construct the query instead of simply appending strings together, as suggested in the pull request comments
* added search unit test using date flags
* added a helper function to create a test post with a createat date manually set, updated the test for search using date flags to create test posts with different createat dates to be able to better test the functionality
* MM-11817 Add support for after/before/on search flags with Elasticsearch
* add support to search posts to perform the search in context of the client's timezone when filtering by createat date using on: after: before: flags
* updated tests to match the new signature
* MM-11065: Allow to search and get archived channels from the API
* Fixing more tests
* Add some unit tests
* Add includeDeleted parameter to session permissions check function
* More test fixing
* Adding archive channels list in channels search
* Add restriction for archived channel edition
* Reverting permissions checks modification
* Changed the query parameter to include_deleted
* Enable search archive channels as true by default
* Adding tests for verify search on deleted channels
* Allowing to override archive channels during the imports
* Fixed test
* Search in archive channels from the API must be explicitly requested
* Removing includeDeleted parameter from GetChannelByName and GetChannelByNameForTeam
* Back to ViewArchivedChannels config
* Fixing tests
* Reverting GetChannelByName parameter
* Add include deleted parameter on GetChannel functions in plugins api
* Fixing tests
* Adding blank request context to plugin hooks for future use.
* Rename RequestContext to Context
* Adding context to ServeHTTP and ExecuteCommand
* Fixing import cycle in test.
* MM-6992 Added highlighting to elasticsearch results
* Added a unique type for post search matches
* Fixed Elasticsearch matches not being sent through API
* Implementing structured logging
* Changes to en.json to allow refactor to run.
* Fixing global logger
* Structured logger initalization.
* Add caller.
* Do some log redirection.
* Auto refactor
* Cleaning up l4g reference and removing dependancy.
* Removing junk.
* Copyright headers.
* Fixing tests
* Revert "Changes to en.json to allow refactor to run."
This reverts commit fd8249e99b.
* Fixing some auto refactor strangeness and typo.
* Making keys more human readable.
* MM-9661: rename POST_MESSAGE_MAX_RUNES to \0_v1
* MM-9661: s/4000/POST_MESSAGE_MAX_RUNES_V1/ in tests
* MM-9661: introduce POST_MESSAGE_MAX_RUNES_V2
* MM-9661: migrate Postgres Posts.Message column to TEXT from VARCHAR(4000)
This is safe to do in a production instance since the underyling type is
not changing. We explicitly don't do this automatically for MySQL, but
also don't need to since the ORM would have already created a TEXT column
for MySQL in that case.
* MM-9661: emit MaxPostSize in client config
This value remains unconfigurable at this time, but exposes the current
limit to the client. The limit remains at 4k in this commit.
* MM-9661: introduce and use SqlPostStore.GetMaxPostSize
Enforce a byte limitation in the database, and use 1/4 of that value as
the rune count limitation (assuming a worst case UTF-8 representation).
* move maxPostSizeCached, lastPostsCache and lastPostTimeCache out of the global context and onto the SqlPostStore
* address feedback from code review:
* ensure sqlstore unit tests are actually being run
* move global caches into SqlPostStore
* leverage sync.Once to address a race condition
* modify upgrade semantics to match new db semantics
gorp's behaviour on creating columns with a maximum length on Postgres
differs from MySQL:
* Postgres
* gorp uses TEXT for string columns without a maximum length
* gorp uses VARCHAR(N) for string columns with a maximum length of N
* MySQL
* gorp uses TEXT for string columns with a maximum length >= 256
* gorp uses VARCHAR(N) for string columns with a maximum length of N
* gorp defaults to a maximum length of 255, implying VARCHAR(255)
So the Message column has been TEXT on MySQL but VARCHAR(4000) on
Postgres. With the new, longer limits of 65535, and without changes to
gorp, the expected behaviour is TEXT on MySQL and VARCHAR(65535) on
Postgres. This commit makes the upgrade semantics match the new database
semantics.
Ideally, we'd revisit the gorp behaviour at a later time.
* allow TestMaxPostSize test cases to actually run in parallel
* default maxPostSizeCached to POST_MESSAGE_MAX_RUNES_V1 in case the once initializer panics
* fix casting error
* MM-9661: skip the schema migration for Postgres
It turns out resizing VARCHAR requires a rewrite in some versions of
Postgres, but migrating VARCHAR to TEXT does not. Given the increasing
complexity, let's defer the migration to the enduser instead.
