What is the problem?
Previously, this was the query used to move change messages into another
channel.
```
INSERT INTO chat_messages(
chat_channel_id, user_id, last_editor_id, message, cooked, cooked_version, created_at, updated_at
)
SELECT :destination_channel_id,
user_id,
last_editor_id,
message,
cooked,
cooked_version,
CLOCK_TIMESTAMP(),
CLOCK_TIMESTAMP()
FROM chat_messages
WHERE id IN (:message_ids)
RETURNING id
```
The problem is that this incorrectly assumes that the insertion will be based on the order of `message_ids`. However, that
is not the case as PostgreSQL provides no such guarantee. Instead we need to explicitly order the messages to ensure
the right order of insertion.
This problem was discovered by a flaky test which exposed the non-guarantee order of insertion.
A category's slug can be encoded when
`SiteSetting.slug_generation_method` has been set to "encoded". As a
result, we have to support non ASCII characters as well.
This commit adds support for excluding categories when using the
`category:` filter with the `-` prefix. For example,
`-category:category-slug` will exclude all topics that belong to the
category with slug "category-slug" and all of its sub-categories.
To only exclude a particular category and not all of its sub-categories,
the `-` prefix can be used with the `=` prefix. For example,
`-=category:category-slug` will only exclude topics that belong to the
category with slug "category-slug". Topics in the sub-categories of
"category-slug" will still be included.
What is the problem?
The TopicTrackingState is a service on the client side that is used to store
state of topics which is new or has unread posts for a given user. The state
is updated via various means and the one in concern here is whenever we load
a new topic list from the server. When a topic list is loaded from the server,
we sync this new topic list with the states in TopicTrackingState. There is also
a hard limit on the number of states that is stored by TopicTrackingState for
performance reasons and the limit is currently set to 4000. It was noticed that
once this limit has been reached, syncing a topic list with TopicTrackingState can
result in the registered state change callbacks to be called unnecessarily. This
is because during `TopicTrackingState#sync` we call `TopicTrackingState#removeTopic`
if the topic in question is neither new or unread to a user. However, `TopicTrackingState#removeTopic`
would call `TopicTrackingState#_afterStateChange` even if nothing was removed.
What is the fix?
This commit fixes the problem by checking that `TopicTrackingState#_afterStateChange` is only
called in `TopicTrackingState#removeTopic` when a topic is actually removed.
See e323628d8a for more details.
This commit speeds up the tests by roughly 10 seconds locally where the
default wait time is 2 seconds. On CI, this speeds up the tests by 20
seconds where the default wait time is 4 seconds.
What is the problem?
We are relying on RSpec custom matchers in system tests by defining
predicates in page objects. The problem is that this can result in a
system test unnecessarily waiting up till the full duration of
Capybara's default wait time when the RSpec custom matcher is used with
`not_to`. Considering this topic page object where we have a `has_post?`
predicate defined.
```
class Topic < PageObject
def has_post?
has_css?('something')
end
end
```
The assertion `expect(Topic.new).not_to have_post` will end up waiting
the full Capybara's default wait time since the RSpec custom matcher is
calling Capybara's `has_css?` method which will wait until the selector
appear. If the selector has already disappeared by the time the
assertion is called, we end up waiting for something that will never
exists.
This commit fixes such cases by introducing new predicates that uses
the `has_no_*` versions of Capybara's node matchers.
For future reference, `to have_css` and `not_to have_css` is safe to sue
because the RSpec matcher defined by Capbyara is smart enough to call
`has_css?` or `has_no_css?` based on the expectation of the assertion.
Specifying more than two tag names when using the `tag:` filter was not
working because of a bug in the code where only the first two value in
the `tag:` filter was being selected.
SearchIndexer is only automatically disabled in `before_all` and `before` blocks which means at the start
of test runs. Enabling the SearchIndexer in one `fab!` block will affect
all other `fab!` blocks which is not ideal as we may be indexing stuff
for search when we don't need to.
What is the problem?
The system tests incorrectly assumes that the discobot user which is
seeded by a core plugin will always be present. This is not true as the
discobot user will only be seeded when the test databases are migrated
with plugins enabled. If we migrate test databases without plugins being
enabled, the core system tests should still pass.
This commit fixes a bug on subfolder setups where the user messages
inbox dropdown will always be blank. This is because we were comparing
URLs using values from `router.currentURL` and `router.urlFor` where
`router.currentURL` does not include `router.rootURL` while
`router.urlFor` does.
`TopicQuery#latest_results` which was being used by
`TopicQuery#list_filter` defaults to ordering by `Topic#bumped_at` in
descending order and that was taking precedent over the order scopes
being applied by `TopicsFilter`.
This allows multiple ordering to be specified by using a comma seperated string.
For example, `order:created,views` would order the topics by
`Topic#created_at` and then `Topic#views.
This commit adds support for the following ordering filters:
1. `order:activity` which orders the topics by `Topic#bumped_at` in descending order
2. `order:activity-asc` which orders the topics by `Topic#bumped_at` in ascending order
3. `order:latest-post` which orders the topics by `Topic#last_posted_at` in descending order
4. `order:latest-post-asc` which orders the topics by `Topic#last_posted_at` in ascending order
5. `order:created` which orders the topics by `Topic#created_at` in descending order
6. `order:created-asc` which orders the topics by `Topic#created_at` in ascending order
7. `order:views` which orders the topics by `Topic#views` in descending order
8. `order:views-asc` which orders the topics by `Topic#views` in ascending order
9. `order:likes` which orders the topics by `Topic#likes` in descending order
10. `order:likes-asc` which orders the topics by `Topic#likes` in ascending order
11. `order:likes-op` which orders the topics by `Post#like_count` of the first post in the topic in descending order
12. `order:likes-op-asc` which orders the topics by `Post#like_count` of the first post in the topic in ascending order
13. `order:posters` which orders the topics by `Topic#participant_count` in descending order
14. `order:posters-asc` which orders the topics by `Topic#participant_count` in ascending order
15. `order:category` which orders the topics by `Category#name` of the topic's category in descending order
16. `order:category-asc` which orders the topics by `Category#name` of the topic's category in ascending order
Multiple order filters can be composed together and the order of ordering is applied based on the position of the filter
in the query string. For example, `order:views order:created` will order the topics by `Topic#views` in descending order
and then order the topics by `Topics#created_at` in descending order.
This commit adds support for the following date filters:
1. `activity-before:<YYYY-MM-DD>` which filters for topics that have been bumped at or before given date
2. `activity-after:<YYYY-MM-DD>` which filters for topics that have been bumped at or after given date
3. `created-before:<YYYY-MM-DD>` which filters for topics that have been created at or before given date
4. `created-after:<YYYY-MM-DD>` which filters for topics that have been created at or after given date
5. `latest-post-before:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or before given date
6. `latest-post-after:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or after given date
If the filter has an invalid value, i.e string that cannot be converted
into a proper date in the `YYYY-MM-DD` format, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
This reverts commit d32709a74f.
It is printing deprecation message due to Rails usage of the PG gem. We
need to wait for a new release of Rails to be cut first.
* DEV: Support `likes-(min:max):<count>` on `/filter` route
This commit adds support for the following filters:
1. `likes-min`
2. `likes-max`
3. `views-min`
4. `views-max`
5. `likes-op-min`
6. `likes-op-max`
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
This commit adds support for the `posters-min:<count>` and
`posters-max:<count>` filters for the topics filtering query language.
`posters-min:1` will filter for topics with at least a one poster while
`posters-max:3` will filter for topics with a maximum of 3 posters.
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
This commit adds support for the `posts-min:<count>` and
`posts-max:<count>` filters for the topics filtering query language.
`posts-min:1` will filter for topics with at least a one post while
`posts-max:3` will filter foor topics with a maximum of 3 posts.
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
Why this change?
Previously `TopicsFilter` was designed in such a way that we act on a
filter sequentially based on the order it was matched. However, this
made it hard to support filters composition where a similar filter may
be present further in the query string. Because of this limitation, I
previously introduced a private API `TopicsFilter.register_scope` which
allows us to act on a filter only after the entire query string has been
scanned. However, I felt that it made the code complicated and hard to
reason about.
In thie commit, I've changed it such that we scan through the entire
query string and group the values of each filter together. This allows
us to act on the values of a given filter in one go which I find easier
to reason about. This also opens up the possibility for us to ignore
certain filters when it has been specified multiple times.
This commit adds support for the `created-by:<username>` query filter
which will return topics created by the specified user. Multiple
usernames can be specified by comma seperating the usernames like so:
`created-by:username1,username2`. This will filter for topics created by
either of the specified users. Multiple `created-by:<username>` can also
be composed together. `created-by:username1 created-by:username2` is
equivalent to `created-by:username1,username2`.
This commit adds support for the `in:<topic notification level>` query
filter. As an example, `in:tracking` will filter for topics that the
user is watching. Filtering for multiple topic notification levels can
be done by comma separating the topic notification level keys. For
example, `in:muted,tracking` or `in:muted,tracking,watching`.
Alternatively, the user can also compose multiple filters with `in:muted
in:tracking` which translates to the same behaviour as
`in:muted,tracking`.
This commit adds support for the `in:pinned` filter to the topics filtering
query language. When the filter is present, it will filter for topics
where `Topic#pinned_until` is greater than `Topic#pinned_at`.
Before this commit, composing multiple category filters with a query such as category:category1 and category:category2 would not return any results. This is because we were filtering for topics that belonged to both category1 and category2, which is impossible since a topic can only belong to a single category.
With this commit, specifying a query like category:category1 category:category2 will now translate to filtering for topics that belong to either the category1 or category2 category.
This commit adds support for filtering for topics in specific
subcategories via the categories filter query language.
For example: `category:documentation:admins` will filter for topics and
subcategory topics in
the category with slug "admins" whose parent category has the slug
"documentation".
The `=` prefix can also be used such that
`=category:documentation:admins` will exclude subcategory topics of the
category with slug "admins" whose parent category has the slug
"documentation".
On the `/filter` route, the categories filtering query language is now
supported in the input per the example provided below:
```
category:bug => topics in the bug category AND all subcategories
=category:bug => topics in the bug category excluding subcategories
category:bug,feature => allow for categories either in bug or feature
=category:bug,feature => allow for exact categories match excluding sub cats
categories: => alias for category
```
Currently composing multiple category filters is not supported as we
have yet to determine what behaviour it should result in. For example,
`category:bug category:feature` would now return topics that are in both
the `bug` and `feature` category but it is not possible for a topic to
belong to two categories.
The following are the changes being introduced in this commit:
1. Instead of mapping the query language to various query params on the
client side, we've decided that the benefits of having a more robust
query language far outweighs the benefits of having a more human readable query params in the URL.
As such, the `/filter` route will just accept a single `q` query param
and the query string will be parsed on the server side.
1. On the `/filter` route, the tags filtering query language is now
supported in the input per the example provided below:
```
tags:bug+feature tagged both bug and feature
tags:bug,feature tagged either bug or feature
-tags:bug+feature excluding topics tagged bug and feature
-tags:bug,feature excluding topics tagged bug or feature
```
The `tags` filter can also be specified multiple
times in the query string like so `tags:bug tags:feature` which will
filter topics that contain both the `bug` tag and `feature` tag. More
complex query like `tags:bug+feature -tags:experimental` will also work.
This change sets the ground work for allowing us to filter topics list
by tags in the following ways:
1. Filter for topics that matches all tags in a given set of tags
2. Filter for topics that matches any tags in a given set of tags
3. Exclude topics that matches all tags in a given set of tags
4. Exclude topics that matches any tags in a given set of tags
Instead of being tied to the old implementation and constraints, a
dedicated route and controller for the `discovery.filter` app route will
allow us to iterate on changes much faster.
Change styling of filter input & remove button.
This follows the same pattern of design we use for search. In the search dropdown we do not have a button to search. We rely on pressing enter. I've also provided an example of Github's PR filter UI at the bottom of this comment.
We also do not have buttons like this on any other topic-list header. On tag and category dropdowns, we also rely on pressing enter to filter the topic list by chosen categories & tags.
Co-authored-by: Jordan Vidrine <jordan@jordanvidrine.com>
Why is this change required?
Prior to this change, we would list all group messages that a user
has access to in the user menu messages notifications panel dropdown.
However, this did not respect the topic's notification level setting and
group messages which the user has set to 'normal' notification level were
being displayed
What does this commit do?
With this commit, we no longer display all group messages that a user
has access to. Instead, we only display group messages that a user is
watching in the user menu messages notifications panel dropdown.
Internal Ref: /t/94392
What is the problem?
When constructing the "Messages" section in Sidebar, we call
startTracking() on the pm-topic-tracking-state service in order to
get the counts for new/unread for the private message inboxes for each
user. However, this is unnecessary because the inboxes are in a
collapsed state by default in the sidebar and are only expanded when the
current route correspond to the inbox's route. Therefore, we can avoid
calling startTracking() on the pm-topic-tracking-state service until
an inbox's route is loaded. This allows us to cut out one extra request
to the server on page load and defer it until it is necessarily.
What is the problem?
We have a hidden site setting `show_category_definitions_in_topic_lists`
which is set to false by default. What this means is that category
definition topics are not shown in the topic list by default. Only the
category definition topic for the category being viewed will be shown.
However, we have a bug where we would show that a category has new
topics when a new child category along with its category definition
topic is created even though the topic list does not list the child
category's category definition topic.
What is the fix here?
This commit fixes the problem by shipping down an additional
`is_category_topic` attribute in `TopicTrackingStateItemSerializer` when
the `show_category_definitions_in_topic_lists` site setting has been set
to false. With the new attribute, we can then exclude counting child
categories' category definition topics when counting new and unread
counts for a category.
In order to use the ruby-lsp vscode extension, the ruby_lsp gem needs to
be added to the project's Gemfile. That may soon change with
https://github.com/Shopify/vscode-ruby-lsp/pull/419 but this will do for
now.
Not eager loading was resulting in the N+1 queries problem when
serializing with the `CurrentUserSerializer` as
`CurrentUserSerializer#sidebar_sections` serializes the sections with
`SidebarSectionSerializer` which fetches all the `SidebarUrl` records
for each `SidebarSection` record.
This commit removes all references to the
`redesigned_user_page_nav_enabled` which was used as a feature flag for
developing the new user profile navigation menu previously.
When the `navigation_menu` site setting has been set to `sidebar` or
`header dropdown`, overriding the site setting via the `navigation_menu`
query params did not work.
Follow-up to c47015b861
Since 359dc1c532, support for the old user
profile navigation menu has been dropped. This commit seeks to remove
code from the client side that still relies on the `currentUser.redesigned_user_page_nav_enabled` prop.
Why this change?
Prior to this change, we placed the identifier for the tag using
CSS classes like `sidebar-section-link-<tag name>`. However, we found that it wasn't obvious
that an identifier for the tag exists since it is first buried in
the CSS classes and second there isn't a way to describe what the
identifier is. Using data attributes, it makes it more obvious that an
identifier exists and what the identifier represents.
Follow-up to 53eb49de72
Why this change?
Prior to this change, we placed the identifier for the category using
CSS classes like `sidebar-section-link-<category slug>`. However, we found that it wasn't obvious
that an identifier for the category exists since it is first buried in
the CSS classes and second there isn't a way to describe what the
identifier is. Using data attributes, it makes it more obvious that an
identifier exists and what the identifier represents.
With the introduction of the sidebar navigation menu, the design team at
Discourse redesigned the user profile navigation to better coexist with
the sidebar.
What does this change do?
This commit the client to override the navigation menu setting
configured by the site temporarily based on the value of the
`navigation_menu` query param. The new query param replaces the old
`enable_sidebar` query param.
Why do we need this change?
The motivation here is to allow theme maintainers to quickly preview
what the site will look like with the various navigation menu site
setting.
1. What is the problem here?
When a user's reviewables count changes, the changes are published via
MessageBus in a background Sidekiq job which means there is a delay before the
client receives the MessageBus message with the updated count. During
the time the reviewables count for a user has been updated and the time
when the client receives the MessageBus message with the updated count,
a user may view the reviewables list in the user menu. When that happens, the number of
reviewables in the list may be out of sync with the count shown.
2. What is the fix?
Going forward, the response for the `ReviewablesController#user_menu_list` action will include the user's reviewables count as
the `reviewables_count` attribute. This is then used by the client side
to update the user's reviewables count to ensure that the reviewables
list and count are kept in sync.
## Why do we need this change?
When loading the ember app, [MessageBus does not start polling immediately](https://github.com/discourse/discourse/blob/f31f0b70f82c43d93220ce6fc0d4f57440452f37/app/assets/javascripts/discourse/app/initializers/message-bus.js#L71-L81) and instead waits for `document.readyState` to be `complete`. What this means is that if there are new messages being created while we have yet to start polling, those messages will not be received by the client.
With sidebar being the default navigation menu, the counts derived from `topic-tracking-state.js` on the client side is prominently displayed on every page. Therefore, we want to ensure that we are not dropping any messages on the channels that `topic-tracking-state.js` subscribes to.
## What does this change do?
This includes the `MessageBus.last_id`s for the MessageBus channels which `topic-tracking-state.js` subscribes to as part of the preloaded data when loading a page. The last ids are then used when we subscribe the MessageBus channels so that messages which are published before MessageBus starts polling will not be missed.
## Review Notes
1. See https://github.com/discourse/message_bus#client-support for documentation about subscribing from a given message id.
Currently `Topic#pm_topic_count` is a count of all personal messages tagged for a given tag. As a result, any user with access to PM tags can poll a sensitive tag to determine if a new personal message has been created using that tag even if the user does not have access to the personal message. We classify this as a minor leak in sensitive information.
With this commit, `Topic#pm_topic_count` is hidden from users by default unless the `display_personal_messages_tag_counts` site setting is enabled.
When the `tags_listed_by_group` site setting is disable, we were seeing
the N+1 queries problem when multiple `CategoryTag` records are listed.
This commit fixes that by ensuring that we are not filtering through the
category `tags` association after the association has been eager loaded.
Currently, `Tag#topic_count` is a count of all regular topics regardless of whether the topic is in a read restricted category or not. As a result, any users can technically poll a sensitive tag to determine if a new topic is created in a category which the user has not excess to. We classify this as a minor leak in sensitive information.
The following changes are introduced in this commit:
1. Introduce `Tag#public_topic_count` which only count topics which have been tagged with a given tag in public categories.
2. Rename `Tag#topic_count` to `Tag#staff_topic_count` which counts the same way as `Tag#topic_count`. In other words, it counts all topics tagged with a given tag regardless of the category the topic is in. The rename is also done so that we indicate that this column contains sensitive information.
3. Change all previous spots which relied on `Topic#topic_count` to rely on `Tag.topic_column_count(guardian)` which will return the right "topic count" column to use based on the current scope.
4. Introduce `SiteSetting.include_secure_categories_in_tag_counts` site setting to allow site administrators to always display the tag topics count using `Tag#staff_topic_count` instead.
Prior to this change, we were parsing `Post#cooked` every time we
serialize a post to extract the usernames of mentioned users in the
post. However, the only reason we have to do this is to support
displaying a user's status beside each mention in a post on the client side when
the `enable_user_status` site setting is enabled. When
`enable_user_status` is disabled, we should avoid having to parse
`Post#cooked` since there is no point in doing so.
When the `tags_listed_by_group` site setting is enabled, we were seeing
the N+1 queries problem when multiple `TagGroup` records are listed.
This commit fixes that by ensuring that we are not filtering through the
`tags` association after the association has been eager loaded.
The `enable_new_notifications_menu` site setting allows sites that have
`navigation_menu` set to `legacy` to use the redesigned notifications
menu before switching to the new sidebar navigation menu.
In Discourse, there are many migration files where we CREATE INDEX CONCURRENTLY which requires us to set disable_ddl_transaction!. Setting disable_ddl_transaction! in a migration file runs the SQL statements outside of a transaction. The implication of this is that there is no ROLLBACK should any of the SQL statements fail.
We have seen lock timeouts occuring when running CREATE INDEX CONCURRENTLY. When that happens, the index would still have been created but marked as invalid by Postgres.
Per the postgres documentation:
> If a problem arises while scanning the table, such as a deadlock or a uniqueness violation in a unique index, the CREATE INDEX command will fail but leave behind an “invalid” index. This index will be ignored for querying purposes because it might be incomplete; however it will still consume update overhead.
> The recommended recovery method in such cases is to drop the index and try again to perform CREATE INDEX CONCURRENTLY . (Another possibility is to rebuild the index with REINDEX INDEX CONCURRENTLY ).
When such scenarios happen, we are supposed to either drop and create the index again or run a REINDEX operation. However, I noticed today that we have not been doing so in Discourse. Instead, we’ve been incorrectly working around the problem by checking for the index existence before creating the index in order to make the migration idempotent. What this potentially mean is that we might have invalid indexes which are lying around in the database which PG will ignore for querying purposes.
This commits adds a migration which queries for all the
invalid indexes in the `public` namespace and reindexes them.
0403cda1d1 introduced a regression where
topics in non read-restricted categories have its TopicTrackingState
MessageBus messages published with the `group_ids: [nil]` option. This
essentially means that no one would be able to view the message.
When a topic belongs to category that is read restricted but permission
has not been granted to any groups, publishing ceratin topic tracking state
updates for the topic will result in the `MessageBus::InvalidMessageTarget` error being raised
because we're passing `nil` to `group_ids` which is not support by
MessageBus.
This commit ensures that for said category above, we will publish the
updates to the admin groups.
```
class Jobs::DummyDelayedJob < Jobs::Base
def execute(args = {})
end
end
RSpec.describe "Jobs.run_immediately!" do
before { Jobs.run_immediately! }
it "explodes" do
current_user = Fabricate(:user)
Jobs.enqueue_in(1.seconds, :dummy_delayed_job)
sign_in(current_user)
end
end
```
The test above will fail with the following error if `ActiveRecord::Base.connection_handler.clear_active_connections!` is called before the configured Capybara server checks out a connection from the connection pool.
```
ActiveRecord::ActiveRecordError:
Cannot expire connection, it is owned by a different thread: #<Thread:0x00007f437391df58@puma srv tp 001 /home/tgxworld/.asdf/installs/ruby/3.1.3/lib/ruby/gems/3.1.0/gems/puma-6.0.2/lib/puma/thread_pool.rb:106 sleep_forever>. Current thread: #<Thread:0x00007f437d6cfc60 run>.
```
We're not exactly sure if this is an ActiveRecord bug or not but we've
invested too much time into investigating this problem. Fundamentally,
we also no longer understand why `ActiveRecord::Base.connection_handler.clear_active_connections!` is being called in an ensure block
within `Jobs::Base#perform` which was added in
ceddb6e0da 10 years ago. This
commit moves the logic for running jobs immediately out of the
`Jobs::Base#perform` method into another `Jobs::Base#perform_immediately` method such that
`ActiveRecord::Base.connection_handler.clear_active_connections!` is not
called. This change will only impact the test environment.
Alan Guo Xiang Tan
