IntenseWebs/discourse
3
0
Fork 0
mirror of https://github.com/discourse/discourse.git synced 2024-11-25 02:11:08 -06:00
Code Issues Packages Projects Releases Wiki Activity
56,352 Commits 214 Branches 500 Tags 884 MiB
2ffe413b0b
Commit Graph

36 Commits

Author SHA1 Message Date
Natalie Tay
a49a6941c6
DEV: Extend /filter's ability to order (#28242) 2024-08-07 16:37:00 +08:00
David Battersby
43aa47b118
FIX: experimental topics filter should allow tags with underscore (#27994) 
When tags contain an underscore we should allow filtering in the same way, previously due to the regex those with underscores were not being found when filtering.
 2024-07-20 00:58:29 +04:00
Isaac Janzen
b3e0e920ed
DEV: Support adding a custom filter on /filter (#27927) 
# Context

Currently there is no way to add a custom filter to the experimental `/filter` endpoint. While you can implement a custom `status:` there is no way to include the user's input in a custom query. 

# PR

This PR adds the ability to implement a custom filter. eg. `CUSTOM_FILTER:foo`

- Add `add_filter_custom_filter` for extension
- Add specs
 2024-07-17 11:36:38 -05:00
Gabriel Grubba
93c2ae585c
FEATURE: Add tag_group option in /filter (#27427) 
* FEATURE: Add tag_group option in `/filter`

* DEV: Update tag_group_filter in `/filter` to use SQL

* DEV: Add guardian to `/filter` by tag_group

* DEV: remove unused code

* DEV: Update tag_group filter implementation

* DEV: Add tests to tag_group filter
 2024-06-13 14:03:49 -03:00
Alan Guo Xiang Tan
2134ca9031
PERF: Optimise query for excluding topics in certain categories in TopicsFilter (#27027) 
This commit optimises the database query generated by
`TopicsFilter#filter_categories` when the `-category:*` filter is used.
Previously, the method will add the `topics.category_id NOT IN
(<category ids to be excluded>)` filter to the resulting query. However,
we noticed that the performance of the query degrades as the number of
rows in the `topics` table grow and when the number of category ids to be
excluded is large.

Sample of query we ran on a large database in production to demonstrate
the improvement:

Before:

```
SELECT topics.id FROM topics WHERE topics.category_id NOT IN (83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87) ORDER BY topics.id DESC LIMIT 5;

                                                                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.317..30.165 rows=5 loops=1)
   ->  Sort  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.316..30.163 rows=5 loops=1)
         Sort Key: id DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Gather  (cost=1000.10..27795.33 rows=1 width=4) (actual time=0.187..26.132 rows=73478 loops=1)
               Workers Planned: 2
               Workers Launched: 2
               ->  Parallel Seq Scan on topics  (cost=0.10..26795.23 rows=1 width=4) (actual time=0.013..22.252 rows=24493 loops=3)
                     Filter: (category_id <> ALL ('{83,136,149,143,153,165,161,123,155,163,144,134,69,135,158,141,151,160,131,133,89,104,150,147,132,145,108,146,122,100,128,154,95,102,140,139,88,91,87}'::integer[]))
                     Rows Removed by Filter: 77276
 Planning Time: 0.140 ms
 Execution Time: 30.181 ms
```

After:

```
SELECT topics.id FROM topics WHERE NOT EXISTS (
  SELECT 1
  FROM unnest(array[83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87]) AS excluded_categories(category_id)
  WHERE topics.category_id IS NULL OR excluded_categories.category_id = topics.category_id
) ORDER BY topics.id DESC LIMIT 5 ;

                                                                        QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..13.52 rows=5 width=4) (actual time=0.028..0.110 rows=5 loops=1)
   ->  Nested Loop Anti Join  (cost=0.42..179929.62 rows=68715 width=4) (actual time=0.027..0.109 rows=5 loops=1)
         Join Filter: ((topics.category_id IS NULL) OR (excluded_categories.category_id = topics.category_id))
         Rows Removed by Join Filter: 239
         ->  Index Scan Backward using forum_threads_pkey on topics  (cost=0.42..108925.71 rows=305301 width=8) (actual time=0.012..0.062 rows=44 loops=1)
         ->  Function Scan on unnest excluded_categories  (cost=0.00..0.39 rows=39 width=4) (actual time=0.000..0.001 rows=6 loops=44)
 Planning Time: 0.126 ms
 Execution Time: 0.124 ms
(8 rows)
```
 2024-05-15 11:03:07 +08:00
Régis Hanol
f7a1272fa4 DEV: cleanup custom filters to prevent leaks 
Ensures we clean up any custom filters added in the specs to prevent any leaks when running the specs.

Follow up to https://github.com/discourse/discourse/pull/26770#discussion_r1582464760
 2024-04-29 16:11:12 +02:00
Régis Hanol
803c275bd7 DEV: add support for adding custom status filter 
Those can be used in the /filter routes.
 2024-04-26 14:04:03 +02:00
Alan Guo Xiang Tan
6c838c73e5
DEV: Add order:title filter to experimental /filter route (#22293) 
This has been requested multiple times by users so it makes sense for us
to support this as well.

See https://meta.discourse.org/t/sort-display-of-topics-alphabetically/53911
 2023-06-28 06:21:56 +08:00
Alan Guo Xiang Tan
0c8d3f8542
DEV: Fix category: filter only supported alphabets and numbers (#21427) 
A category's slug can be encoded when
`SiteSetting.slug_generation_method` has been set to "encoded". As a
result, we have to support non ASCII characters as well.
 2023-05-09 08:10:08 +08:00
Alan Guo Xiang Tan
7d0ef338e4
DEV: tag: filter on /filter only supported alphabets and numbers (#21405) 
A tag's name can consist of any Unicode characters as well
 2023-05-09 08:02:11 +08:00
Alan Guo Xiang Tan
963bb3406e
DEV: Support excluding categories with the category: filter (#21432) 
This commit adds support for excluding categories when using the
`category:` filter with the `-` prefix. For example,
`-category:category-slug` will exclude all topics that belong to the
category with slug "category-slug" and all of its sub-categories.

To only exclude a particular category and not all of its sub-categories,
the `-` prefix can be used with the `=` prefix. For example,
`-=category:category-slug` will only exclude topics that belong to the
category with slug "category-slug". Topics in the sub-categories of
"category-slug" will still be included.
 2023-05-08 14:04:47 +08:00
Alan Guo Xiang Tan
d4e2f764ad
DEV: Fix tags filter with more than 2 tag names in value not working (#21380) 
Specifying more than two tag names when using the `tag:` filter was not
working because of a bug in the code where only the first two value in
the `tag:` filter was being selected.
 2023-05-04 15:55:31 +08:00
Alan Guo Xiang Tan
b4cf990a51
DEV: Support both tag: as an alias for tags: filter for /filter (#21353) 
We already support `category:` as an alias for `categories` so it makes
sense to support `tag:` as an alias for `tags:`.
 2023-05-03 14:51:04 +08:00
Alan Guo Xiang Tan
691b9fb919
DEV: Support comma seperated value in order filter for /filter route (#21318) 
This allows multiple ordering to be specified by using a comma seperated string.
For example, `order:created,views` would order the topics by
`Topic#created_at` and then `Topic#views.
 2023-05-03 12:39:52 +08:00
Alan Guo Xiang Tan
93f7c24240
DEV: Change created-by topics query filter to created-by:@<username> (#21317) 
We want each username to be prefixed with the `@` symbol.
 2023-05-03 12:39:11 +08:00
Alan Guo Xiang Tan
b06a154bb1
DEV: Use \A and \z instead of ^ and $ in TopicsFilter (#21316) 
`^` and `$` matches per line which is technically not what we want.
 2023-05-01 08:09:08 +08:00
Alan Guo Xiang Tan
6e5e607072
DEV: Support ordering filters on /filter route (#21275) 
This commit adds support for the following ordering filters:

1. `order:activity` which orders the topics by `Topic#bumped_at` in descending order
2. `order:activity-asc` which orders the topics by `Topic#bumped_at` in ascending order
3. `order:latest-post` which orders the topics by `Topic#last_posted_at` in descending order
4. `order:latest-post-asc` which orders the topics by `Topic#last_posted_at` in ascending order
5. `order:created` which orders the topics by `Topic#created_at` in descending order
6. `order:created-asc` which orders the topics by `Topic#created_at` in ascending order
7. `order:views` which orders the topics by `Topic#views` in descending order
8. `order:views-asc` which orders the topics by `Topic#views` in ascending order
9. `order:likes` which orders the topics by `Topic#likes` in descending order
10. `order:likes-asc` which orders the topics by `Topic#likes` in ascending order
11. `order:likes-op` which orders the topics by `Post#like_count` of the first post in the topic in descending order
12. `order:likes-op-asc` which orders the topics by `Post#like_count` of the first post in the topic in ascending order
13. `order:posters` which orders the topics by `Topic#participant_count` in descending order
14. `order:posters-asc` which orders the topics by `Topic#participant_count` in ascending order
15. `order:category` which orders the topics by `Category#name` of the topic's category in descending order
16. `order:category-asc` which orders the topics by `Category#name` of the topic's category in ascending order

Multiple order filters can be composed together and the order of ordering is applied based on the position of the filter
in the query string. For example, `order:views order:created` will order the topics by `Topic#views` in descending order
and then order the topics by `Topics#created_at` in descending order.
 2023-04-27 15:44:58 +08:00
Alan Guo Xiang Tan
141555136a
DEV: Support filtering by date columns on /filter route (#21233) 
This commit adds support for the following date filters:

1. `activity-before:<YYYY-MM-DD>` which filters for topics that have been bumped at or before given date
2. `activity-after:<YYYY-MM-DD>` which filters for topics that have been bumped at or after given date
3. `created-before:<YYYY-MM-DD>` which filters for topics that have been created at or before given date
4. `created-after:<YYYY-MM-DD>` which filters for topics that have been created at or after given date
5. `latest-post-before:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or before given date
6. `latest-post-after:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or after given date

If the filter has an invalid value, i.e string that cannot be converted
into a proper date in the `YYYY-MM-DD` format, the filter will be ignored.

If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
 2023-04-27 15:43:47 +08:00
Alan Guo Xiang Tan
1f0207ba06
DEV: Add support for more filters for /filter route (#21097) 
* DEV: Support `likes-(min:max):<count>` on `/filter` route

This commit adds support for the following filters: 

1. `likes-min` 
2. `likes-max`
3. `views-min`
4. `views-max`
5. `likes-op-min`
6. `likes-op-max`

If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.

If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
 2023-04-14 10:21:04 +08:00
Alan Guo Xiang Tan
782b26d0eb
DEV: Support posters-(min|max):<count> on /filter route (#21095) 
This commit adds support for the `posters-min:<count>` and
`posters-max:<count>` filters for the topics filtering query language.
`posters-min:1` will filter for topics with at least a one poster while
`posters-max:3` will filter for topics with a maximum of 3 posters.

If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.

If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
 2023-04-14 07:48:38 +08:00
Alan Guo Xiang Tan
bc4a9c50f2
DEV: Support posts-min:<count> and posts-max:<count> on /filter (#21090) 
This commit adds support for the `posts-min:<count>` and
`posts-max:<count>` filters for the topics filtering query language.
`posts-min:1` will filter for topics with at least a one post while
`posts-max:3` will filter foor topics with a maximum of 3 posts.

If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.

If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
 2023-04-14 06:05:55 +08:00
Alan Guo Xiang Tan
5b1306cb54
DEV: Refactor TopicsFilter (#21071) 
Why this change?

Previously `TopicsFilter` was designed in such a way that we act on a
filter sequentially based on the order it was matched. However, this
made it hard to support filters composition where a similar filter may
be present further in the query string. Because of this limitation, I
previously introduced a private API `TopicsFilter.register_scope` which
allows us to act on a filter only after the entire query string has been
scanned. However, I felt that it made the code complicated and hard to
reason about.

In thie commit, I've changed it such that we scan through the entire
query string and group the values of each filter together. This allows
us to act on the values of a given filter in one go which I find easier
to reason about. This also opens up the possibility for us to ignore
certain filters when it has been specified multiple times.
 2023-04-13 13:22:11 +08:00
Alan Guo Xiang Tan
a1524b84e2
DEV: Support created-by:<username> filter on /filter route (#21067) 
This commit adds support for the `created-by:<username>` query filter
which will return topics created by the specified user. Multiple
usernames can be specified by comma seperating the usernames like so:
`created-by:username1,username2`. This will filter for topics created by
either of the specified users. Multiple `created-by:<username>` can also
be composed together. `created-by:username1 created-by:username2` is
equivalent to `created-by:username1,username2`.
 2023-04-12 09:25:06 +08:00
Alan Guo Xiang Tan
2809d7ba8e
DEV: Support in:<notification level> filter on /filter route (#21038) 
This commit adds support for the `in:<topic notification level>` query
filter. As an example, `in:tracking` will filter for topics that the
user is watching. Filtering for multiple topic notification levels can
be done by comma separating the topic notification level keys. For
example, `in:muted,tracking` or `in:muted,tracking,watching`.
Alternatively, the user can also compose multiple filters with `in:muted
in:tracking` which translates to the same behaviour as
`in:muted,tracking`.
 2023-04-11 08:48:07 +08:00
Alan Guo Xiang Tan
b2a951e4a5
DEV: Support in:bookmarked filter for the /filter route (#21000) 
This filters the topics list to the topics that the current user has bookmarks in.
 2023-04-06 12:55:28 +08:00
Alan Guo Xiang Tan
ab54a616c1
DEV: Introduce in:pinned filter for experimental /filter route (#20974) 
This commit adds support for the `in:pinned` filter to the topics filtering
query language. When the filter is present, it will filter for topics
where `Topic#pinned_until` is greater than `Topic#pinned_at`.
 2023-04-06 10:13:02 +08:00
Alan Guo Xiang Tan
62696b9ee7
DEV: Properly support composing multiple category filters on /filter (#20953) 
Before this commit, composing multiple category filters with a query such as category:category1 and category:category2 would not return any results. This is because we were filtering for topics that belonged to both category1 and category2, which is impossible since a topic can only belong to a single category.

With this commit, specifying a query like category:category1 category:category2 will now translate to filtering for topics that belong to either the category1 or category2 category.
 2023-04-05 07:16:37 +08:00
Alan Guo Xiang Tan
fd34032db2
DEV: Support filter for topics in specific subcategories on /filter (#20927) 
This commit adds support for filtering for topics in specific
subcategories via the categories filter query language.

For example: `category:documentation:admins` will filter for topics and
subcategory topics in
the category with slug "admins" whose parent category has the slug
"documentation".

The `=` prefix can also be used such that
`=category:documentation:admins` will exclude subcategory topics of the
category with slug "admins" whose parent category has the slug
"documentation".
 2023-04-03 18:36:59 +08:00
Alan Guo Xiang Tan
0162f0ccb0
DEV: Update experimental /filter route with categories support (#20911) 
On the `/filter` route, the categories filtering query language is now
supported in the input per the example provided below:

```
category:bug => topics in the bug category AND all subcategories
=category:bug => topics in the bug category excluding subcategories
category:bug,feature => allow for categories either in bug or feature
=category:bug,feature => allow for exact categories match excluding sub cats
categories: => alias for category
```

Currently composing multiple category filters is not supported as we
have yet to determine what behaviour it should result in. For example,
`category:bug category:feature` would now return topics that are in both
the `bug` and `feature` category but it is not possible for a topic to
belong to two categories.
 2023-03-31 14:32:12 +08:00
Alan Guo Xiang Tan
4e11014693
DEV: Support status:public in topics filtering query language (#20889) 
This commit adds support for the `status:public` filter which only
return topics that belong to public categories.
 2023-03-30 10:57:26 +08:00
Alan Guo Xiang Tan
49e7e639cc
DEV: Update experimental /filter route with tags support (#20874) 
The following are the changes being introduced in this commit:

1. Instead of mapping the query language to various query params on the
client side, we've decided that the benefits of having a more robust
query language far outweighs the benefits of having a more human readable query params in the URL.
As such, the `/filter` route will just accept a single `q` query param
and the query string will be parsed on the server side.

1. On the `/filter` route, the tags filtering query language is now
   supported in the input per the example provided below:

   ```
   tags:bug+feature tagged both bug and feature
   tags:bug,feature tagged either bug or feature
   -tags:bug+feature excluding topics tagged bug and feature
   -tags:bug,feature excluding topics tagged bug or feature
   ```

   The `tags` filter can also be specified multiple
times in the query string like so `tags:bug tags:feature` which will
filter topics that contain both the `bug` tag and `feature` tag. More
complex query like `tags:bug+feature -tags:experimental` will also work.
 2023-03-30 09:00:42 +08:00
Alan Guo Xiang Tan
4624cca00f
DEV: Fix TopcisFilter#filter_tags not working for a single tag (#20840) 
Follow-up to dd88fdeabc
 2023-03-27 16:58:40 +08:00
Alan Guo Xiang Tan
dd88fdeabc
DEV: Introduce TopicsFilter#filter_tags method (#20839) 
This change sets the ground work for allowing us to filter topics list
by tags in the following ways:

1. Filter for topics that matches all tags in a given set of tags
2. Filter for topics that matches any tags in a given set of tags
3. Exclude topics that matches all tags in a given set of tags
4. Exclude topics that matches any tags in a given set of tags
 2023-03-27 14:16:53 +08:00
Alan Guo Xiang Tan
56fbdde0e5
FIX: Broken ?status=(listed|unlisted) query param support (#20834) 
In 66c5054, the support for filtering a
topics list based on the visible attribute of a topic via the status query param
was accidentally removed.
 2023-03-27 07:30:19 +08:00
Alan Guo Xiang Tan
b06e31f8e7
DEV: Remove experimental support for query string on /filter route (#20632) 2023-03-22 10:04:57 +08:00
Alan Guo Xiang Tan
66c50547b4
DEV: Experimental /filter route to filter through topics (#20494) 
This commit introduces an experimental `/filter` route which allows a
user to input a query string to filter through topics.

Internal Ref: /t/92833
 2023-03-03 09:46:21 +08:00