Commit Graph
100 Commits
Author SHA1 Message Date
Alan Guo Xiang Tan 839caa274a DEV: Drop show_filter_by_tag site settings.
Follow-up to 83519e7f3a
2021-03-11 10:48:18 +08:00
Alan Guo Xiang Tan a1df45c6bc UX: Fix broken styling on login modal when local login is disabled. 2021-03-10 15:53:22 +08:00
Alan Guo Xiang Tan 1106aee152 FIX: Incorrect parameter when using omniauth providers on invites page.
Follow-up to ce04db8610
2021-03-09 13:54:33 +08:00
Alan Guo Xiang Tan ce04db8610 FEATURE: Allow invites redemption with Omniauth providers. 2021-03-09 09:27:18 +08:00
Alan Guo Xiang Tan ebe4896e48 FEATURE: Change very high/low search priority to rank at absolute ends.
Prior to this change, we had weights for very_high, high, low and
very_low. This means there were 4 weights to tweak and what weights to
use for `very_high/high` and `very_low/low` pair was hard to explain.
This change makes it such that `very_high` search priority will always
ensure that the posts are ranked at the top while `very_low` search
priority will ensure that the posts are ranked at the very bottom.
2021-03-09 09:20:37 +08:00
Alan Guo Xiang Tan 4a41f72f09 DEV: DRY up wavingHandUrl code.
Allows for resuse in other controllers and components.
2021-03-04 15:18:54 +08:00
Alan Guo Xiang Tan 51483e1aef FIX: Make email a required param when accepting invite links.
A missing email when accepting an invite link does not make sense so we
should make it a required param which helps to catch bugs in our test
suite and also prevent potential bugs in our code base when the code
trips on a `nil` email.
2021-03-04 09:27:04 +08:00
Alan Guo Xiang Tan 3fc72543de DEV: Avoid duplicating method attributes in Auth::Result.
This ensures that within the class, we have a consistent order of
attributes that is passed to `UsernameSuggester`.
2021-02-26 10:20:31 +08:00
Alan Guo Xiang Tan 4b3f65bb26 FIX: Select earliest post when aggregating posts in a topic for search.
This is a revert of
https://github.com/discourse/discourse/commit/d8c796bc44d84a9e177737153793e5acc276adec
and
https://github.com/discourse/discourse/commit/5bf0a0893b3e508383e1fe2558cc54e12147a4f3.

Linking to the post within a topic that has the highest rank was
confusing users and hard to explain because ranking is determined via
the PG ranking function. See the following meta topics for the
complaints after we switch to the new ordering:

1. https://meta.discourse.org/t/title-search-not-working-as-expected/157737
2. https://meta.discourse.org/t/search-results-should-prioritize-first-post-in-topic-when-title-matches-search-term/175154
2021-02-05 09:52:53 +08:00
Alan Guo Xiang Tan 0cc178d58b FIX: Avoid pulling hotlinked images of post that have been deleted. (#11913) 2021-02-03 16:45:07 +11:00
Alan Guo Xiang Tan 7e4dad3c56 DEV: Update msgpack to 1.4.2.
It seems like 1.4.1 was yanked.
2021-02-01 11:46:38 +08:00
Alan Guo Xiang Tan 05e8c1ebf8 FIX: Unescape emoji when pg headlines is used for search result blurb.
Follow-up to 8336c2235e
2021-01-20 09:31:25 +08:00
Alan Guo Xiang Tan d9109ed436 FIX: Show all topic statuses on full page search. 2020-12-23 10:30:35 +08:00
Alan Guo Xiang Tan db235c5ff9 DEV: Improve specs to use objects that have already been fabricated. 2020-12-23 10:25:23 +08:00
Alan Guo Xiang Tan d2a0462186 FIX: Re-run remove flag and queued post web hook migrations.
There have been production instances where the migrations has run but the records
that were meant to be deleted have not been deleted. The root cause is
unknown but the migration is safe and simple to re-run. The problem is
not reproducible locally so we're not spending too much time on digging
up the root. Time vs business cost tradeoff.

Follow-up fb15da43da
2020-12-21 10:17:08 +08:00
Alan Guo Xiang Tan 38b6b098bc FIX: Bypass AnonymousCache for /srv/status route. (#11491)
`/srv/status` routes should not be cached at all. Also, we want to
decouple the route from Redis which `AnonymouseCache` relies on. The
`/srv/status` should continue to return a success response even if Redis
is down.
2020-12-16 16:47:46 +11:00
Guo Xiang Tan f70b330e7a DEV: Fix the build.
Follow-up to 650da7b626
2020-11-09 14:25:14 +08:00
Guo Xiang Tan 650da7b626 PERF: Update index for category in a background job.
Search indexing can get expensive and there is no need for us to block
the entire request just to wait for index to finish.
2020-11-09 13:51:26 +08:00
Alan Guo Xiang Tan 0e4d966a74 DEV: Remove stale ignored_columns. (#11160) 2020-11-09 15:57:59 +11:00
Guo Xiang Tan 68fc2a18b1 FIX: Properly handle quotes and backslash in Search.set_tsquery_weight_filter 2020-10-23 08:43:34 +08:00
Guo Xiang Tan 2607bb602e Fix broken spec.
Follow-up to 3c678df942
2020-10-08 10:52:46 +08:00
Guo Xiang Tan cabbc32024 FIX: Scheduled jobs not showing up in /sidekiq/schedule in dev. 2020-10-08 10:24:51 +08:00
Guo Xiang Tan 495c79da1a Revert "FIX: Use topic title headline in search menu result."
This reverts commit 4b8c15b857.

This breaks topic title for topic similar search.
2020-10-02 13:12:31 +08:00
Guo Xiang Tan 4b8c15b857 FIX: Use topic title headline in search menu result. 2020-10-02 09:04:54 +08:00
Guo Xiang Tan 911d47934a DEV: Prefer update! over update when return value is not checked. 2020-09-24 09:56:09 +08:00
Guo Xiang Tan d83d9ec408 DEV: Clear all DistributedCache on Redis fallback. 2020-09-18 11:31:49 +08:00
Guo Xiang Tan b47b640598 FEATURE: Hidden SiteSetting.keep_old_ip_address_count to track IP history. 2020-09-17 12:50:39 +08:00
Guo Xiang Tan ddcf0d7d01 Update rails_failover to 0.5.7. 2020-09-16 13:53:45 +08:00
Guo Xiang Tan 49f53bdc50 DEV: Fix another MessageBus::InvalidMessageTarget error.
Follow-up to c2f2b57a9b.
2020-09-15 14:25:10 +08:00
Guo Xiang Tan f7438fc64b DEV: Fix MessageBus::InvalidMessageTarget error after upgrading mb.
Follow-up to c2f2b57a9b
2020-09-15 14:15:42 +08:00
Guo Xiang Tan c2f2b57a9b Update message_bus to 3.3.2. 2020-09-15 13:33:11 +08:00
Guo Xiang Tan 2ff16b3650 FIX: TopicQuery.list_private_messages_unread ignore notification level 2020-09-15 13:33:11 +08:00
Guo Xiang Tan e3b4632f04 Revert "Create proper shim for xss library"
This reverts commit 30e9112535.
2020-09-15 11:38:40 +08:00
Guo Xiang Tan 543e972fec PERF: Remove category_user lookup when loading private messages.
Private messages do not belong to categories so the query is unnecessary
overhead.
2020-09-11 16:26:38 +08:00
Guo Xiang Tan cd78bcee3a DEV: Avoid creating the same object repeatedly.
Less objects created per request means less objects for the GC to
collect.
2020-09-11 15:51:11 +08:00
Guo Xiang Tan dbc630f45b PERF: Fix N+1 queries on private messages route. 2020-09-11 15:20:27 +08:00
Guo Xiang Tan 245d29e5a3 SECURITY: Mod should not see group_users and second_factor_enabled.
Moderators should not be able to see `UserSerializer#group_users` and `UserSerializer#second_factor_enabled` of other users.

Impact of leaking this is low because the information leaked is not
exploitable.
2020-09-11 10:23:35 +08:00
Guo Xiang Tan 468417a716 PERF: Fix broken memoization in GroupShowSerializer. 2020-09-11 10:08:06 +08:00
Guo Xiang Tan 7fc70900d2 Revert "FEATURE: moderators allowed to view groups which members can see."
Build is broken.

This reverts commit 2bf0c4188b.
2020-09-11 10:04:54 +08:00
Guo Xiang Tan f27de87bf3 FIX: Update first_pm_unread_at of user's groups without unread.
If a user always read all group messages, we will never update the
`first_pm_unread_at` column since the previous query will not return the
group_user. Instead, we should update `first_pm_unread_at` to the
current timestamp if the user has read everything.

Follow-up to 9b75d95fc6
2020-09-10 17:19:38 +08:00
Guo Xiang Tan 122cf8d3fb Update Rails to 6.0.3.3. 2020-09-10 15:41:06 +08:00
Guo Xiang Tan 81d3c4a7a1 DEV: Remove unused function. 2020-09-10 15:11:35 +08:00
Guo Xiang Tan 5732e4288e DEV: Remove redundant variable. 2020-09-10 14:35:15 +08:00
Guo Xiang Tan 5ef9d4d537 PERF: Use 1 query to load group and tag notification default. 2020-09-10 13:19:43 +08:00
Guo Xiang Tan 521782fc9c FIX: Search checkboxes incorrectly being checked on similar prefix.
Incorrect search filters like `in:personalasd` will end up checking the
checkbox for `in:personal` because the regexp used was only doing prefix
matching.
2020-09-10 11:49:52 +08:00
Guo Xiang Tan e6ca1b4326 FIX: Admin search for PMs should only search own PMs.
In c6ceda8c, a bug was introduced where an admin searching for his own
private messages will actually end up searching through all private
messages on the site.

Follow-up to c6ceda8c4e
2020-09-10 11:37:18 +08:00
Guo Xiang Tan 611d9f61fd Fix the build. 2020-09-09 15:43:38 +08:00
Guo Xiang Tan 4994b0ed34 PERF: Remove an unncessary query when check for dark mode. 2020-09-09 15:18:52 +08:00
Guo Xiang Tan fb7bbae3f4 PERF: Memoize calls to ApplicationHelper#scheme_id.
The same query was executed 4 times per full page load.
2020-09-09 15:15:15 +08:00
Guo Xiang Tan 9b75d95fc6 PERF: Keep track of first unread PM and first unread group PM for user.
This optimization helps to filter away topics so that the joins on
related tables when querying for unread messages is not expensive.
2020-09-09 14:05:41 +08:00
Guo Xiang Tan 0398271f87 FIX: Admin should always be able to view group messages.
Follow-up to b0f22f2523.

Before this fix, `SiteSetting.enable_personal_messages = false` will
prevent admin from viewing group messages.
2020-09-09 08:58:21 +08:00
Guo Xiang Tan a1d135f12a DEV: Correct use of sanitize_sql_array in TopicQuery. 2020-09-08 12:30:09 +08:00
Guo Xiang Tan d3ebaa41ce DEV: Address review comments for 5ed84d9885. 2020-09-08 11:17:35 +08:00
Guo Xiang Tan b0f22f2523 SECURITY: Remove indication that a group exists if user can't see it.
Minor security fix but we should not leak any hints that a group exists
even if a user does not have access to the group.
2020-09-08 10:53:48 +08:00
Guo Xiang Tan 5ed84d9885 SECURITY: Don't allow moderators to list PMs of all groups.
* Also return 404 when a user is trying to list PMs of a group that
cannot be accessed by the user.
2020-09-08 10:37:00 +08:00
Guo Xiang Tan 0d3239bf21 Revert "SECURITY: Don't allow moderators to view the admins inbox"
Superseeded by d9a5280f5665d12bf46efd8cdcc6200da2cdedd8

This reverts commit 18d35bf64a.
2020-09-08 10:36:49 +08:00
Guo Xiang Tan 01600492de FIX: Don't raise error in update username job if user has been deleted. 2020-09-02 11:17:17 +08:00
Guo Xiang Tan 069a109cbb DEV: Require scheduled job in development to avoid loading file twice.
This removes the need to memoize constant in order to avoid the "warning: already initialized constant".
2020-09-01 10:14:40 +08:00
Guo Xiang Tan 09a89cff1a DEV: Increase log levels for more assets/static paths. 2020-09-01 09:52:50 +08:00
Guo Xiang Tan 20bd6d9797 DEV: Only eagerload scheduled jobs in a Sidekiq process. 2020-08-31 14:28:38 +08:00
Guo Xiang Tan 5270cd97ef DEV: Leaking state in PostActionNotifier in tests take 2.
Follow-up to a3b48826
2020-08-31 12:55:42 +08:00
Guo Xiang Tan a3b4882665 DEV: Leaking state in PostActionNotifier in tests.
Follow-up to 8a0478b97d
2020-08-31 12:04:17 +08:00
Guo Xiang Tan 31652c1ad3 DEV: Improve assertions for spec.
This was showing up as a flaky spec.
2020-08-31 11:08:18 +08:00
Guo Xiang Tan cff57c1883 DEV: Throw an error instead of a string. 2020-08-28 11:40:16 +08:00
Guo Xiang Tan a4ec4c06e3 Fix the build. 2020-08-27 13:50:07 +08:00
Guo Xiang Tan 40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Guo Xiang Tan dbfb2a1e11 Minor follow-up bug fix for 5ff6c10. 2020-08-26 17:25:28 +08:00
Guo Xiang Tan 5ff6c10320 FIX: Ensure that topic timers belonging to trashed topics are destroyed. 2020-08-26 16:11:10 +08:00
Guo Xiang Tan 3cc761ac44 FIX: Clean up toggle closed topic timer when user is not authorized. 2020-08-26 12:59:05 +08:00
Guo Xiang Tan 83314d1534 DEV: Fix the broken build caused by 1e8f216. 2020-08-26 12:36:51 +08:00
Guo Xiang Tan 5bca1aec48 FIX: Clean up topic_timers when no longer valid.
This was causing a bug where the jobs for the topic_timers will keep
getting enqueued over and over again.
2020-08-26 12:18:51 +08:00
Guo Xiang Tan 1e8f216e17 FIX: Ensure autoclose based on last post is executed by system user. 2020-08-26 12:08:46 +08:00
Guo Xiang Tan ffc5bb7278 Revert "FIX: Topic Timer auto opening closed topics (#10524)"
This reverts commit 7cfd5f87ff.
2020-08-26 09:32:11 +08:00
Guo Xiang Tan f08d440ea0 DEV: Disable migration advisory locks in dev and test environment.
There is a bug in Rails that leaks an idle PG session which will prevent
the dropping of the database.
2020-08-25 14:20:58 +08:00
Guo Xiang Tan 9c997ae117 Revert "DEV: allow plugins to send extra silenced message params"
This reverts commit c3aeb442d7.

Broken specs so helping Jeff to revert first.
2020-08-25 09:57:57 +08:00
Guo Xiang Tan 9354ee35d2 DEV: Ensure test resets to original constant.
Also simplifies the test to reduce amount of test setup done.
2020-08-25 09:24:59 +08:00
Guo Xiang Tan 05174df5c0 FIX: Restrict personal_messages: advanced search filter to admin.
The filter noops if an incorrect username is passed. This filter is not
exposed as part of the UI but is only used when an admin transitions
from a search within a user's personal messages to the full page search.

Follow-up to 4b30799054.
2020-08-24 13:53:48 +08:00
Guo Xiang Tan c6ceda8c4e PERF: Avoid extra subquery when searching within PMs for normal user.
Note the following query being generated where the filter for a user's
private messages is executed twice.

```sql
SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", (TS_RANK_CD(
  post_search_data.search_data,
  TO_TSQUERY('english', '''test'':*ABCD'),
  0|32
)
 * (
  CASE categories.search_priority
  WHEN 2
  THEN 0.6
  WHEN 3
  THEN 0.8
  WHEN 4
  THEN 1.2
  WHEN 5
  THEN 1.4
  ELSE
    CASE WHEN topics.closed
    THEN 0.9
    ELSE 1
    END
  END
)
) rank, topics.bumped_at topic_bumped_at FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3) AND (topics.visible) AND (topics.archetype = 'private_message' AND post_search_data.private_message) AND (posts.topic_id IN (SELECT topic_id
FROM topic_allowed_users
WHERE user_id = 99999
UNION ALL
SELECT tg.topic_id
FROM topic_allowed_groups tg
JOIN group_users gu ON gu.user_id = 99999 AND gu.group_id = tg.group_id
)) AND (post_search_data.search_data @@ TO_TSQUERY('english', '''test'':*ABCD')) AND (posts.topic_id IN (SELECT topic_id
FROM topic_allowed_users
WHERE user_id = 99999
UNION ALL
SELECT tg.topic_id
FROM topic_allowed_groups tg
JOIN group_users gu ON gu.user_id = 99999 AND gu.group_id = tg.group_id
)) AND ((categories.id IS NULL) OR (NOT categories.read_restricted) OR (categories.id IN (999999))) ORDER BY rank DESC, topic_bumped_at DESC
```
2020-08-24 13:49:43 +08:00
Guo Xiang Tan 2f043dc89a Fix lint. 2020-08-24 12:38:46 +08:00
Guo Xiang Tan 4b30799054 FIX: Correct personal_messages:<username> advanced search filter.
Renamed from `private_messages` to `personal_messages` without
deprecation because the `private_messages` advanced search filter never
worked in the first place when it was implemented.
2020-08-24 11:54:30 +08:00
Guo Xiang Tan 106a2f58a2 DEV: Drop support for deprecated in:private search filter. 2020-08-21 17:18:39 +08:00
Guo Xiang Tan ab5d738231 DEV: Improve search spec to test for actual posts.
Testing for count is a pitfall since a wrong post can be returned and
the tests will still pass.
2020-08-21 15:49:26 +08:00
Guo Xiang Tan 0684118008 DEV: Remove array_agg from search orders that does not need it. 2020-08-21 14:39:07 +08:00
Guo Xiang Tan daddad7fd6 DEV: Update annotations. 2020-08-21 11:36:53 +08:00
Guo Xiang Tan d12d8fb7fd DEV: Include more information when reporting search indexing failures. 2020-08-21 11:02:00 +08:00
Guo Xiang Tan aae9e6e5fd FIX: Topic.similar_to results in error if cooked raw is blank. 2020-08-21 10:54:08 +08:00
Guo Xiang Tan 337f062f0f PERF: Defer indexing post for search when saving a post.
Indexing a post for search is slow and there is no reason for us to have
to block saving a post due to search indexing.
2020-08-21 07:52:43 +08:00
Guo Xiang Tan f4a0d200ee DEV: Drop search index on non-pm posts take 2.
Follow-up to 87de8948c0
2020-08-21 07:27:37 +08:00
Guo Xiang Tan 87de8948c0 DEV: Drop search index on non-pm posts.
The problem with this index is that on sites with a high non-pm to pm
posts ratio, the index is esstentially duplicating the existing index on
`PostSearchData#search_data`. If the site is huge, the index ends up
taking up more diskspace.
2020-08-21 07:21:34 +08:00
Guo Xiang Tan 0967ce478d DEV: Remove migration that is no longer required.
Avoid creating the index twice.

Follow-up to 628319aad3
2020-08-20 14:25:20 +08:00
Guo Xiang Tan 7e414da0d9 DEV: Fix lint. 2020-08-18 16:59:57 +08:00
Guo Xiang Tan 2161abfabd DEV: Move data migration of PostSearchData#private_message into post_migration.
Follow-up to 92b7fe4c62
2020-08-18 16:46:14 +08:00
Guo Xiang Tan 92b7fe4c62 PERF: Add partial index for non-pm search. 2020-08-18 15:55:08 +08:00
Guo Xiang Tan 872ef82f4d Revert "PERF: Prefer joins over subquery for User#private_posts_for_user."
This reverts commit 05b43e5ae4.
2020-08-17 14:58:47 +08:00
Guo Xiang Tan 05b43e5ae4 PERF: Prefer joins over subquery for User#private_posts_for_user.
The subquery here prevents the planner from optimizing the query. As
such, we prefer joining against the requried tables instead.
2020-08-17 14:47:48 +08:00
Guo Xiang Tan 248bebb8cd PERF: Remove extra subquery in search.
I also noticed that removing the subquery helps the planner to plan
better.
2020-08-17 13:52:12 +08:00
Guo Xiang Tan 8b811533b1 DEV: Improve readability of setting weights in SearchIndexer. 2020-08-14 23:11:41 +08:00
Guo Xiang Tan 5819c4cb3b PERF: Switch to ActiveRecord's upsert in SearchIndexer.
On insertion, it uses a single query instead of 2.
2020-08-14 16:15:14 +08:00
Guo Xiang Tan c2605d4194 DEV: Fix search to be more intentional about what it is testing.
Asserting for the posts length is not accurate since an incorrect post
can be returned and the assertion will still pass.
2020-08-14 15:32:18 +08:00
Guo Xiang Tan 808e84a856 FIX: Expose PG headline highlighting for all search results. 2020-08-12 16:43:47 +08:00