This commit introduces new features and utilities related to the backup
and restore system that make use of remote URLs:
- `discourse restore` accepts a URL to a backup file
- `discourse backup_url` generates a URL of a backup file (S3 only)
- `discourse import_backup_url` downloads a backup file from a URL to
the configured backup store
This can be used to move content between two Discourse instances by
backing up the entire site, copying the backup URL, importing or
restoring it on the other instance.
Commit f1700ca589 ensures that categories
are loaded lazily, in pages, if the number of visible categories is over
1000. This affected the list of subcategories on the category page too.
The logic has been changed to only paginate if the number of categories
that would have been returned is grater than 1000. For example, if there
is a parent category filter, pagination will only be enforced if the
number of subcategories is over 1000.
This commit changes the way Sidekiq is restarted when memory limit is
exceeded. The HUP signal was replaced with TERM, as mentioned in the
official documentation. The stopping timeout has been set to 10 seconds
to account for the Sidekiq timeout (5 seconds) and another for it
shutdown cleanly (5 more seconds).
See https://github.com/sidekiq/sidekiq/wiki/Deployment.
Smoke test worked only when `full_page_login` site setting was disabled
as it expected specific elements from the login modal to exist. This
commit replaces those selectors with more specific ones (prefer IDs over
classes), but which are present in both the modal or the full page.
According to Zlib documentation, this is not a fatal error. In this case
we attempt to deflate the asset three times before giving up.
Sprockets will be replaced in the long term and this is an acceptable fix.
If Jobs::DiscourseAutomation::Tracker fails during execution of a
pending automation, then the remaining automation were no longer
executed.
Automation may fail for various reasons: a person has PMs disabled, rate
limit is reached, etc.
This commit improves a few aspects regarding group name validation:
- `min_username_length` and `max_username_length` are validated with the
shortest and longest names of users and groups
- skip validation of the group name when the group is an automatic one
When `revoke_ungranted_titles!` was invoked, the optional list of
`user_ids` was not passed from the argument list to the query. This
resulted in an exception because the placeholder `:user_ids` existed in
the query.
The weekly job can take more than 2 hours to run on larger sites. It is
ideal for the jobs to be as small as possible and this is what this
commit attempts.
The hierarchical search for categories is composed of several complex
nested queries. This change ensures that the secured categories are
filtered out as soon as possible to ensure that the default limit of 5
categories is reached.
Without this fix, the search can return less than 5 categories if any
of the first 5 categories cannot be displayed due to permissions.
Sometimes `Jobs::PushNotification` gets stuck, probably because of the
network call. This commit replaces `Excon` with `FinalDestination::HTTP`
which is safer.
This spec helper was introduced as a temporary solution to the problem
of mismatched types between primary key and foreign key columns. All
plugins have been migrated and the only remaining use of this helper is
in core Discourse.
This commit removes the new optimized category style introduced in
previous commits (d37a0d40, 9a80d718 and 430c42ac), in favour of the
existent `categories_only`.
A "bad upload" in this context is a upload with a mismatched URL. This can happen when changing the S3 bucket used for uploads and the upload records in the database have not been remapped correctly.
The primary key is usually a bigint column, but the foreign key columns
are usually of integer type. This can lead to issues when joining these
columns due to mismatched types and different value ranges.
This was using a temporary plugin / test API to make tests pass. After
more careful consideration, we concluded that it is safe to alter the
tables directly.
Even for larger communities, with about 1M chat messages, the
slowest `ALTER` query runs in about 15 seconds, which well under the 30
seconds query timeout limit. As a result, chat messages will be delayed
for a few seconds, but the system will remain operational.
Multiple category styles can be used on the same site. The category and
subcategories page will use the "desktop_category_page_style" setting
and individual category pages will use the style selected in settings,
if any.
Commit c1f078ca tried to use the same style for both the category and
subcategories page, but the route matching did not take into account
the "discovery.categoryAll" and "discovery.categoryNone" variants of
the "discovery.category" route.
The primary key is usually a bigint column, but the foreign key columns
are usually of integer type. This can lead to issues when joining these
columns due to mismatched types and different value ranges.
This was using a temporary plugin / test API to make tests pass. After
more careful consideration, we concluded that it is safe to alter the
tables directly.
The `categories_only_optimized` category page style has been introduced
in commit d37a0d401c. This commit makes
sure that style is enforced for users who can see over 1000 categories
in order to keep `/categories` page functional.
The new style is called `categories_only_optimized` and it is designed
to show only the parent categories, without any subcategories. This
works best for communities with many categories (over a thousand).
The subcategories page was not paginated and it was using the
subcategory style from the category settings. The same page style should
be used for categories and subcategories page.
The `id` column of `notifications` table and `notification_id` columns
of the other tables have been migrated to bigint in previous commits
(for example, 799a45a).
In order to run the migrations with zero downtime, the data had to be
copied to new columns and swapped, but the old columns have been kept
to allow for rollback. They are no longer needed now.
This helps uncover issues with bigint columns that are joined with int
columns. It also introduces a temporary API for plugins to migrate int
columns to bigint in test environment to make tests pass.
Permanently deleting posts that no longer have a user associated was not
working as expected because of UserAction.log which expected user_id to
be present.
This commit introduces a way to fetch the "serialized voters" for
multiple polls.
* Use a single query to fetch voters for all types of polls
* Refactor to introduce all_serialized_voters
* Cache serialized voters
The user option 'hide_profile_and_presence' is necessary to figure out
if the user status has to be displayed or not. In order to avoid N+1s
generated by `include_status?` method, both `user_status` and
`user_option` relations have to be included.
`track_sql_queries` only returned queries that were executed by
ActiveRecord. All queries executed through DB.exec, DB.query and others
were not returned.
These fields are often used when serializing topics which may contain
multiple polls. On average, serializing a poll took 2+N queries where N
is the number of options. This change reduces the number of queries to
3, one for each field (Poll#voters_count, PollOption#voters_count and
Poll#has_voted?).
The recorded runtime log was specified when running the plugin specs,
but it failed to match the entries from the runtime log and the passed
as command line arguments because the runtime log entries were not
prefixed with `./`.
* DEV: Split slow test in multiple smaller tests
This might be faster because the smaller chunks of the test may run in
parallel.
* DEV: Fabricate reviewables only once
Introduce a new log line for Sidekiq jobs that consume more than
`DISCOURSE_LIVE_SLOTS_SIDEKIQ_LIMIT` live slots. This is useful to
track down jobs that may leak memory.
This is enabled only when Sidekiq's job instrumenter is enabled (set
`DISCOURSE_LOG_SIDEKIQ` to `1`).
The `notifications.id` has been migrated to bigint in previous commit
799a45a291. This commit migrates one of
the related columns, `user_badges.notification_id`, to `bigint`.
The `notifications.id` column is the most probable column to run out of
values. This is because it is an `int` column that has only 2147483647
values and many notifications are generated on a regular basis in an
active community. This commit migrates the column to `bigint`.
These migrations do not use `ALTER TABLE ... COLUMN ... TYPE` in order
to avoid the `ACCESS EXCLUSIVE` lock on the entire table. Instead, they
create a new `bigint` column, copy the values to the new column and
then sets the new column as primary key.
Related columns (see `user_badges`, `shelved_notifications`) will
be migrated in a follow-up commit.
Drafts used to be deleted instead of being destroyed. The callbacks that
clean up the upload references were not being called. As a result, the
upload references were not cleaned up and uploads were not deleted
either. This has been partially fixed in 9655bf3e.
The featured topics have not been rendered correctly since 2190c9b and
it has been fixed for desktop recently in commit d2a52c3. This commit
implements similar changes that initialize Category and Topic object
instances from the serialized data.
When a post is cooked the links are extracted and `TopicLink` instances
are created for each of them. These links are used in various places,
including the topic view, user summary page, etc.
In previous commit 48e5d1a, hotlinked images from Oneboxes have been
ignored from the texts, but hotlinked images turned into Lightboxes
were still extracted.
Navigation to subcategories page was possible through the category drop
and the "view all" link or through the category drop from the categories
page. This commit removes the latter method.
Follow up to commit 77b032c2b5.
Using the CategoryDrop on the categories page redirected the user to the
"latest topics" page with topics only from that category. With these
changes, selecting a category will take the user to a "subcategories
page" where only the subcategories of the selected property will be
displayed.
Categories are loaded in the composer via the category chooser, but
that only loads a subset of the fields. With these changes, the category
will be loaded async to make sure that the template is updated.
This commit changes request method for "categories/search" from GET to
POST to make sure that long filters can be passed to the server. For
example, category selectors with many categories are setting the full
list of selected category IDs to ensure these are filtered out from the
list of choices. This can result in a long URL that exceeds the maximum
length.
It used to embed the objects which could lead to duplicated objects
when the same user or category was used multiple times (user was admin,
moderator and category or category was parent for multiple categories).
Selecting the +subcategories option does not work sometimes when "lazy
load categories" is enabled because the subcategories may not be
fetched. This ensures that subcategories are loaded by requesting them
before being used.
CategoryBadgeSerializer and TopicCategorySerializer are too similar and
are used in similar contexts. This commit merges the two into a single
CategoryBadgeSerializer.
Full text search does not return ideal results for category dropdown.
Usually, in category dropdowns we want to search for categories as we
type. For example, while typing "theme", the dropdown should show
intermediary results for "t", "th", "the", "them" and finally "theme".
For some of these substrings (like "the"), full text search does not
return any results, which leads to an unpleasant user experience.
The breadcrumbs were updated everytime there were changes to the
categories which was not efficient and caused unnecessary rerendering
of the CategoryDrop elements when "lazy load categories" is enabled.
This commit also ensures that all category fields are serialized for
ancestors too for the categories#search endpoint.
When lazy load categories is enabled, categories should be loaded with
user activity items and drafts because the categories may not be
preloaded on the client side.
The "new topic" route can open the composer with a category preselected.
This commit ensures that the category is loaded before the composer is
opened.
Some of the properties, like 'categoriesById', 'parentCategory' and
'subcategories', were updated manually when categories were loaded.
This was not ideal because it required a lot of code to keep the
objects in sync and some of the properties were not updated correctly.
When "lazy load categories" is enabled and parent_category_id was set,
the query fetching categories contained a contradiction filtering both
by parent_category_id and parent_category_id = NULL.
When categories are loaded by the frontend, the parent category is
looked up by ID and the `parentCategory` is set with the result. If the
categories returned are not in order, the parent category may miss.
There is no need to use an initializer and manually update the DOM when
a Glimmer component can do it and ensure that the DOM is updated as more
categories are being loaded (for example, when lazy loaded categories
are enabled).
This commit improves the "+ X subcategories" option that shows sometimes
in the category selector. It used to show when there was a single match,
but now it also shows up on exact matches even though there are multiple
results.
It also makes it work when lazy_load_categories is enabled by searching
the subcategories before rendering them.
When "lazy load categories" is enabled, the CategoryDrop component will
render at most 15 categories. If there are more categories, a "Show
more" link pointing to the categories page will be displayed.
This option was introduced at some point in the past, but was removed
during the work necessary to make Discourse work with a large number of
categories.
Follow up to commit 2e68ead45b.
CategoryChooser component usually displays just categories, but
sometimes it can show two none values: a "no category" or Uncategorized.
This commit makes sure that these are rendered correctly.
The problem was that the "none" item was automatically inserted in the
list of options, but that should not always happen. Toggling option
`autoInsertNoneItem` requires setting `none` too.
Returned results are ordered by ID, but the fabricated category and
private_category IDs are not always generated in the same order. This
caused the expected and actual results to be in different orders.
Subcategories were not returned when lazy loaded categories are enabled.
This commit implements the old behavior which displayed only top level
categories when there was no search term, but when there was one, it
searched subcategories too.
This commit also removes the client-side ordering of categories, because
the results are already ordered on the server-side.
The uncategorized category was not rendered correctly and it was also
sometimes displayed twice. This commit is a similar bug fix to commit
76647d3a34 and is a follow up to commit
63a50b12fd.
Add categories to the serialized search results together with the topics
when lazy load categories is enabled. This is necessary in order for the
results to be rendered correctly and display the category information.
When "lazy load categories" is enabled, only the categories present in
the sidebar are preloaded. This is insufficient because the parent
categories are necessary too for the sidebar to be rendered properly.
This commit includes several changes to make hashtags work when "lazy
load categories" is enabled. The previous hashtag implementation use the
category colors CSS variables, but these are not defined when the site
setting is enabled because categories are no longer preloaded.
This commit implements two fundamental changes:
1. load colors together with the other hashtag information
2. load cooked hashtag data asynchronously
The first change is implemented by adding "colors" to the HashtagItem
model. It is a list because two colors are returned for subcategories:
the color of the parent category and subcategory.
The second change is implemented on the server-side in a new route
/hashtags/by-ids and on the client side by loading previously unseen
hashtags, generating the CSS on the fly and injecting it into the page.
There have been minimal changes outside of these two fundamental ones,
but a refactoring will be coming soon to reuse as much of the code
and maybe favor use of `style` rather than injecting CSS into the page,
which can lead to page rerenders and indefinite grow of the styles.
This fixes a bug where the sidebar categories would not be loaded when
the categories were lazy loaded because the sidebar uses the preloaded
category list, which was empty.
CategoryRow component uses allowUncategorized SelectKit option to
decide whether to show the "Uncategorized" category or not. This was
undefined which lead to "Uncategorized" category being always hidden
causing a minor visual glitch.
The old strategy used to load 25 categories at a time, including the
subcategories. The new strategy loads 20 parent categories and at most
5 subcategories for each parent category, for a maximum of 120
categories in total.
When navigating straight to a topic the category was not displayed at
all because the categories were not loaded. Similarly, the categories
for suggested topics were not loaded either.
This commit adds a list of categories to topic view model class and
serializer.
Bianca Nenciu