discourse/lib
Sam 32393f72b1
PERF: backoff background requests when overloaded (#10888)
When the server gets overloaded and lots of requests start queuing server
will attempt to shed load by returning 429 errors on background requests.

The client can flag a request as background by setting the header:
`Discourse-Background` to `true`

Out-of-the-box we shed load when the queue time goes above 0.5 seconds.

The only request we shed at the moment is the request to load up a new post
when someone posts to a topic.

We can extend this as we go with a more general pattern on the client.

Previous to this change, rate limiting would "break" the post stream which
would make suggested topics vanish and users would have to scroll the page
to see more posts in the topic.

Server needs this protection for cases where tons of clients are navigated
to a topic and a new post is made. This can lead to a self inflicted denial
of service if enough clients are viewing the topic.

Due to the internal security design of Discourse it is hard for a large
number of clients to share a channel where we would pass the full post body
via the message bus.

It also renames (and deprecates) triggerNewPostInStream to triggerNewPostsInStream

This allows us to load a batch of new posts cleanly, so the controller can
keep track of a backlog

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-10-13 16:56:03 +11:00
..
auth DEV: Move UserApiKey scopes to dedicated table (#10704) 2020-09-29 10:57:48 +01:00
autospec REFACTOR: Move javascript tests inside discourse app 2020-10-02 11:29:36 -04:00
backup_restore FIX: Prevent "uploads are missing in S3" alerts after restoring a backup 2020-09-10 21:37:48 +02:00
common_passwords DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
compression FIX: Decompressing lots of small files triggered error 2020-01-09 15:11:31 +01:00
content_security_policy Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
demon DEV: Review fixes (#10641) 2020-09-10 13:41:46 +10:00
email FIX: Confirm new email not sent for staff if email disabled with "non-staff" option (#10794) 2020-10-08 13:52:17 +10:00
emoji DEV: supports unicorn emoji 13.0beta (#8402) 2019-11-25 10:23:18 +01:00
file_store Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
freedom_patches DEV: Remove unnecessary rack patches. 2020-08-03 14:17:10 +08:00
generators REFACTOR: Move javascript tests inside discourse app 2020-10-02 11:29:36 -04:00
guardian FEATURE - SiteSetting to disable user option to hide their profiles and presences (#10885) 2020-10-09 17:18:44 -04:00
highlight_js DEV: already defined constant 'HIGHLIGHTJS_DIR' 2019-01-21 10:12:23 +01:00
i18n FIX: Pluralized translation overrides didn't work for en_US 2020-08-29 00:11:46 +02:00
imap DEV: IMAP log to database (#10435) 2020-08-14 12:01:31 +10:00
import DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
import_export FEATURE: Include category position when exporting categories (#9658) 2020-05-07 12:17:15 +10:00
javascripts REFACTOR: Support bundling our admin section as an ember addon 2020-09-22 15:14:29 -04:00
middleware PERF: backoff background requests when overloaded (#10888) 2020-10-13 16:56:03 +11:00
migration FIX: Allow post migrations using #change to carry out unsafe migration 2020-05-15 14:23:27 +08:00
onebox FEATURE: Additional control of iframes in oneboxes (#10523) 2020-08-27 20:12:13 +01:00
plugin DEV: Add discourse-teambuild to official plugins (#10874) 2020-10-09 14:50:00 -07:00
pretty_text FIX: Missing files from previous commit 2020-09-15 10:59:41 -04:00
rate_limiter DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
reviewable FEATURE: Nokogumbo (#9577) 2020-05-05 13:46:57 +10:00
scheduler FEATURE: log long running jobs in the defer queue 2018-10-12 17:03:47 +11:00
search FIX: Limit PG headline based search blurb generation to 200 characters. 2020-08-12 15:34:27 +08:00
seed_data DEV: correct some Ruby 2.7 deprecations 2020-07-16 17:43:20 +10:00
sidekiq DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
site_settings FIX: second factor cannot be enabled if SSO is enabled (#10880) 2020-10-09 22:36:38 +05:30
stylesheet FEATURE: Separate base and heading font site_settings (#10807) 2020-10-05 13:40:41 -04:00
svg_sprite FIX: update meh-o icon to far-meh (#10256) 2020-07-16 16:26:17 -07:00
tasks REFACTOR: Remove .erb file from javascript tests 2020-10-08 18:36:09 -04:00
theme_store FIX: allow plugin pinning to fetch missing commits 2020-08-12 20:03:37 -07:00
turbo_tests DEV: Add seed explicitly in turbo_rspec so that runs can be reproduced 2020-08-07 16:51:53 +01:00
validators FIX - limit number of embedded media items in a post (#10391) 2020-08-07 12:08:59 -04:00
webauthn SECURITY: 2FA with U2F / TOTP 2020-01-15 11:27:12 +01:00
wizard Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
admin_confirmation.rb DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
admin_constraint.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
admin_user_index_query.rb DEV: Standardize table sorting verbiage (#9757) 2020-05-14 20:10:59 -06:00
age_words.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
archetype.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
auth.rb DEV: Drop legacy OpenID 2.0 support (#8894) 2020-02-07 17:32:35 +00:00
backup_restore.rb FIX: Ensure multisite DB is set correctly when spawning backup process 2020-09-18 20:06:49 +01:00
badge_posts_view_manager.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
badge_queries.rb FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
base62.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
bookmark_manager.rb FIX: Make deleted topic post bookmarks more resilient (#10619) 2020-09-07 14:52:14 +10:00
bookmark_query.rb FIX: Bookmark search fixes (#10239) 2020-07-17 15:55:07 +10:00
bookmark_reminder_notification_handler.rb FIX: dont error when bookmark topic is nil (#10555) 2020-08-31 09:15:36 +10:00
browser_detection.rb FIX: Detect DiscourseHub user agent. 2019-08-09 11:58:15 +03:00
cache.rb DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
canonical_url.rb FEATURE: default canonical URL (#9738) 2020-05-12 09:13:20 +10:00
category_badge.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
comment_migration.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
composer_messages_finder.rb DEV: pluck_first 2019-10-21 12:08:20 +01:00
configurable_urls.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
content_buffer.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
content_security_policy.rb FIX: Allow CSP to work correctly for non-default hostnames/schemes (#9180) 2020-03-19 19:54:42 +00:00
cooked_post_processor.rb FIX: Make sure rel attributes are correctly set. (#10645) 2020-09-10 12:59:51 -03:00
crawler_detection.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
csrf_token_verifier.rb DEV: Provide method for auth plugins to generate a CSRF token 2019-08-13 01:13:08 +01:00
current_user.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
custom_renderer.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
custom_setting_providers.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
db_helper.rb DEV: Skip readonly tables when remapping content (#10461) 2020-08-18 08:53:12 +01:00
directory_helper.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
discourse_cookie_store.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
discourse_diff.rb FEATURE: Nokogumbo (#9577) 2020-05-05 13:46:57 +10:00
discourse_event.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
discourse_hub.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
discourse_ip_info.rb FIX: MaxMind DB file not downloading correctly 2020-01-05 22:08:13 +11:00
discourse_js_processor.rb REFACTOR: Support bundling our admin section as an ember addon 2020-09-22 15:14:29 -04:00
discourse_logstash_logger.rb FIX: Use 'hostname' when Discourse.os_hostname is not available 2020-02-18 13:37:39 +02:00
discourse_plugin_registry.rb DEV: Add plugin api to permit bulk_action parameters (#10638) 2020-09-10 17:18:45 +01:00
discourse_redis.rb DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
discourse_tagging.rb FIX: Return 422 when creating topics with tags w/out permission (#10400) 2020-08-10 16:14:15 -06:00
discourse_updates.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
discourse.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
disk_space.rb FIX: correct upload statistics report for external storage 2020-02-20 15:15:53 +11:00
distributed_cache.rb REFACTOR: distributed_cache is moved to the message_bus gem 2018-10-15 15:01:45 -04:00
distributed_memoizer.rb DEV: Replace Time.new with Time.now (#9142) 2020-03-09 17:37:49 +01:00
distributed_mutex.rb FIX: Off-by-one error setting the distributed mutex key to expire 2020-02-03 14:54:50 +00:00
edit_rate_limiter.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
email_backup_token.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
email_cook.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
email_updater.rb FIX: When admin changes an email for the user the user must confirm the change (#10830) 2020-10-07 13:02:24 +10:00
email.rb FEATURE: IMAP delete email sync for group inboxes (#10392) 2020-08-12 10:16:26 +10:00
encodings.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
enum_site_setting.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
enum.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
excerpt_parser.rb FIX: do not raise error if 'class' attribute is not found. 2020-05-01 10:03:40 +05:30
feed_element_installer.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
feed_item_accessor.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
file_helper.rb SECURITY: Add content-disposition: attachment for SVG uploads 2020-07-09 13:31:48 +10:00
filter_best_posts.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
final_destination.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
flag_query.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
flag_settings.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
gaps.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
global_path.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
guardian.rb FIX: respect moderator group permissions in guardian (#10713) 2020-09-21 12:32:43 -04:00
has_errors.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
hijack.rb Take 2 of 0f5161af19. 2019-04-29 16:41:35 +08:00
homepage_constraint.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
html_prettify.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
html_to_markdown.rb Revert "DEV: Add logging for stack level too deep exception in HtmlToMarkdown" 2020-07-09 13:41:33 +01:00
http_language_parser.rb FIX: Include resolved locale in anonymous cache key (#10289) 2020-07-22 18:00:07 +01:00
image_sizer.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
import_export.rb FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
inline_oneboxer.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
introduction_updater.rb FIX: replace default welcome topic post with new value from wizard 2020-04-01 15:42:45 -04:00
ip_addr.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
js_locale_helper.rb PERF: ensure we run full GC on contexts 2020-05-15 14:01:54 +10:00
json_error.rb FIX: Fix build. 2019-05-22 17:39:44 +03:00
letter_avatar.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
markdown_linker.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
mem_info.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
message_bus_diags.rb PERF: avoid shelling to get hostname aggressively 2020-02-18 15:13:19 +11:00
method_profiler.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
mini_sql_multisite_connection.rb DEV: Allow DB.after_commit to be used outside of a transaction 2020-05-04 09:42:41 +01:00
mobile_detection.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
new_post_manager.rb FEATURE: Review posts with media. (#10693) 2020-09-18 12:45:09 -03:00
new_post_result.rb Support for custom messages and redirects when creating posts (#8434) 2019-11-29 09:30:54 -05:00
notification_levels.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
oneboxer.rb FEATURE: Additional control of iframes in oneboxes (#10523) 2020-08-27 20:12:13 +01:00
onpdiff.rb FIX: Show a correct diff when editing consecutive paragraphs (#8177) 2019-10-11 03:50:37 -04:00
pbkdf2.rb DEV: Switch our fast_xor gem for xorcist (#10565) 2020-08-31 13:20:44 -06:00
permalink_constraint.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
pinned_check.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
plain_text_to_markdown.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
plugin_gem.rb FEATURE: allow specifying platform name for platform specific gems (#10559) 2020-08-31 12:36:04 +10:00
plugin_initialization_guard.rb DEV: Print backtrace of error when plugin fails to initialize. 2020-06-09 10:25:43 +08:00
post_action_creator.rb FIX: Reviews that are auto-hidden by a trusted spam flagger should always have enough weight. (#10284) 2020-07-22 11:42:15 -03:00
post_action_destroyer.rb FIX: Unlike own posts on ownership transfer (#10446) 2020-08-19 09:21:02 -06:00
post_action_result.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
post_creator.rb FIX: Ensure auto close notice is posted with system locale 2020-08-17 15:40:47 +01:00
post_destroyer.rb DEV: skip topic deletion staff log when user skip new user tips. 2020-09-23 12:25:39 +05:30
post_jobs_enqueuer.rb FIX: the muted message should be sent after edit (#9593) 2020-05-01 08:33:57 +10:00
post_locker.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
post_merger.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
post_revisor.rb FIX: Ensure uploads are linked to post when revising a post 2020-09-11 11:49:15 +02:00
pretty_text.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
promotion.rb FIX: TL2 promotion message and advance training (#10679) 2020-09-22 10:17:52 +10:00
quote_comparer.rb FEATURE: Nokogumbo (#9577) 2020-05-05 13:46:57 +10:00
rake_helpers.rb Try fix upload_spec flakys and remove logging from tasks/uploads_spec 2020-02-18 15:08:58 +10:00
rate_limiter.rb FIX: Ignore Redis readonly errors in RateLimiter#rollback!. 2020-06-11 15:13:11 +08:00
read_only_header.rb DEV: rename ReadOnly module to ReadOnlyHeader 2019-05-06 16:07:49 +02:00
retrieve_title.rb FEATURE: Nokogumbo (#9577) 2020-05-05 13:46:57 +10:00
route_format.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
rtl.rb Check site default locale if Rtl class is initialized without a user (#8417) 2019-11-26 15:01:37 -05:00
s3_helper.rb FIX: AWS S3 errors don't necessarily include a message 2020-08-12 17:00:09 -04:00
s3_inventory.rb DEV: Change upload verified column to be integer (#10643) 2020-09-17 13:35:29 +10:00
score_calculator.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
screening_model.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
search.rb Fix broken spec. 2020-10-08 10:52:46 +08:00
secure_session.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
shrink_uploaded_image.rb FIX: Handle the case where upload goes missing during downsizing 2020-07-06 18:51:38 +02:00
single_sign_on_provider.rb FIX: Handle missing provider return sso url 2020-05-12 18:16:50 -06:00
single_sign_on.rb FEATURE: support SSO website and location overrides 2020-04-28 16:06:35 +10:00
site_icon_manager.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
site_setting_extension.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
slug.rb FIX: If a prettified slug is a number, return defaultt (#8554) 2019-12-17 10:34:20 +10:00
socket_server.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
spam_handler.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
sql_builder.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
staff_constraint.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
staff_message_format.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
suggested_topics_builder.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
system_message.rb FIX: TL2 promotion message and advance training (#10679) 2020-09-22 10:17:52 +10:00
text_cleaner.rb FEATURE: English locale with international date formats 2019-05-20 13:47:20 +02:00
text_sentinel.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
theme_javascript_compiler.rb DEV: Remove Discourse.RAW_TEMPLATES (#9630) 2020-05-05 12:15:03 -04:00
theme_modifier_helper.rb DEV: Allow plugins to add theme modifiers via db migrations (#9192) 2020-03-12 16:35:28 +00:00
theme_settings_manager.rb FEATURE: add support for upload format in theme settings. 2020-04-15 18:34:02 +05:30
theme_settings_parser.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
theme_translation_manager.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
theme_translation_parser.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
timeline_lookup.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
topic_creator.rb FIX - downcase strings consistently (#10861) 2020-10-08 11:28:07 +11:00
topic_list_responder.rb DEV: Refactor draft attributes for CategoryList and TopicList. 2020-07-24 10:11:30 +08:00
topic_publisher.rb FIX: Use destroy_all instead of delete_all for shared drafts 2020-03-05 11:13:43 -08:00
topic_query_params.rb Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse"" 2020-05-23 00:56:13 -04:00
topic_query_sql.rb DEV: Rails 5.2 upgrade and global gem upgrade 2018-06-07 14:21:33 +10:00
topic_query.rb FIX: Respect show_category_definitions_in_topic_lists in category lists (#10853) 2020-10-07 14:19:48 -04:00
topic_retriever.rb FIX: An opts hash was not, in fact, optional :) 2020-04-20 14:17:13 -04:00
topic_subtype.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
topic_upload_security_manager.rb FEATURE: Update upload security status on post move, topic conversion, category change (#8731) 2020-01-23 12:01:10 +10:00
topic_view.rb FIX: Make deleted topic post bookmarks more resilient (#10619) 2020-09-07 14:52:14 +10:00
topics_bulk_action.rb FEATURE: bulk remove tags (#10831) 2020-10-06 22:41:55 +05:30
trust_level.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
turbo_tests.rb FIX: Made turbo_rspec display errors in shared groups correctly 2019-08-29 12:41:14 +01:00
twitter_api.rb FIX: Add playsinline to twitter GIFs 2020-08-18 00:26:51 +01:00
unicorn_logstash_patch.rb DEV: Fix lint. 2020-07-21 15:55:03 +08:00
unread.rb DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
upload_creator.rb FIX: deprecate whitelist constants (#10716) 2020-09-28 13:52:05 +10:00
upload_fixer.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
upload_markdown.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
upload_recovery.rb DEV: Recover missing files of existing uploads (#10757) 2020-10-01 14:54:45 +02:00
upload_security.rb DEV: Fix flaky UploadSecurity spec 2020-09-18 14:36:41 -04:00
url_helper.rb Replace base_uri with base_path (#10879) 2020-10-09 12:51:24 +01:00
user_lookup.rb PERF: Combine avatar_lookup and primary_group_lookup into user_lookup (#10253) 2020-07-17 10:48:08 +01:00
user_name_suggester.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
version.rb FIX: Handle .discourse-compatibility syntax errors (#10891) 2020-10-12 18:25:06 +02:00
webauthn.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
wizard.rb DEV: Allow plugins to add wizard steps after specific steps (#9315) 2020-04-01 08:36:50 -05:00