Alan Guo Xiang Tan 39da9106ba DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811) ... What motivated this change? Our builds on Github actions have been extremely flaky mostly due to system tests. This has led to a drop in confidence in our test suite where our developers tend to assume that a failed job is due to a flaky system test. As a result, we have had occurrences where changes that resulted in legitimate test failures are merged into the `main` branch because developers assumed it was a flaky test. What does this change do? This change seeks to reduce the flakiness of our builds on Github Actions by automatically re-running RSpec tests once when they fail. If a failed test passes subsequently in the re-run, we mark the test as flaky by logging it into a file on disk which is then uploaded as an artifact of the Github workflow run. We understand that automatically re-runs will lead to lower accuracy of our tests but we accept this as an acceptable trade-off since a fragile build has a much greater impact on our developers' time. Internally, the Discourse development team will be running a service to fetch the flaky tests which have been logged for internal monitoring. How is the change implemented? 1. A `--retry-and-log-flaky-tests` CLI flag is added to the `bin/turbo_rspec` CLI which will then initialize `TurboTests::Runner` with the `retry_and_log_flaky_tests` kwarg set to `true`. 2. When the `retry_and_log_flaky_tests` kwarg is set to `true` for `TurboTests::Runner`, we will register an additional formatter `Flaky::FailuresLoggerFormatter` to the `TurboTests::Reporter` in the `TurboTests::Runner#run` method. The `Flaky::FailuresLoggerFormatter` has a simple job of logging all failed examples to a file on disk when running all the tests. The details of the failed example which are logged can be found in `TurboTests::Flaky::FailedExample.to_h`. 3. Once all the tests have been run once, we check the result for any failed examples and if there are, we read the file on disk to fetch the `location_rerun_location` of the failed examples which is then used to run the tests in a new RSpec process. In the rerun, we configure a `TurboTests::Flaky::FlakyDetectorFormatter` with RSpec which removes all failed examples from the log file on disk since those examples are not flaky tests. Note that if there are too many failed examples on the first run, we will deem the failures to likely not be due to flaky tests and not re-run the test failures. As of writing, the threshold of failed examples is set to 10. If there are more than 10 failed examples, we will not re-run the failures.		2023-12-13 07:18:27 +08:00
..
auth	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
backup_restore	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
common_passwords	DEV: Fix various spec linting issues (#24672)	2023-12-04 13:45:19 +01:00
compression	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
concern	FIX: Allow setting an array custom field to a singleton value (#24636)	2023-11-29 14:18:47 -06:00
content_security_policy	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
discourse_webauthn	DEV: Update webauthn authentication documentation (#23787)	2023-10-05 15:22:43 -04:00
email	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
file_store	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
final_destination	DEV: Update the rubocop-discourse gem	2023-06-26 11:41:52 +02:00
freedom_patches	DEV: Update TranslateAccelerator missing translation string (#22158)	2023-06-16 15:28:03 +01:00
guardian	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
highlight_js	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
i18n	DEV: Update I18n to 1.13.0 (#21685)	2023-05-24 08:59:37 +08:00
imap	DEV: Update the rubocop-discourse gem	2023-06-26 11:41:52 +02:00
import	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
middleware	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
migration	DEV: Allow DROP NOT NULL in pre-deploy migrations (#20775)	2023-03-22 14:43:32 +00:00
onebox	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
plugin	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
pretty_text	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
rate_limiter	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
scheduler	DEV: Prevent defer stats exception when thread aborted (#19863)	2023-01-16 09:08:44 +11:00
second_factor	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
seed_data	FIX: Use category hashtag instead of link in discourse_welcome_topic.body (#22875)	2023-08-01 13:53:23 +02:00
site_settings	FIX: min/max not passed to NumberField for site settings (#24402)	2023-11-16 12:37:05 +10:00
stylesheet	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
summarization	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
svg_sprite	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
theme_store	FEATURE: Theme settings migrations (#24071)	2023-11-02 08:10:15 +03:00
topic_query	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
turbo_tests/flaky	DEV: Introduce automatic reruns to RSpec tests on Github actions (#24811)	2023-12-13 07:18:27 +08:00
validators	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
wizard	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
admin_confirmation_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
admin_user_index_query_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
archetype_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
bookmark_manager_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
bookmark_query_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
bookmark_reminder_notification_handler_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
browser_detection_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
cache_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
category_badge_spec.rb	UX: Restore category badge colours on 404 page (#24754)	2023-12-06 17:49:19 +00:00
category_guardian_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
color_math_spec.rb	UX: Calculate missing hover/selected colors from existing colors (#20105)	2023-02-01 09:55:21 +00:00
composer_messages_finder_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
content_buffer_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
content_security_policy_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
cooked_post_processor_spec.rb	DEV: remove duplicate spec (#24691)	2023-12-04 15:45:11 +00:00
crawler_detection_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
current_user_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
db_helper_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
directory_helper_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
discourse_diff_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
discourse_event_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
discourse_hub_spec.rb	DEV: Ability to collect stats without exposing them via API (#23933)	2023-11-10 00:44:05 +04:00
discourse_js_processor_spec.rb	DEV: Use esbuild to make DiscourseJsProcessor (#23223)	2023-08-24 16:36:22 +02:00
discourse_plugin_registry_spec.rb	FEATURE: Add hooks for email poller plugins (#21384)	2023-06-26 13:16:03 +08:00
discourse_redis_spec.rb	DEV: Fix various spec linting issues (#24672)	2023-12-04 13:45:19 +01:00
discourse_sourcemapping_url_processor_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
discourse_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
discourse_tagging_spec.rb	FIX: Remove multiple consecutive dashes from tag names (#24520)	2023-11-23 00:57:12 +01:00
discourse_updates_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
distributed_cache_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
distributed_memoizer_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
distributed_mutex_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
email_cook_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
email_updater_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
ember_cli_spec.rb	FIX: EmberCli cache clearance issue in production (#24343)	2023-11-13 10:34:06 +00:00
encodings_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
enum_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
excerpt_parser_spec.rb	FIX: Internal oneboxes with github links (#22829)	2023-07-28 09:07:53 -06:00
feed_element_installer_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
feed_item_accessor_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
file_helper_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
filter_best_posts_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
final_destination_spec.rb	FIX: Don't error out when trying to retrieve title and URL won't encode (#24660)	2023-12-01 15:03:06 +08:00
flag_settings_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
gaps_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
gc_stat_instrumenter_spec.rb	DEV: Seperate concerns of tracking GC stat from MethodProfiler (#22921)	2023-08-02 10:46:37 +08:00
git_repo_spec.rb	FEATURE: display commit hash for each plugin on /admin/plugins page. (#22176)	2023-06-26 10:09:57 +05:30
git_url_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
global_path_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
group_email_credentials_check_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
group_lookup_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
guardian_spec.rb	DEV: Convert min_trust_to_edit_wiki_post to groups (#24766)	2023-12-12 15:20:37 +11:00
has_errors_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
hijack_spec.rb	FIX: Ensure app-cdn CORS is not overridden by cors_origin setting (#24661)	2023-12-01 12:57:11 +00:00
html_prettify_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
html_to_markdown_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
image_sizer_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
inline_oneboxer_spec.rb	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
js_locale_helper_spec.rb	DEV: Update js_locale_helper_spec following 76b75fae	2023-11-06 18:03:32 +00:00
json_error_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
letter_avatar_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
message_id_service_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
method_profiler_spec.rb	DEV: Seperate concerns of tracking GC stat from MethodProfiler (#22921)	2023-08-02 10:46:37 +08:00
mini_sql_multisite_connection_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
mobile_detection_spec.rb	Drop support for iOS < 15.7 (#19847)	2023-01-16 17:28:59 +00:00
new_post_manager_spec.rb	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
new_post_result_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
onebox_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
oneboxer_spec.rb	DEV: Revert guardian changes (#24742)	2023-12-06 16:37:32 +10:00
onpdiff_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
password_hasher_spec.rb	FEATURE: Persist password hashing algorithm/params in database (#20980)	2023-04-11 10:16:28 +01:00
pbkdf2_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
pinned_check_spec.rb	DEV: Fix various spec linting issues (#24672)	2023-12-04 13:45:19 +01:00
plain_text_to_markdown_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
post_action_creator_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
post_action_destroyer_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
post_creator_spec.rb	Apply embed unlisted setting consistently (#24294)	2023-12-12 09:35:26 -05:00
post_destroyer_spec.rb	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
post_jobs_enqueuer_spec.rb	DEV: Update the rubocop-discourse gem	2023-06-26 11:41:52 +02:00
post_locker_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
post_merger_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
post_revisor_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
presence_channel_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
pretty_text_spec.rb	DEV: Modernise highlightjs loading (#24197)	2023-11-10 20:39:48 +00:00
promotion_spec.rb	FIX: Do not validate email in TL promotion (#20892)	2023-03-30 13:52:10 +10:00
quote_comparer_spec.rb	DEV: Correct Style/RedundantReturn rubocop issues (#23052)	2023-08-10 02:03:38 +02:00
quote_rewriter_spec.rb	DEV: Update display name in new quote format - Part 2 (#22104)	2023-06-26 11:01:59 +08:00
rate_limiter_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
retrieve_title_spec.rb	FIX: Don't error out when trying to retrieve title and URL won't encode (#24660)	2023-12-01 15:03:06 +08:00
rtl_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
s3_cors_rulesets_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
s3_helper_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
s3_inventory_multisite_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
s3_inventory_spec.rb	DEV: update syntax tree to latest (#24623)	2023-11-29 16:38:07 +11:00
score_calculator_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
scss_checker_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
search_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
secure_session_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
shrink_uploaded_image_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
sidebar_section_links_updater_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
site_icon_manager_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
site_setting_extension_multisite_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
site_setting_extension_spec.rb	DEV: Remove flaky setting deprecation logging specs (#24322)	2023-11-10 10:46:28 +10:00
slug_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
spam_handler_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
suggested_topics_builder_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
system_message_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
text_cleaner_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
text_sentinel_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
theme_flag_modifier_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
theme_javascript_compiler_spec.rb	DEV: Add gjs support for themes (#23473)	2023-10-02 12:36:06 +02:00
theme_settings_manager_spec.rb	FIX: Use the default value correctly for theme settings of type uploads (#20541)	2023-03-06 11:41:47 +03:00
theme_settings_parser_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
timeline_lookup_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
tiny_japanese_segmenter_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
topic_creator_spec.rb	Apply embed unlisted setting consistently (#24294)	2023-12-12 09:35:26 -05:00
topic_publisher_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
topic_query_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
topic_retriever_spec.rb	FEATURE: Update topic/comment embedding parameters (#20181)	2023-02-28 14:31:59 +02:00
topic_upload_security_manager_spec.rb	DEV: Update the rubocop-discourse gem	2023-06-26 11:41:52 +02:00
topic_view_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
topics_bulk_action_spec.rb	DEV: Fix various rubocop lints (#24749)	2023-12-06 23:25:00 +01:00
topics_filter_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
trashable_spec.rb	DEV: Added .only_deleted scope in the Trashable module (#20196)	2023-02-07 15:28:59 -03:00
truncate_logs_formatter_spec.rb	SECURITY: Add a default limit as to when logs should be truncated	2023-10-16 10:34:38 -04:00
trust_level_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
twitter_api_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
unread_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
upload_creator_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
upload_markdown_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
upload_recovery_spec.rb	DEV: Allow fab! without block (#24314)	2023-11-09 16:47:59 -06:00
upload_security_spec.rb	DEV: Fix various spec linting issues (#24672)	2023-12-04 13:45:19 +01:00
url_helper_spec.rb	FIX: Wrong argument error being thrown in UrlHelper (#24506)	2023-11-22 15:19:40 +10:00
user_comm_screener_spec.rb	DEV: Update the rubocop-discourse gem	2023-06-26 11:41:52 +02:00
user_lookup_spec.rb	DEV: Apply syntax_tree formatting to spec/*	2023-01-09 11:49:28 +00:00
user_name_suggester_spec.rb	DEV: Fix various spec linting issues (#24672)	2023-12-04 13:45:19 +01:00
version_spec.rb	SECURITY: Add limits for themes and theme assets	2023-09-12 15:31:31 -03:00
work_queue_spec.rb	SECURITY: Don't allow a particular site to monopolize the defer queue	2023-07-28 12:53:51 +01:00