Previously we would unconditionally fetch all images via HTTP to grab
original sizing from cooked post processor in 2 different spots.
This was wasteful as we already calculate and cache this info in upload records.
This also simplifies some specs and reduces use of mocks.
The hidden site setting `suppress_secured_categories_from_admin` will
suppress visibility of categories without explicit access from admins
in a few key areas (category drop downs and topic lists)
It is not intended to be a security wall since admins can amend any site
setting. Instead it is feature that allows hiding the categories from the
UI.
Admins will still be able to see topics in categories without explicit
access using direct URLs or flags.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* FIX: allow tl4 to bulk select
- Also allows tl4 to perform batch tagging
---
Long term this needs to be rewritten to account for "bulk action" permission
given from the server.
Co-authored-by: Martin Brennan <martin@discourse.org>
RS256 was added for Windows Hello and as a side effect we speculatively added
RS384 and RS512. These ciphers were not tested and are now failing on solo
keys. It may be the case that the ciphers are not configured correctly on
our side. It may be the case that this is a Solo key bug.
Regardless, we are removing the ciphers and will only consider adding them
again if absolutely needed.
Certain HTML can be rejected by nokogumbo, specifically cases where there
are enormous amounts of attributes
This ensures that malformed HTML is simply skipped instead of leaking out
an exception and terminating downstream processes.
normalized_encode in addressable has a number of issues, including https://github.com/sporkmonger/addressable/issues/472
To temporaily work around those issues for the majority of cases, we try parsing with `::URI`. If that fails (e.g. due to non-ascii characters) then we will fall back to addressable.
Hopefully we can simplify this back to `Addressable::URI.normalized_encode` in the future.
This commit also adds support for unicode domain names and emoji domain names with escape_uri.
This removes an unneeded hack checking for pre-signed urls, which are now handled by the general case due to starting off valid and only being minimally normalized. Previous test case continues to pass.
UrlHelper.s3_presigned_url? which was somewhat wide was removed.
Our internal implementation of #perform on jobs performs remapping.
This happens cause we do "exception aggregation".
Scheduled jobs run on every site in the multisite cluster, and we report
one error per site that failed. During this aggregation we reshape the
context from the original object shape returned by mini_scheduler
The new integration test ensures this interface will remain stable even if
decoupled parts of the code change shapes.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
* FIX: don't memoize site setting in guardian
Memoizing site settings can make tests more fragile and harder to debug
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
* FEATURE: track stats around failing scheduled jobs
Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.
This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
* FIX: bots could generate errors when slug generation method is encoded
When slug generation method is encoded (non default) then bots could
cause errors in the logs for urls containing special chars.
ó for example in a URL can be requested in a valid ASCII-8BIT string, and
later when joined to UTF-8 would result in encoding issues.
Fix here ensures we force encoding correctly for outlier cases.
Browser tend to always encode these chars, hence we did not notice this.
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Certain rogue bots such as Yandex may send across invalid CSP reports
when CSP report collection is enabled.
This ensures that invalid reports will not cause log floods and simply
returns a 422 error.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Due to some changes we started notifying via push notifications on other
families of notifications. There are a total of about 30 or so possible
notification you could get, some can be pushed.
This fallback means that if for any reason we are unable to find an icon
for a push notification we just fallback to the Discourse logo.
Also go with a simple reply icon for watching first post.
Note, that in production `image_url` can return an exception if an image is
missing. This is not the case in test / development.
Previous to this fix, in production we were getting 500 errors from
bootstrap.json cause script_asset_path can not find plugin-tests
Opted for the simplest fix that is omitting the files in production
Previously we limited Discourse Connect provider to 1 secret per domain.
This made it pretty awkward to cycle secrets in environments where config
takes time to propagate
This change allows for the same domain to have multiple secrets
Also fixes internal implementation on DiscourseConnectProvider which was
not thread safe as it leaned on class variables to ferry data around
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: David Taylor <david@taylorhq.com>
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.
Fix also updates logster, we held off on upgrading it due to missing functions
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.
This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.
To resolve this limitation all js assets in about.json are also cached on
local domain.
{
"name": "Header Icons",
"assets" : {
"worker" : "assets/worker.js"
}
}
This can then be referenced in JS via:
settings.theme_uploads_local.worker
local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache
Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.
Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.
This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`
* more usages of upload.content
* add a specific test for upload.content
* Adjust logic to ensure that after upgrades we still get a cached local js
on save
Previous to this change an optimisation stripped crawler content from
all mobile browsers.
This had a side effect that meant that when we dropped support for an old
mobile platform we would stop rendering topic and topic list pages.
The new implementation ensures we only perform the optimisation on modern
mobile browsers.
All users are members of the EVERYONE group, but this group is special and
is omitted from the group_users table. When checking permission we need to
make sure we also add a bypass.
This also fixes a very buggy test in post_alerter, it was confirming the
broken behavior due to fabricator flow.
When it defined the tag group the everyone group automatically had full access
then the additional permission fabricated just added one more group. After
fix was made to code the test started failing. Fabricators can be risky.
Previous to this change if any of the assets were not allowed extensions
they would simply be silently ignored, this could lead to broken themes
that are very hard to debug
* DEV: add testing for multi del on keys
Following #15905 we were missing some tests, this covers cases where
del is used in the form of .del(key1,key2)
We rolled out a change to disable canonical indexing.
The goal behind it was to limit crawl budget by Google being spent
scanning non canonical topic links.
Since this change was applied we rolled out 2 fixes that made the change
no longer needed.
1. Topic RSS feeds are no longer followed, links in the RSS feeds are
not followed.
2. Post RSS feeds now contain canonical links.
Combined these two changes mean crawlers no longer discover a large
amount on non-canonical links on Discourse sites.
* FEATURE: use canonical links in posts.rss feed
Previously we used non canonical links in posts.rss
These links get crawled frequently by crawlers when discovering new
content forcing crawlers to hop to non canonical pages just to end up
visiting canonical pages
This uses up expensive crawl time and adds load on Discourse sites
Old links were of the form:
`https://DOMAIN/t/SLUG/43/21`
New links are of the form
`https://DOMAIN/t/SLUG/43?page=2#post_21`
This also adds a post_id identified element to crawler view that was
missing.
Note, to avoid very expensive N+1 queries required to figure out the
page a post is on during rss generation, we cache that information.
There is a smart "cache breaker" which ensures worst case scenario is
a "page drift" - meaning we would publicize a post is on page 11 when
it is actually on page 10 due to post deletions. Cache holds for up to
12 hours.
Change only impacts public post RSS feeds (`/posts.rss`)
Our @mention user search prioritized users based on prefix matches.
So if searching for `sa` we will display `sam`, `asam` in that order
Previously, we did not prioritize group matches based on prefix. This change ensures better parity.
Implementation notes:
1. User search only prioritizes based on username prefix, not name prefix. TBD if we want to change that.
2. @mention on client side will show 0 group matches if we fill up all the spots with user matches. TBD if we want to unconditionally show the first / second group match.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Previously email validations could fire when deleting posts if for
certain reasons any user validations fail on the user objects
This kind of condition could happen in core due to a corruption of a
user record, or via a plugin that introduces a new validation on User
Previously cached counting made redis calls in main thread and performed
the flush in main thread.
This could lead to pathological states in extreme heavy load.
This refactor reduces load and cleans up the interface
Previously calls such as `Emoji["smile"]` would force a full dehydration of
objects from Redis.
This introduces a version safe site and global emoji cache so lookups are
cheap. It eliminates iterating through the list of emojis and pulling from
redis.
Distributed cache uses a normalized name as the key and stores an Array tuple
with version and Emoji. Successful hits always confirm version matches.
Interface to Emoji object remains unchanged.
We opted for 2 caches to improve reuse on multisites. misses though will be
stored in both caches. If there is a hit on the global cache we can avoid
looking up in site local cache and storing a miss there.
* FEATURE: hide_email_address_taken forces use of email in forgot password form
This strengthens this site setting which is meant to be used to harden sites
that are experiencing abuse on forgot password routes.
Previously we would only deny letting people know if forgot password worked on not
New change also bans usage of username for forgot password when enabled
This column was dropped in a previous commit, in post migrations.
Unfortunatly that causes smoke tests to fail as there is a period between
migration and post migrations where records can not be inserted into the
table.
Old OnceOff job could perform pretty slowly on sites with millions of emails
New implementation operates in batches in a migration, minimizing locking.
The lazy route initialization cuts down boot time of rails.
On my local system it cuts out 200ms of boot time taking me from 3.2 to 3 seconds.
This is not a radically enormous amount of time, but paper cuts add up, and a faster boot in dev will make everyone happy.
TBD if we want to also include this in production.
Gem is heavily maintained by @amatsuda, last commit 3 days ago.
Previously due to "rowheader" role we would read out topic titles twice.
This adjusts it so we apply the heading role only to the topic link.
In turn this makes navigation through topic lists more accurate (h) only
lands you on topic links. It also reduces the amount of duplicate reading
NVDA does.
Before:
Topic title link new topic link support link b481 link 19h link 2 button...
After:
Topic title link
This reduces noise, up and down once you land on a topic link can give you
more context.
Previously we would need to launch unicorn separately this achieves
the same goal by making 2 modifications:
1. If -u is supplied ember-cli binary will launch and monitor ember cli and unicorn
2. We suppress 200 requests to keep console clean (we may consider moving to development rails logs)
Also cleans out output a bit by supplying silent flags to yarn.
Over the years we have found that a few communities never discovered tags.
Instead of having them default off we now have them default on, ensuring
that everyone finds out about them.
Co-authored-by: Dan Ungureanu <dan@ungureanu.me>
Previous to this change we would switch off MessageBus updating after 20
minutes.
This ensures that when the user becomes present again we turn on long polling.
Without long polling updates can be delayed for minutes.
* UX: alert screen readers when there is an issue saving a post
Adds a "alert" role to various popup-input-tips.
This means screen reader users can now tell why a post refuses to save.
Also ensures like icon in the "try the like button" has screen reader support
Previously auto focus would only work on modals that include buttons or
inputs.
To avoid a situation where information modals such as keyboard shortcuts
do not get focus, simply focus on the close button as a fallback.
Previously we would retry push notifications indefinitely for all errors
except for ExpiredSubscription
Under certain conditions other persistent errors may arise such as a persistent
rate limit.
If we track more than 3 errors in a period of time longer than a day we will
delete the subscription
Also performs a bit of internal cleanup to ensure protected methods really
are private.
Previously we had no role set for various topic links, nor did we have any
headers.
This teaches screen readers that topic links in topic lists are to be treated
as H2. We opted for this less radical change cause a change of the element
type would probably result in many broken themes.
Confirmed on NVDA you can very quickly breeze through topic lists now. Minor
edge case is pinned topics which can be a bit annoying due to multiple links.
NVDA does not detect HTML5 articles as regions. This explicitly sets a
region with an aria-label denoting post numbers making it much easier to
know where you are in a topic.
Note role: article which is more semantically correct is not respected by
NVDA d/D shortcut, hence the much more generic "region" role.
Under certain conditions admins would miss messages when posting action in
topics where they have permission.
This also fixes an error where we would sometimes explode when publishing to
an empty group.
Previously watched words ignored topic titles when applying auto tagging rules.
Also copy has been improved to reflect how the system behaves.
The text hints that we are only watching first post now
scopes are incredibly annoying to preload, simply adding :user_emails is not
enough.
Instead of relying on scopes simply iterate through user_emails which is
properly preloaded.
This removes 2 * N+1 when generating user reports.
We do not really support Discourse on Alpine linux / musl based distros.
We depend heavily on libc.
Besides this it looks like rubygems/bundler tends to handle this nuance incorrectly.
Remove it for now.
New version of mini_racer support M1 and ARM builds of Ruby
This should make developing on Discourse easier
Co-authored-by: Robin Ward <robin.ward@gmail.com>
Previously certain images may lead to convert / identify to run for unreasonable
amounts of time
This adds a maximum amount of time these commands can run prior to forcing
them to stop
Previously we used the raw data indexed to generate blurbs even for cases
when Chinese/Korean/Japanese text was used.
This caused superfluous spaces to show up in excerpts.
* DEV: upgrade mini_sql
Even though we are not planning on using this quite yet, mini_sql now supports
prepared statements.
Would like this upgrade merged so we can do some benchmarking.
Note, this will not work with pg_bouncer, but sites that are not using it
may benefit from the feature.
* implement multisite friendly prepared statements
In extreme circumstances when the uploads table is huge, the old version of
this migration could take a very long time.
The rewrite extracts the sha1 directly from the badges table and does an index
based match on the uploads table
Corrects flaky spec
Previously we were only clearing state after our spec ran, leaving possible
edge cases where `Discourse.plugins` had data.
Clean-up source of the plugin leak
1 pop was not enough to clear the plugin, plus make specs a bit more
deliberate
Added a new task to test if indexes are coherent with a blank database
This allows us to detect for cases where somehow indexes are out of sync
FIX_INDEXES=1 or `rake db:validate_indexes[fix]` to correct the issues it finds.
Detects:
- Badly named indexes that need to be renamed
- Missing indexes
- Extra indexes
Can correct all 3 with the fix option
This setting is confusing admins, it is now hidden by default
It only applies to major updates of the rendering engine or imports and
very infrequently needs tweaking
* FIX: optimise MoveNewSinceToTable
Avoids shuffling all ids around to the app (only use min / max)
Ensure the query for boundaries is ordered by user_id
This little helper script allows for easy ember cli development.
To see the options run `bin/ember-cli -h`
It allows you to proxy try.discourse.org with the `bin/ember-cli --try`
switch, which effectively allows for some development without a rails installed.
It passes on arguments to ember-cli so you can customize port and so on.
It makes the assumption that on local people are using `bin/unicorn` for
development. (it includes some extra discourse specific helpers)
These are a few small tweaks that slightly improve performance.
- we omitted 1 query from the post guardian which could cause an N+1
- cook_url has been sped up a bit
- url helper avoids re-creating sets for no reason
Inventory on S3 always lagged, over the past few weeks we are noticing that
1 day of lag is not enough.
We are increasing this to 2, to ensure that we do not get false positive
reports.
These 2 indexes optimise performance on profile pages.
The summary page displays:
1. A list of "Top Link" - links sorted by number of clicks posted by user
2. A list of "Top Replies" - replies made by a user that go the most hearts
These two areas could devolve into full index or table scans, new indexes are there to avoid this cost on large dbs
One minor downside is that storage requirements go a tiny bit up to maintain the new indexes
see: https://bundler.io/blog/2020/12/09/bundler-v2-2.html
New version of bundler includes explicit callouts for each platform supported
This means we need to bundle on all our supported platforms, the number should be fairly low as x86 linux covers most
Final sigma is not lower cased correctly in Ruby causing issues with routing.
This works around the issue by downcasing all usernames containing a sigma using JS.
Googlebot handles no-index headers very elegantly. It advises to leave as many routes as possible open and uses headers for high fidelity rules regarding indexes.
Discourse adds special `x-robot-tags` noindex headers to users, badges, groups, search and tag routes.
Following up on b52143feff we now have it so Googlebot gets special handling.
Rest of the crawlers get a far more aggressive disallow list to protect against excessive crawling.
You can now use `@me` to search for posts created by yourself, this is particularly handy if you have a long username.
`@me rainbow` will find all posts you created with the word rainbow.
Also cleans up test suite so it has no warnings.
The notification panel gets resized and the JS uses maxWidth of 320.
This tends to fight with the CSS causing notifications to "jump" a bit when a new one lands.
chromium may report float device pixel ratio below 1.5 that is still clearly retina:
```
window.devicePixelRatio
1.4999998807907104
```
We used to round this down to 1 and not provide these browsers with retina avatars.
New algorithm is much more forgiving, anything over 1.1 gets 2x images, anything over 2.1 gets 3x images.
Rapid concurrent SSO attempts is something that happens quite frequently
in the wild at large enough scale.
When this happens conditions such as adding a user to a group could possibly
fire concurrently causing a user to be added to the same group twice and
erroring out.
To avoid all concurrency issues here we protect with a coarse distributed
mutex. This heavily mitigates the risk around concurrent group additions and
concurrent updates to user related records.
disable_search_queue_threshold needs to be coerced to a float so it is not
treated as a string when sub second values are provided.
Longer term fix is to possibly provide hints in the config so we do the
coersion automatically. However this would be a far more complex change.
Previous to this change our anonymous rate limits acted as a throttle.
New implementation means we now also consider rate limited requests towards
the limit.
This means that if an anonymous user is hammering the server it will not be
able to get any requests through until it subsides with traffic.
Discourse used to break from convention by logging out all sessions on any
specific session logout.
This would leave users confused about why mobile is logged out when the user
logged out of desktop.
log_out_strict is too conservative for most and not the pattern the industry
has adopted (google/twitter/facebook all perform no strict logouts)
Sam
