Very large batches can take an enormous amount of time due to churn
Limiting to 200k changes at a time gives us a far larger chance of finishing
the job without timing out or deadlocking.
A giant transaction in a post migration can be very risky.
This splits the large amount of work this migration needs to do into 2 parts:
1. A re-runnable cleanup job prior to transaction
2. A minimally sized transaction to add the database constraint
This avoids large amounts of churn on the table
There is an fk to user_profile that can make destroying uploads fail
if they happen to be set as user profile.
This ensures we clear this information when destroying uploads.
There are more relationships, but this makes some more progress.
`rake uploads:analyze_missing` can be used get rich information regarding
uploads missing from s3 (where verified is false)
`rake uploads:fix_missing` is a work in progress task for automatically
correcting certain historic issues. At the moment it simply rebakes all
posts with missing uploads, but it will improve over time
This stops sync of tracking state when list is filtered, in the past this
would cause the tracking state to go off wack.
Additionally this introduces an alias for "filter=tracking", called "f=tracking"
This was done cause the term "filter" is used internally in 2 different ways
the main way is for /unread /new filtering.
Trying to also call a query param "filter" causes enormous amounts of
internal pain, this circumvents the issue.
`topicTrackingState.forEachTracked(topic,isNew,isUnread)` can be used to
iterate through tracking state efficiently.
This is handy for extension looking at subsets of tags and categories.
After visiting a topic list (by tag / category / top level) we track the list
Once a list is tracked the combo `g` `j` can be used to go to the next topic
in the list and `g` `k` to go to previous topic.
This allows you to quickly work through subsets of topics without having
to navigate back to the top level lists
The shortcut does not work in PM lists yet, or search results, both are
under consideration.
This code was intended to bypass iterating through tags in absence of a new
or unread topic.
Instead it always fired cause it was checking for function existence which
was clearly always true.
This pushes v8 from Chrome 73 (March 2019) -> 84 (July 14 2020)
Not expecting any user facing changes, but it is super nice to be on latest
v8 :confetti:
This adds a special filter to topic lists that will filter to tracked and
watched categories.
To use it you can visit:
`https://sitename/?filter=tracked`
`https://sitename/unread?filter=tracked`
and so on
Note, we do not include explicitly tracked and watched topics **outside** of
the tracked categories and tags.
We can consider a `filter=all_tracked` to cover this edge case.
Not ready for an upgrade due to: https://github.com/mdp/rotp/issues/98
The policy here is that for cases like this we pin the version and add
a comment explaining why it is pinned.
We can revisit in a few months depending on upstream.
This is very minor, see: https://github.com/advisories/GHSA-j6w9-fv6q-3q52
An attacker can elevate own cookie usage to bypass server cookie restrictions
Technically this is a security commit, but the surface area is extremely
low, we do not expect any real world impact.
This includes a fix for CVE-2020-8185 we are not vulnerable as we do not use
the impacted middleware. However it still makes sense to stay upgraded, other
small fixes exist in this release.
Previously we would include this section, unfortunately
1. It is usually elided in gmail
2. It can make the emails longer and more confusing
3. Omission is a feature, it means people need to visit site to get context
08044b4f regressed emoji auto complete logic since we (I) forgot to add the
space into the not capturing group at the beginning.
This meant that
`hello :+1:t` would not trigger an autocomplete to pick skin tone.
autocomplete resolving to [] was causing it to stop working.
Instead we have a special const (SKIP) which ensures it will
continue to be evaluated and only this instance is skipped.
There is a feature in search where we take over from the tokenizer
in postgres and attempt to inject more words into search.
So for example: sam.i.am will inject the words i and am.
This is not ideal cause there are many edge cases and this can
cause extreme index bloat.
This is an opening move commit to make it configurable, over the
next few weeks we will evaluate and decide if we disable this by
default or simply remove.
The logic of adding additional search results does not seem to be
needed anymore.
It appears to be a relic of an old implementation.
This saves an entire search query for every search made.
#b19dcac2 improved the serializer so it sends default notification
levels to users to work around cases where a category edit would
would result in clients being left with invalid notification state
Unfortunately this did not address the root issue.
When we edit categories we publish state to multiple users this
means that the serializer is executed unscoped with no user.
The client already handles this case per:
https://github.com/discourse/discourse/blob/dcad720a4ce0aa67e94299ff8370fafd20cc41b2/app/assets/javascripts/discourse/app/models/site.js#L119-L119
If a property is not shipped to it, it will leave it alone on the
existing category.
This fix ensures that these wide category info updates do not
include notification state to avoid corruption of local state.
If people wish to opt out they can use `DISABLE_BOOTSNAP = 1`
Bootsnap is production ready and was tested on our production
servers for safety.
Promoting it now so it is enabled globally.
Will result in faster application boot
This test lasted about 7 years prior to it becoming flaky.
Today ... for whatever reason the test suite created 100 users
prior to running this spec. So the new user becomes user id 101
And... lo and behold:
```
1) PostSerializer a hidden post with add_raw enabled a hidden post shows the raw post only if authorized to see it
Failure/Error: expect(serialized_post_for_user(Fabricate(:user))[:raw]).to eq(nil)
expected: nil
got: "Raw contents of the post."
(compared using ==)
# ./spec/serializers/post_serializer_spec.rb:127:in `block (4 levels) in <main>'
```
Previous to this fix the we were checking for non letters.
This was mismatching what pretty-text/addon/emoji.js was doing.
`ù:su`
and
`1:su`
Would lead to an emoji autocomplete popup in the composer.
Bootsnap is designed to work in production per:
https://github.com/Shopify/bootsnap
Over the years we have seen very few issues with it, none of which
were corruption.
This allows us to enable bootsnap in production
Having bootsnap enabled means that we can speed up deploys by
about 5 seconds per server. It also means a lot less waiting for
various production rake tasks and so on.
In some restricted setups all JS payloads need tight control.
This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.
There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.
Use an example like this to enable:
`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`
By default this feature is not enabled and no changes are made.
One exception is that default theme id was missing a security check
this was added for correctness.
This introduces a new core API to get counts per tag from topic
tracking state
This API will only be useful if a plugin enable tags in topic
tracking state using
`TopicTrackingState.include_tags_in_report = true`
We previously did not account for completely untagged topics when
looking at muted tags, this caused new/unread counts to be off if
1. You had muted tags
2. You had an unread/new topic
3. This topic had no tags
This feature allows certain plugins to output tag information
to topic tracking state, this allows per tag stats which can be
used by sidebars and other plugins that need per tag stats
Not enabled by default cause this would add cost to a critical
query
This reverts commit 75f46ca632.
This unfortunately breaks
`RAILS_ENV=production bin/rake assets:precompile`
I don't think discourse-loader.js is transpiled
There was a race condition where drafts could be either saving
or queued to be saved and a user canceled draft leading to destroying
it.
This cancels debounce save and waits for save in the pipeline to
be over prior to firing off a DELETE on the draft
cancelComposer would leak a promise that never got resolved if
you aborted cancelling a composer.
This change ensured the promise will always be resolved
Previously we had a partial fix in place where non human users
were not allowed draft sequences, this left edges around where non
human users asked for drafts yet had none.
For example system could already have a few drafts in place.
This also removes and extensibility point we added that is not in use
This commit reorganises the delete dependencies on users and make sure
all are covered. We forgot some on bookmarks, security keys, anon users
and so on.
PG already handles English stop words, the list in cppjieba is
bigger than the list PG uses, which in turn causes confusion cause
words such as "volume" are stripped using cppijieba stop word list
We will follow up with another commit here to apply the Chinese
word stopwords, but for now to eliminate the confusion we are
skipping applying the stopword list when the dictionary in PG is
in English.
Prior to this change we would never clear memory from contexts and
rely on V8 reacting to pressure
This could lead to bloating of PrettyText and Transpiler contexts
This optimisations ensures that we will clear memory 2 seconds after
the last eval on the context
Previously we would raise a warning in the logs if downloading
a file (from s3) takes longer than 60 seconds.
At scale this happens reasonably frequently.
1. Raised the duration to 3 minutes
2. Pulled the resizing mutex out of the downloading mutex
so we have less and clearer error logs
Previously if saving a draft took longer than 2 seconds there
could be conditions where drafts could be saved concurrently.
This meant the composer could race with itself and raise conflicts.
This is likely to happen on bad internet connections or where
latency is really high.
Additionally a throttle was added so drafts save unconditionally
every 15 seconds.
Save draft in the model now properly and consistently returns a
promise.
Previously the code was very race condition prone leading to
odd failures in production
It was re-written in raw SQL to avoid conditions where rows
conflict on inserts
There is no clean way in ActiveRecord to do:
Insert, on conflict do nothing and return existing id.
This also increases test coverage, we were previously not testing
the code responsible for crawling external sites directly
Previously we only changed sequence on ownership change, this
cause a race condition between tabs where user could type for a
long time without being warned of an out of date draft.
This change is a radical change and we should watch closely.
Code was already in place to track sequence on the client so no
changes are needed there.
posts and topics tables can be huge, this migration change
ensures the migration performs less work in one giant transaction
reducing risk of deadlocks
Migration is re-runnable thanks to NOT EXISTS checks
In production `title` was set to undefined causing a
discrepancy between originalTitle and title
This attempts to work around the issue in the production bundle
Google insists on indexing pages so it can figure out if they
can be removed from the index.
see: https://support.google.com/webmasters/answer/6332384?hl=en
This change ensures the we have special behavior for Googlebot
where we allow indexing, but block the actual indexing via
X-Robots-Tag
Expand SiteSetting.allow_index_in_robots_txt so it also adds a
noindex header if set to false.
This makes sure that nothing is indexed even if it somehow reaches
Google.
Previously originalTitle was set to undefined leading to
titleDirty being unconditionally undefined for new topics
That mean you could not dismiss the composer via ESC
1. Total 6 attempts per day per user
2. Total of 5 per unique email/login that is not found per hour
3. If an admin blocks an IP that IP can not request a reset
Upgrades Rails to latest, this version has better compatibility
with Ruby 2.7
During the upgrade we needed a new cleaner mechanism for configuring
message bus.
All tests are green.
If anything weird pops up please revert.
The failover spec is very fragile and tests specific implementation
vs actual behavior
We rely on a different script during the build process to test
failover operates correctly
This ensures that at a minimum you are notified once a day of
repeat edits by the same user.
Long term we may consider winding this down to say 1 hour or
making it configurable.
Due to a refactor in e90f9e5cc4 we stopped notifying on edits if
a user liked a post and then edited.
The like could have happened a long time ago so this gets extra
confusing.
This change makes the suppression more deliberate. We only want
to suppress quite/link/mention if the user already got a reply
notification.
We can expand this suppression if it is not enough.
rspec-rails 4.0 was released so we no longer need to depend on a
beta version. Also updates minor on a bunch of rspec gems.
Thanks to @ryanwi for raising this.
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.
For files with `# frozen_string_literal: true`
```
puts %w{a b}[0].frozen?
=> true
puts "hi".frozen?
=> true
puts "a #{1} b".frozen?
=> true
puts ("a " + "b").frozen?
=> false
puts (-("a " + "b")).frozen?
=> true
```
For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
We need to recreate the badge_posts view each time we change
column on the posts table.
The p.* is auto expanded on view create, cascade should not have
been used
Sam Saffron