Commit Graph

7512 Commits

Author SHA1 Message Date
Sam Saffron
2a7490149c DEV: don't fail if in uploads:fix_missing_s3 when fix fails
Previously a single error on a file like invalid extension could fail the
entire rake task
2020-08-18 17:55:49 +10:00
Guo Xiang Tan
92b7fe4c62 PERF: Add partial index for non-pm search. 2020-08-18 15:55:08 +08:00
David Taylor
33c21c5921 DEV: Skip readonly tables when remapping content (#10461) 2020-08-18 08:53:12 +01:00
Sam Saffron
24fe08230f FEATURE: ensure posts are rebaked when missing is fixed
This ensures any corrupt optimized images are removed and re-created
2020-08-18 15:37:24 +10:00
David Taylor
645b65adfa FIX: Add playsinline to twitter GIFs 2020-08-18 00:26:51 +01:00
David Taylor
eb981b8fbb FEATURE: Autoplay oneboxed twitter GIF media 2020-08-17 23:56:41 +01:00
David Taylor
a895e4a15b FIX: Skip rendering twitter video if matching format not found
Followup to d5a4318a
2020-08-17 20:53:14 +01:00
David Taylor
d5a4318ac1 FIX: Replace iframe with <video> for twitter videos
This avoids the samesite cookie related error on chrome. It also adds support for twitter 'GIF' content, and allows videos to resize smoothly for narrow devices.
2020-08-17 20:37:36 +01:00
jahan-ggn
65649eaef0 User card settings (#10302)
* settings implemented

* prettier

* settings updated

* rubocop

* prettier

* Revert "rubocop"

This reverts commit 7805145a7d.

* Revert "prettier"

This reverts commit 2c53f4fa12.

* settings updated and changed

* rubocop

* changes applied

* final changes done

* Server side feature added

* spec changed

* changed user_updater and profile file

* Fix user card specs

* web hook serializer solved

* site-setting changed

Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
2020-08-17 12:37:45 -04:00
David Taylor
0a5376084d FIX: Ensure auto close notice is posted with system locale
Previously it was created with the locale of the user who created the final post in the topic
2020-08-17 15:40:47 +01:00
Guo Xiang Tan
248bebb8cd PERF: Remove extra subquery in search.
I also noticed that removing the subquery helps the planner to plan
better.
2020-08-17 13:52:12 +08:00
David Taylor
8ac85f54fb DEV: Remove blob: workers from CSP (#10440)
Ace editor is reconfigured to load workers directly from their JS URL. Workers must be on the same origin as the site, so they will not use the CDN.
2020-08-14 18:15:30 +01:00
Vinoth Kannan
476d26159a FEATURE: add new user option skip_new_user_tips. (#10437)
And add new site setting `default_other_skip_new_user_tips` in user preferences category.
2020-08-14 19:10:56 +05:30
David Taylor
3cf93e9a8f Revert "DEV: Disable ACE editor worker blobs"
This reverts commit d5463d2a4d.

With S3 assets, CORS prevents loading worker assets directly.
2020-08-14 14:12:11 +01:00
David Taylor
a5608025aa FIX: Add script asset locations to worker-src CSP directives
We no longer need :blob worker src since d5463d2a. But we do want to allow workers to be loaded from all our existing script-src options.
2020-08-14 12:31:50 +01:00
Martin Brennan
4670b62969 DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Penar Musaraj
eae8b0465c DEV: Upgrade ACE Editor to 1.4.12, remove extra ACE files (#10431) 2020-08-13 21:53:27 -04:00
Martin Brennan
ffb31b8d2b FIX: Do not require tagging to be enabled for IMAP archive and delete (#10426)
Previously we did an early return if either SiteSetting.tagging_enabled or SiteSetting.allow_staff_to_tag_pms was false when updating the email on the IMAP server -- however this also stopped us from archiving or deleting emails if either of these were disabled.
2020-08-13 14:04:40 +10:00
Sam Saffron
5be26c7d24 DEV: add error handling in case download fails 2020-08-13 13:48:23 +10:00
Jeff Wong
787ad7d84d FIX: allow plugin pinning to fetch missing commits
Add update for fetching git commits if they do not exist, eg with
clone --depth 1 - only can fetch via git fetch --depth 1 {remote} {ref}
the ref needs to be a full, non-ambiguous reference.
2020-08-12 20:03:37 -07:00
Sam Saffron
5011435ec7 DEV: do not correct sha when correctly uploads 2020-08-13 11:52:57 +10:00
Sam Saffron
f48fa30ecd DEV: fix_missing_s3 attempts to re-download if missing
unverified uploads get re-downloaded and corrected if they exist in the task
2020-08-13 11:22:14 +10:00
David Taylor
bd0a7553c4 DEV: Detect when s3 inventory failure is caused by etag difference (#10427) 2020-08-13 09:30:28 +10:00
David Taylor
451b9b245f DEV: Rename new upload rake tasks
These tasks are s3-specific, so update the names to make that very clear
2020-08-12 23:26:13 +01:00
Michael Brown
c25dc43f54 FIX: AWS S3 errors don't necessarily include a message
* If the error doesn't have a message, the class name will help
* example:
  before: "Failed to download #{filename} because "
  after: "Failed to download #{filename} because Aws::S3::Errors::NotFound"
2020-08-12 17:00:09 -04:00
Rafael dos Santos Silva
28669dfeb2 PERF: Faster TL3 promotion replies needed calculation (#10416)
Removing the LIMIT makes PostgreSQL use index_posts_on_user_id_and_created_at
which is much faster overall.

Before: 22 seconds
After: 100 ms
2020-08-12 11:28:34 -03:00
Penar Musaraj
6dd9f2eca2 FIX: color scheme selection with non-default theme
This fixes an issue where a non-default theme set to use the base color
scheme (i.e. the theme had an empty `color_scheme_id`) was loading the
default theme's color scheme instead.
2020-08-12 08:49:13 -04:00
Guo Xiang Tan
93f8396b4b FIX: Limit PG headline based search blurb generation to 200 characters.
* Recovers omission characters '...' in blurb as well.
2020-08-12 15:34:27 +08:00
Sam Saffron
ec173a72d9 DEV: add more counts to analyze missing 2020-08-12 17:28:58 +10:00
Sam Saffron
97fe68e980 DEV: look for avatars when analyzing missing uploads 2020-08-12 14:04:21 +10:00
Sam Saffron
990cd8c2e4 FEATURE: introduce tasks for dealing with legacy broken uploads
`rake uploads:analyze_missing` can be used get rich information regarding
uploads missing from s3 (where verified is false)

`rake uploads:fix_missing` is a work in progress task for automatically
correcting certain historic issues. At the moment it simply rebakes all
posts with missing uploads, but it will improve over time
2020-08-12 13:33:04 +10:00
Martin Brennan
95b71b35d6 FEATURE: IMAP delete email sync for group inboxes (#10392)
Adds functionality to reflect topic delete in Discourse to IMAP inbox (Gmail only for now) and reflecting Gmail deletes in Discourse.

Adding lots of tests, various refactors and code improvements.

When Discourse topic is destroyed in PostDestroyer mark the topic incoming email as imap_sync: true, and do the opposite when post is recovered.
2020-08-12 10:16:26 +10:00
Penar Musaraj
c05aced094 FIX: Invalidate cache when updating color scheme colors (#10417) 2020-08-11 16:28:59 -04:00
Martin Brennan
b950b3fb3f DEV: Add verified to uploads and fill in S3 inventory (#10406)
When we run the S3 inventory, mark uploads that exist as verified true, those that don't as verified false, and uploads not included in the check / not yet checked as verified nil.
2020-08-11 14:43:51 +10:00
Blake Erickson
2032c11f78 FIX: Return 422 when creating topics with tags w/out permission (#10400)
The UI prevents users from trying to create tags on topics when they
don't have permission, but if you are trying to add tags to a topic via
the API and you don't have permission before this change it would
silently succeed in creating the topic, but it wouldn't have any tags.

Now a 422 error will be returned with an error message when trying to
create a topic with tags when tagging is disabled or you don't have
enough trust level to add tags to a topic.

Bug report: https://meta.discourse.org/t/-/70525/14
2020-08-10 16:14:15 -06:00
jbrw
d67f7a7984 Allows mod posts to be created for category group moderators on closed/archived topics (#10399) 2020-08-10 15:21:01 -04:00
David Taylor
4476680d4b UX: Set silence_reason using the system locale
Previously we would use the user's locale, which can be confusing for moderators.
2020-08-10 18:51:54 +01:00
Penar Musaraj
f179510a68 FIX: include both name and id in color scheme stylesheet filename slugs (#10397) 2020-08-07 13:43:45 -04:00
jbrw
1399e87cdc Add recently renamed category group moderation setting to DeprecatedSettings 2020-08-07 13:21:25 -04:00
Régis Hanol
bc63232d2e FIX: sync reviewable count when opening the hamburger menu (#10368)
When a tab is open but left unattended for a while, the red, green, and blue
pills tend to go out of sync.

So whevener we open the notifications menu, we sync up the notification count
(eg. blue and green pills) with the server.

However, the reviewable count (eg. the red pill) is not a notification and
is located in the hamburger menu. This commit adds a new route on the server
side to retrieve the reviewable count for the current user and a ping
(refreshReviewableCount) from the client side to sync the reviewable count
whenever they open the hamburger menu.

REFACTOR: I also refactored the hamburger-menu widget code to prevent repetitive uses
of "this.".

PERF: I improved the performance of the 'notify_reviewable' job by doing only 1 query
to the database to retrieve all the pending reviewables and then tallying based on the
various rights.
2020-08-07 18:13:02 +02:00
jbrw
3593e582a3 FIX - limit number of embedded media items in a post (#10391)
* FIX - limit number of embedded media items in a post

* Add renamed settings to DeprecatedSettings
2020-08-07 12:08:59 -04:00
Daniel Waterworth
abebc4e05e DEV: Use verbose mode when running tests in CI 2020-08-07 17:04:55 +01:00
Daniel Waterworth
e55578681e DEV: Add seed explicitly in turbo_rspec so that runs can be reproduced 2020-08-07 16:51:53 +01:00
Penar Musaraj
86cb5803ba Convert HEIC uploads to JPG by default (#10395) 2020-08-07 11:17:50 -04:00
Régis Hanol
c5c5bb4fad DEV: prevent already initialized constant warning 2020-08-07 16:28:43 +02:00
Guo Xiang Tan
053cbe3112 PERF: Limit characters used to generate headline for search blurb.
We determined using the following benchmark script that limiting to 2500 chars would mean a maximum of
25ms spent generating headlines.

```
require 'benchmark/ips'

string = <<~STRING
Far far away, behind the word mountains...
STRING

def sql_excerpt(string, l = 1000000)
  DB.query_single(<<~SQL)
  SELECT TS_HEADLINE('english', left('#{string}', #{l}), PLAINTO_TSQUERY('mountains'))
  SQL
end

def ruby_excerpt(string)
  output = DB.query_single("SELECT '#{string}'")[0]
  Search::GroupedSearchResults::TextHelper.excerpt(output, 'mountains', radius: 100)
end

puts "Ruby Excerpt: #{ruby_excerpt(string)}"
puts "SQL Excerpt: #{sql_excerpt(string)}"
puts

Benchmark.ips do |x|
  x.time = 10

  [1000, 2500, 5000, 10000, 20000, 50000].each do |l|
    short_string = string[0..l]

    x.report("ts_headline excerpt #{l}") do
      sql_excerpt(short_string, l)
    end

    x.report("actionview excerpt #{l}") do
      ruby_excerpt(short_string)
    end
  end

  x.compare!
end
```

```
actionview excerpt 1000:    20570.7 i/s
actionview excerpt 2500:    17863.1 i/s - 1.15x  (± 0.00) slower
actionview excerpt 5000:    14228.9 i/s - 1.45x  (± 0.00) slower
actionview excerpt 10000:    10906.2 i/s - 1.89x  (± 0.00) slower
actionview excerpt 20000:     6255.0 i/s - 3.29x  (± 0.00) slower
ts_headline excerpt 1000:     4337.5 i/s - 4.74x  (± 0.00) slower
actionview excerpt 50000:     3222.7 i/s - 6.38x  (± 0.00) slower
ts_headline excerpt 2500:     2240.4 i/s - 9.18x  (± 0.00) slower
ts_headline excerpt 5000:     1258.7 i/s - 16.34x  (± 0.00) slower
ts_headline excerpt 10000:      667.2 i/s - 30.83x  (± 0.00) slower
ts_headline excerpt 20000:      348.7 i/s - 58.98x  (± 0.00) slower
ts_headline excerpt 50000:      131.9 i/s - 155.91x  (± 0.00) slower
```
2020-08-07 14:36:52 +08:00
Guo Xiang Tan
e60c74d3c1 FEATURE: Use PG ts_headline for highlighting topic title in search. 2020-08-07 12:43:09 +08:00
Krzysztof Kotlarek
12a00d6dc5 FEATURE: add advanced order to search (#10385)
Similar to `advanced_filter` I introduced `advanced_order`.

I needed a new option because default orders are evaluated after advanced_filter so I couldn't use it.

Also, that part is a little bit more generic
```
elsif word =~ /order:\w+/
  @order = word.gsub('order:', '').to_sym
nil
```

After those changes, I can use them in plugins in this way:
```
Search.advanced_order(:votes) do |posts|
  posts.reorder("COALESCE((SELECT dvvc.counter FROM discourse_voting_vote_counters dvvc WHERE dvvc.topic_id = subquery.topic_id), 0) DESC")
end
```
2020-08-07 12:47:00 +10:00
Guo Xiang Tan
ab2b6f8dea FIX: Specify config when generating tsquery using ts_headline. 2020-08-07 10:21:14 +08:00
Robin Ward
7df57b35da REFACTOR: Remove Discourse.__widget_helpers
It's now a variable in the context where the templates are created.
2020-08-06 14:35:46 -04:00