This PR improves the UI of bulk select so that its context is applied to the Dismiss Unread and Dismiss New buttons. Regular users (not just staff) are now able to use topic bulk selection on the /new and /unread routes to perform these dismiss actions more selectively.
For Dismiss Unread, there is a new count in the text of the button and in the modal when one or more topic is selected with the bulk select checkboxes.
For Dismiss New, there is a count in the button text, and we have added functionality to the server side to accept an array of topic ids to dismiss new for, instead of always having to dismiss all new, the same as the bulk dismiss unread functionality. To clean things up, the `DismissTopics` service has been rolled into the `TopicsBulkAction` service.
We now also show the top Dismiss/Dismiss New button based on whether the bottom one is in the viewport, not just based on the topic count.
When editing the first post for the topic we do two AJAX requests
to two separate controllers in this order:
PUT /t/topic-name
PUT /posts/2489523
This causes two post revisor calls, which end up triggering the
:post_edited DiscourseEvent twice. This is then picked up and sent
as a WebHook event twice. However we do not need to send a :post_edited
webhook event if the first post is being edited and topic_changed is
true from the :post_edited DiscourseEvent, because a second event will
shortly come through for just the post.
See https://meta.discourse.org/t/post-webhook-fires-two-times-on-post-edited-for-first-post-in-a-topic/162408
Continued on from https://github.com/discourse/discourse/pull/10590
When uploads are created from the composer (editing or creating a post),
for sites with secure uploads enabled we assume security by default and
that new upload is set to secure. When the post is created, we then
check whether the post uploads _actually_ need to be secure and adjust
accordingly.
We were not doing this when revising a post, so when a new upload was
created when editing a post in a public topic, the secure status stayed
true erroneously causing issues with image previews, among other things.
This is a recent regression introduced by https://github.com/discourse/discourse/pull/12937 which makes it so that when looking at a user profile that is not your own, specifically the category and tag notification settings, you would see your own settings instead of the target user. This is only a problem for admins because regular users cannot see these details for other users.
The issue was that we were using `scope` in the serializer, which refers to the current user, rather than using a scope for the target user via `Guardian.new(user)`.
However, on further inspection the `notification_levels_for` method for `TagUser` and `CategoryUser` did not actually need to be accepting an instance of Guardian, all that it was using it for was to check guardian.anonymous? which is just a fancy way of saying user.blank?. Changed this method to just accept a user instead and send the user in from the serializer.
When issuing the discobot certificate, we were not closing the
iframe tag, which meant that the final message instruction to
the user was swallowed up.
We have a few places in the code where we need to validate various email related settings, and will have another soon with the improved group email settings UI. This PR introduces a class which can validate POP3, IMAP, and SMTP credentials and also provide a friendly error message for issues if they must be presented to an end user.
This PR does not change any existing code to use the new service. I have added a TODO to change POP3 validation and the email test rake task to use the new validator post-release.
Some emails coming in via the mail receiver can still end up
with bad encoding when trying to enqueue the job. This catches
the last encoding issue and forces iso-8559-1 and encodes to
UTF-8 to circumvent the issue.
This was needed to fix a bookmark back button issue but it
broke category topic links, causing a full reload. Now it appears
something has changed in core and this is no longer necessary for
the bookmark back button to work, so I am removing it again.
We have found when receiving and posting inbound emails to the handle_mail route, it is better to POST the payload as a base64 encoded string to avoid strange encoding issues. This introduces a new param of `email_encoded` and maintains the legacy param of email, showing a deprecation warning. Eventually the old param of `email` will be dropped and the new one `email_encoded` will be the only way to handle_mail.
The user may have changed their category or tag tracking settings since a topic was tracked/watched based on those settings in the past. In that case we need to alter the reason message we show them otherwise it is very confusing for the end user to be told they are tracking a topic because of a category, when they are no longer tracking that category.
For example: "You will see a count of new replies because you are tracking this category." becomes: "You will see a count of new replies because you were tracking this category in the past."
To do this, it was necessary to add tag and category tracking info to current user serializer. I improved the serializer code so it only does 3 SQL queries instead of 9 to get the tracking information for tags and categories for the current user.
See: https://meta.discourse.org/t/navigating-back-to-bookmarks/188912/4
Instead of taking the user back to the bookmark list after selecting
a topic and navigating back, the user was navigated back to the page
before that. This is because the topic-link component was missing
the data-auto-route attribute which tells the intercept-click library
not to use DiscourseURL.routeTo to handle the transition (so it is just
handled internally by Ember)
The aim of this PR is to improve the topic tracking state JavaScript code and test coverage so further modifications can be made in plugins and in core. This is focused on making topic tracking state changes easier to respond to with callbacks, and changing it so all state modifications go through a single method instead of modifying `this.state` all over the place. I have also tried to improve documentation, make the code clearer and easier to follow, and make it clear what are public and private methods.
The changes I have made here should not break backwards compatibility, though there is no way to tell for sure if other plugin/theme authors are using tracking state methods that are essentially private methods. Any name changes made in the tracking-state.js code have been reflected in core.
----
We now have a `_trackedTopicLimit` in the tracking state. Previously, if a topic was neither new nor unread it was removed from the tracking state; now it is only removed if we are tracking more than `_trackedTopicLimit` topics (which is set to 4000). This is so plugins/themes adding topics with `TopicTrackingState.register_refine_method` can add topics to track that aren't necessarily new or unread, e.g. for totals counts.
Anywhere where we were doing `tracker.states["t" + data.topic_id] = newObject` has now been changed to flow through central `modifyState` and `modifyStateProp` methods. This is so state objects are not modified until they need to be (e.g. sometimes properties are set based on certain conditions) and also so we can run callback functions when the state is modified.
I added `onStateChange` and `onMessageIncrement` methods to register callbacks that are called when the state is changed and when the message count is incremented, respectively. This was done so we no longer need to do things like `@observes("trackingState.states")` in other Ember classes.
I split up giant functions like `sync` and `establishChannels` into smaller functions for readability and testability, and renamed many small functions to _functionName to designate them as private functions which not be called by consumers of `topicTrackingState`. Public functions are now all documented (well...at least ones that are not immediately obvious).
----
On the backend side, I have changed the MessageBus publish events for TopicTrackingState to send back tags and tag IDs for more channels, and done some extra code cleanup and refactoring. Plugins may override `TopicTrackingState.report` so I have made its footprint as small as possible and externalised the main parts of it into other methods.
The "last custom date and time" shortcut for the topic timer and
bookmarks could get into a state where it had an Invalid Date if
the user opened the topic timer modal, clicked Custom Date and then
closed the modal without making changes. This has been fixed, the
last custom date + time will no longer be set in this case and if
somehow the last custom date + time is invalid that option will not
show.
Also improve the wording from just "Last" to "Last custom datetime"
When the user has not selected any tags and minimum_required_tags
is specified for a category, they get a clientside validation error
to tell them this. We were not doing the same thing when
min_tags_from_required_group and required_tag_groups was specified.
There is a category setting that enforces 1 or more tags must be added to a topic from a specific tag group before creating it. This validation was not being run before the topic was being sent to a review queue for categories that have that setting enabled.
There was an existing validation in `TopicCreator` but it was not correct; it was only validating when the tags did _not_ exist and also only happened on `create`. I now run the validation in `TopicCreator.valid?`
I also improved the error message shown to the user when they have not added the tags required (showing the tag names from the tag group), and changed the composer tag selector to not show "optional" if there are N tags required from a certain group.
This PR adds a new category setting which is a column in the `categories` table, `allow_unlimited_owner_edits_on_first_post`.
What this does is:
* Inside the `can_edit_post?` method of `PostGuardian`, if the current user editing a post is the owner of the post, it is the first post, and the topic's category has `allow_unlimited_owner_edits_on_first_post`, then we bypass the check for `LimitedEdit#edit_time_limit_expired?` on that post.
* Also, similar to wiki topics, in `PostActionNotifier#after_create_post_revision` we send a notification to all users watching a topic when the OP is edited in a topic with the category setting `allow_unlimited_owner_edits_on_first_post` enabled.
This is useful for forums where there is a Marketplace or similar category, where topics are created and then updated indefinitely by the OP rather than the OP making new topics or additional replies. In a way this acts similar to a wiki that only one person can edit.
* Fixes the z-index of the prompt so it is behind the quick access panels
* Adds a dismiss `X` button (made sure the click target of this was quite big)
* Change structure of HTML to address template lint issues
* Fix aria-hidden not returning true/false
* Reload current page instead of navigating to / when clicking on the prompt message
When posts are moved from one topic to another, the `topic_user.bookmarked` column for all users in the new and the old topic needs to be resynced, for example because a user bookmarks post 12 in topic 1, then it is moved to topic 2, the topic_user record for topic 1 should no longer be bookmarked. A background job has been added to sync the column for a specified topic, or for no topic at all, which does it for all topics like the migration.
Also includes a migration that we have run in the past to fix bad data.
----
This has been addressed in other places in the past:
https://github.com/discourse/discourse/pull/10211https://github.com/discourse/discourse/pull/10188
We introduced a cap on the number of bookmarks the user can add in https://github.com/discourse/discourse/commit/be145ccf2fe117d97b36aa1e2fc500e44319d58c. However this has caused unintended side effects; when the `jobs/scheduled/bookmark_reminder_notifications.rb` runs we get this error for users who already had more bookmarks than the limit:
> Job exception: Validation failed: Sorry, you have too many bookmarks, visit #{url}/my/activity/bookmarks to remove some.
This is because the `clear_reminder!` call was triggering a bookmark validation, which raised an error because the user already had to many, holding up other reminders.
This PR also adds `max_bookmarks_per_user` hidden site setting (default 2000). This replaces the BOOKMARK_LIMIT const so we can raise it for certain sites.
This moves the "This site was just updated" modal asking the user if they want to refresh into a subtle prompt that slides down from the header.
Also in this PR I've added a helper to publish message bus messages in JS tests. So instead of this:
```javascript
// Mimic a messagebus message
MessageBus.callbacks
.filterBy("channel", "/global/asset-version")
.map((c) => c.func("somenewversion"));
```
We can have:
```javascript
publishToMessageBus("/global/asset-version", "somenewversion");
```
If the user has not been sent any messages, show a message in the quick access menu with an educational message. If the user can send private messages, also show a link to open the "new message" composer:
This also adds a general improvement to the quick-access-panel, to be able to show an `emptyStateWidget` instead of just a message if there is nothing to show in the panel, as well as initial general styles for empty state.
Because bookmarks have both topic and post ID, when the post was moved into another topic the bookmark was still attached to the post but did not show in the UI. This PR makes it so the all topic IDs for bookmarks attached to a post are updated when a post is moved.
Also included is a migration to fix affected records (e.g. on Meta there are 20 affected records).
See: https://meta.discourse.org/t/improved-bookmarks-with-reminders/144542/203
This is called in DiscourseTagging.tag_topic_by_names only after
all the validations etc. have been passed, and after topic.tags = X
has been called (because this is when the associations are created/
destroyed). The event has the topic, then a second param with the
old and new tag names in arrays for easy inspection.
The instance of the PostRevisor is passed to the post_edited
event. It is useful to know what has happened to the topic in
this event (we already pass a boolean for topic_changed? but that
is not so helpful by itself).
This form does not need to show if discourse connect is enabled
because generally the fields that would be filled in here are
filled in by the SSO provider. There is also an issue right now
where enable_local_logins and enable_discourse_connect can be
true at the same time which is not right.
The regular expression to detect private IP addresses did not always detect them successfully.
Changed to use ruby's in-built IPAddr.new(ip_address).private? method instead
which does the same thing but covers all cases.
Users can now pin bookmarks from their bookmark list. This will anchor the bookmark to the top of the list, and show a pin icon next to it. This also applies in the nav bookmarks panel. If there are multiple pinned bookmarks they sort by last updated order.
This PR adds MethodProfiler.output_sql_to_stderr! for easier debugging of SQL queries and their timings from the console.
This is almost the same as ensure_discourse_instrumentation! but should not
be used in production (save for debugging in the console), and is only instrumenting
PostgresSQL queries.
This is almost the same as ensure_discourse_instrumentation! but should not
be used in production. This logs all SQL queries run and their durations
between start and stop.
It also works for super long running queries. If you interrupt the long-running
query the latest query data will still be logged after stopping the profiler.
Usage:
```
MethodProfiler.output_sql_to_stderr!(filter_transactions: true)
MethodProfiler.start
# some code that runs queries
timings = MethodProfiler.stop
```
This PR allows invitations to be used when the DiscourseConnect SSO is enabled for a site (`enable_discourse_connect`) and local logins are disabled. Previously invites could not be accepted with SSO enabled simply because we did not have the code paths to handle that logic.
The invitation methods that are supported include:
* Inviting people to groups via email address
* Inviting people to topics via email address
* Using invitation links generated by the Invite Users UI in the /my/invited/pending route
The flow works like this:
1. User visits an invite URL
2. The normal invitation validations (redemptions/expiry) happen at that point
3. We store the invite key in a secure session
4. The user clicks "Accept Invitation and Continue" (see below)
5. The user is redirected to /session/sso then to the SSO provider URL then back to /session/sso_login
6. We retrieve the invite based on the invite key in secure session. We revalidate the invitation. We show an error to the user if it is not valid. An additional check here for invites with an email specified is to check the SSO email matches the invite email
7. If the invite is OK we create the user via the normal SSO methods
8. We redeem the invite and activate the user. We clear the invite key in secure session.
9. If the invite had a topic we redirect the user there, otherwise we redirect to /
Note that we decided for SSO-based invites the `must_approve_users` site setting is ignored, because the invite is a form of pre-approval, and because regular non-staff users cannot send out email invites or generally invite to the forum in this case.
Also deletes some group invite checks as per https://github.com/discourse/discourse/pull/12353
When transitioning from a tag topic list e.g. /tag/alerts
to the / route the topic list was not reloaded because the
same preload key was used for both lists (topic_list_latest).
The topic list was only reloaded when clicking on the / route
a second time because then it is forced to reload.
In the topic list adapter, we call `PreloadStore.getAndRemove` to
get the topic lists:
https://github.com/discourse/discourse/blob/534777f5fdbf48a6a77a362636cc936a19c17c43/app/assets/javascripts/discourse/app/adapters/topic-list.js#L34-L41
Now instead of both / and /tag/alerts sharing the same preload
key of `topic_list_latest`, the tag has a key of `topic_list_tag/alerts/l/latest`
The recalculation of the `forceActive` function for the nav items was being run _before_ the querystring parameters are getting updated. For example for the Unassigned link:
```javascript
i.addNavigationBarItem({
name: "unassigned",
customFilter: (category) => {
return category && category.enable_unassigned_filter;
},
customHref: (category) => {
if (category) {
return getURL(category.url) + "/l/latest?status=open&assigned=nobody";
}
},
forceActive: (category, args, router) => {
const queryParams = router.currentRoute.queryParams;
return (
queryParams &&
Object.keys(queryParams).length === 2 &&
queryParams["assigned"] === "nobody" &&
queryParams["status"] === "open"
);
},
before: "top",
});
```
When forceActive is hit going from e.g.`http://localhost:3000/c/some-category/5/l/top` to `http://localhost:3000/c/some-category/5/l/latest?assigned=nobody&status=open` the `queryParams` are empty and the URL does not seem to change until after the transition and so `active` ends up being false in this `navigation-item` function which controls whether or not to do the highlight:
```javascript
@discourseComputed("content.filterType", "filterType", "content.active")
active(contentFilterType, filterType, active) {
if (active !== undefined) {
return active;
}
return contentFilterType === filterType;
},
```
Also sometimes this is not even recalculated, for example going from `http://localhost:3000/c/some-category/5/l/latest?status=open` to `http://localhost:3000/c/some-category/5/l/latest?assigned=nobody&status=open`. This PR fixes the issue where the query parameters changing was not forcing this recalculation. This was especially noticable in conjunction with https://github.com/discourse/discourse-loading-slider.
remove 3 month option for topic timer
move relative time input inside the custom
date and time shortcut
make sure special options are always at the bottom
Add a new year interval option to relative time picker, and also fix some rounding issues (Math.floor is not ideal because it gets rid of half days etc.)
Also adding some component tests here for relative-time-picker.
We were sending 2 emails for user silencing if a message was provided in the UI. Also always send email for user silence and user suspend with reason regardless of whether message provided.
Uses discourse/onebox@ff9ec90
Adds a hidden site setting called disable_onebox_media_download_controls which will add controlslist="nodownload" to video and audio oneboxes, and also to the local video and audio oneboxes within Discourse.
Previously when inheriting category auto-close settings for a topic, those settings were disrupted if another topic timer was assigned or if a topic was closed then manually re-opened.
This PR makes it so that when a topic is manually re-opened the topic auto-close settings are inherited from the category. However, they will now be based on the topic created_at date. As an example, for a topic with a category auto close hours setting of 72 (3 days):
* Topic was created on 2021-02-15 08:00
* Topic was closed on 2021-02-16 10:00
* Topic was opened again on 2021-02-17 06:00
Now, the topic will inherit the auto close timer again and will close automatically at **2021-02-18 08:00**, which is based on the creation date. If the current date and time is greater than the original auto-close time (e.g. we were at 2021-02-20 13:45) then no auto-close timer is created.
Note, this will not happen if the topic category auto-close setting is "based on last post".
This PR adds an edit button to the topic timer info message which opens the modal.
Also, I have cleaned up a few more places where we were referencing "topic status update" which is what these were called prior to being called topic timers.
The category settings for auto-close topic hours has now also been modified to use the new relative-time-picker component.
Finally, the relative-time-picker input step and min is dynamic based on mins/other intervals selected, see https://review.discourse.org/t/feature-relative-time-input-for-timers-and-bookmarks-and-promote-auto-close-after-last-post-timer-12063/19204/7?u=martin
This PR adds a new relative-time component, that is an input box with a SK dropdown of minutes, hours, days, and months which outputs the duration selected in minutes. This new component is used in the time shortcuts list (used by bookmarks and topic timers) as a new Relative Time shortcut.
Also in this PR, I have made the "Auto-Close After Last Post" timer into a top level timer type in the UI, and removed the "based on last post" custom time shortcut.
Because of a where clause of duration_minutes != duration, where
duration_minutes was NULL, the previous migration to fill the new
duration_minutes column failed. This corrects the failed migration
by just running the update where duration_minutes is NULL and duration
IS NOT NULL.
Previous commit is 4af77f1
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.
This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.
I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
This moves all the rate limiting for user second factor (based on `params[:second_factor_token]` existing) to the one place, which rate limits by IP and also by username if a user is found.
This PR revamps the topic timer UI, using the time shortcut selector from the bookmark modal.
* Fixes an issue where the duration of hours/days after last reply or auto delete replies was not enforced to be > 0
* Fixed an issue where the timer dropdown options were not reloaded correctly if the topic status changes in the background (use `MessageBus` to publish topic state in the open/close timer jobs)
* Moved the duration input and the "based on last post" option from the `future-date-input` component, as it was only used for topic timers. Also moved out the notice that is displayed which was also only relevant for topic timers.
This PR makes it so the bookmark name shows on hover in the quick access menu. A change was necessary to quick-access-item for the title to render for the link.
* sometimes the AJAX promise to create/save the bookmark did
not come back before the component destroyed, causing an error
when trying to set the model id afterward. this just eliminates
the set code and uses the response.id instead
This PR moves all of the time picking functionality from the bookmark modal and controller into a reusable time-shortcut-picker component, which will be used for the topic timer UI revamp. All of the utility JS for getting dates like tomorrow/next week/next month etc. have also been moved into a separate utility lib.
The time-shortcut-picker has a couple of options that can be passed in:
* prefilledDatetime - The date and time to parse and prefill into the custom date and time section, useful for editing interfaces.
* onTimeSelected (callback) - Called when one of the time shortcuts is clicked, and passes the type of the shortcut (e.g. tomorrow) and the datetime selected.
* additionalOptionsToShow - An array of option ids to show (by default `later_today` and `later_this_week` are hidden)
* hiddenOptions - An array of option ids to hide
* customOptions - An array of custom options to display (e.g. the option to select a post date for the bookmarks modal). The options should have the below properties:
* id
* icon
* label (I18n key)
* time (moment datetime object)
* timeFormatted
* hidden
The other major work in this PR is moving all of the bookmark functionality out of the bookmark modal controller and into its own component, where it makes more sense to be able to access elements on the page via `document`. Tests have been added to accompany this move, and existing acceptance tests for bookmark are all passing.
Adds a new column/setting to groups, allow_unknown_sender_topic_replies, which is default false. When enabled, this scenario is allowed via IMAP:
* OP sends an email to the support email address which is synced to a group inbox via IMAP, creating a group topic
* Group user replies to the group topic
* An email notification is sent to the OP of the topic via GroupSMTPMailer
* The OP has several email accounts and the reply is sent to all of them, or they forward their reply to another email account
* The OP replies from a different email address than the OP (gloria@gmail.com instead of gloria@hey.com for example)
* The a new staged user is created, the new reply is accepted and added to the topic, and the staged user is added to the topic allowed users
Without allow_unknown_sender_topic_replies enabled the new reply creates an entirely new topic (because the email address it is sent from is not previously part of the topic email chain).
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.
As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
Improvements to make console access to IncomingEmail more pleasant, and stopping certain IMAP logs from landing in the DB because they just create too much noise,
This should make it easier to track down how the incoming email was created, which is one of four locations:
The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
There was an issue that occurred with this order of operations:
* An IMAP topic was created by emailing a group
* A second user was invited to the topic (not the OP and not the group)
* A user with access to the group replies to the topic
* The second user receives a user_private_message notification email because of their involvement in the topic
* The second user replies to the email via email
This new reply would then go and notify the other group PM users, except for those who emailed the group topic directly, which is handled via the group SMTP mailer. However because the new post already has an incoming email because it is parsed via the Email::Receiver via POP3 the group SMTP section of the post alerter is skipped, and the group's email address is not ignored for the user_private_message notification.
This PR fixes it so the group is not ever sent an email via the PM notification. This is important because any new emails in the group's IMAP inbox will be picked up by the Imap::Sync code and created as a new topic which is not at all desirable.
Also in this PR I split up the specs a bit more for group SMTP in the post alerter to make them easier to read and they each only test one thing.
As per @davidtaylorhq 's comment at 6e2be3e#r46069906, this fixes an oversight where if login_required is enabled and an anon user follows a confirm new email link they are forced to login, which is not what the intent of #10830 was.
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.
The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.
This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
This adds a safe default to not process pop3 emails when the pop3 polling option is set up that are > 1 week old. This is to avoid the situation where an older mailbox is used, which causes us to go and process all emails in that mailbox, sending out error emails to the senders of emails which cannot be parsed successfully.
When selecting the "date in post" option from the bookmark reminder modal, it was not converting the date from the post, which may be in a completely different timezone, to the current user's timezone.
This PR fixes it so the timezone from the post is used to parse the date then converted to the user's timezone.
Fixes bug introduced by https://github.com/discourse/discourse/commit/bd25627198dd74442cc4014975e7af2d3b142622
What happens is we send notifications to everyone involved in the group inbox topic about new posts, however we pass the param `skip_send_email_to: email_addresses`. In the above commit I removed the group email address from this `email_addresses` array. This breaks the IMAP inbox because we email the group with the reply, and the IMAP sync tool finds this email and opens a new unrelated topic with it.
This PR fixes a race condition with the IMAP notification code. In the `Email::Receiver` we call the `NewPostManager` to create the post and enqueue jobs and sends alerts via `PostAlerter`. However, if the post alerter reaches the `notify_pm_users` and the `group_notifying_via_smtp` method _before_ the incoming email is updated with the post and topic, we unnecessarily send a notification to the person who just posted. The result of this is that the IMAP syncer re-imports the email sent to the user about their own post, which looks like this in the group inbox:
To fix this, we skip the jobs enqueued by `NewPostManager` and only enqueue them with `PostJobsEnqueuer` manually _after_ the incoming email record has been updated with the post and topic.
Other improvements:
* Moved code to calculate email addresses from `IncomingEmail` records into the topic, with a group passed in, for easier testing and debugging. It is not the responsibility of the post alerter to figure this stuff out.
* Add shortcut methods on `IncomingEmail` to split or provide an empty array for to and cc addresses to avoid repetition.
This PR adds functionality for the IMAP sync code to detect if a UID that is missing from the mail group mailbox is in the Spam/Junk folder for the mail account, and if so delete the associated Discourse topic. This is identical to what we do for emails that are moved for Trash.
If an email is missing but not in Spam or Trash, then we mark the incoming email record with imap_missing: true. This may be used in future to further filter or identify these emails, and perhaps go hunting for them in the email account in bulk.
Note: This adds some code duplication because the trash and spam email detection and handling is very similar. I intend to do more refactors/improvements to the IMAP sync code in time because there is a lot of room for improvement.
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.
Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.
Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
Fixes a rare race condition causing the `Imap::Sync` class to create an incoming email and associated post/topic, which then kicks off the PostAlerter to notify others in the PM about a reply in the topic, but for the OP which is not necessary (because the person emailing the IMAP inbox already knows about the OP). Basically, we should never be sending the group SMTP email for the first post in a topic.
Also in this PR:
* Custom attribute accessors for the to/from/cc addresses on `IncomingEmail`, to parse them from an array to a joined string so the logic for this is only in one place.
* Store extra detail against the `IncomingEmail` created in `GroupSmtpMailer`
* regex test Mail header Reply-To as string instead of Field, which fixes `warning: deprecated Object#=~ is called on Mail::Field; it always returns nil`
* Add DEBUG_IMAP to log all IMAP logs as warnings for easier debugging
* Changed the Rails logging to `ImapSyncLog` in the `GroupSmtpMailer`
When calculating whether the attached uploads went over the SiteSetting.email_total_attachment_size_limit_kb.kilobytes limit, we were using the original_upload for the calculation instead of the actually attached_upload, which will be smaller in most cases because it can be an optimized image.
When embedding secure images which have been oneboxed, we checked to see if the image's parent's parent had the class onebox-body. This was not always effective as if the image does not get resized/optimized then it does not have the aspect-image div wrapping it. This would cause the image to embed in the email but be huge.
This PR changes the check to see if any of the image's ancestors have the class onebox-body, or if the image has the onebox-avatar class to account for variations in HTML structure.
There are issues around displaying images on published pages when secure media is enabled. This PR temporarily makes it appear as if published pages are enabled if secure media is also enabled.
See #10794 for original context.
I did not mean to add invite to the BYPASS_TYPES for Email::Sender, it was supposed to be invite_password_instructions.
We had an issue where onebox thumbnail was too large and thus was optimized, and we are using the image URLs in post to redact and re-embed, based on the sha1 in the URL. Optimized image URLs have extra stuff on the end like _99x99 so we were not parsing out the sha1 correctly. Another issue I found was for posts that have giant images, the original was being used to embed in the email and thus would basically never get included because it is huge.
For example the URL http://localhost:3000/secure-media-uploads/optimized/2X/7/787b17ea6140f4f022eb7f1509a692f2873cfe35_2_690x335.jpeg was not parsed correctly; we would end up with 787b17ea6140f4f022eb7f1509a692f2873cfe35_2_690x335.jpeg as the sha1 which would not find the image to re-embed that was already attached to the email.
This fix will use the first optimized image of the detected upload when we are redacting and then re-embedding to make sure we are not sending giant things in email. Also, I detect if it is a onebox thumbnail or the site icon and force appropriate sizes and styles.
There is a site setting reply_by_email_enabled which when combined with reply_by_email_address creates a Reply-To header in emails in the format "test+%{reply_key}@test.com" along with a PostReplyKey record, so when replying Discourse knows where to route the reply.
However this conflicts with the IMAP implementation. Since we are sending the email for a group via SMTP and from their actual email account, we want all replys to go to that email account as well so the IMAP sync job can pick them up and put them in the correct place. So if the group has IMAP enabled and configured, then the reply-to header will be correct.
This PR also makes a further fix to 64b0b50 by using the correct recipient user for the PostReplyKey record. If the post user is used we encounter this error:
if destination.user_id != user.id && !forwarded_reply_key?(destination, user)
raise ReplyUserNotMatchingError, "post_reply_key.user_id => #{destination.user_id.inspect}, user.id => #{user.id.inspect}"
end
This is because the user above is found from the from_address, but the destination which is the PostReplyKey is made by the post.user, which will be different people.
On forums with a large amount of posts when a user had a bookmark in the topic, PostgreSQL was using an inefficient query plan to fetch the first post of the topic. When running this ActiveRecord query:
```
topic.posts.with_deleted.where(post_number: 1).first
```
The following query plan was produced:
```
Limit (cost=0.43..583.49 rows=1 width=891) (actual time=3850.515..3850.515 rows=1 loops=1)
-> Index Scan using posts_pkey on posts (cost=0.43..391231.51 rows=671 width=891) (actual time=3850.514..3850.514
rows=1 loops=1)
Filter: ((topic_id = 160918) AND (post_number = 1))
Rows Removed by Filter: 2274520
Planning time: 0.200 ms
Execution time: 3850.559 ms
(6 rows)
```
The issue here is the combination of ORDER BY and LIMIT causing the ineficcient Index Scan using posts_pkey on posts to be used. When we correct the AR call to this:
```
topic.posts.with_deleted.find_by(post_number: 1)
```
We end up with a query that still has a LIMIT but no ORDER BY, which in turn creates a much more efficient query plan:
```
Limit (cost=0.43..1.44 rows=1 width=891) (actual time=0.033..0.034 rows=1 loops=1)
-> Index Scan using index_posts_on_topic_id_and_post_number on posts (cost=0.43..678.82 rows=671 width=891) (actua
l time=0.033..0.033 rows=1 loops=1)
Index Cond: ((topic_id = 160918) AND (post_number = 1))
Planning time: 0.167 ms
Execution time: 0.072 ms
(5 rows)
```
This query plan uses the correct index, `Index Scan using index_posts_on_topic_id_and_post_number on posts`. Note that this is only a problem on forums with a larger amount of posts; tiny forums would not notice the difference. On large forums a query for a topic that takes 1s without a bookmark can take 8-30 seconds, and even end up with 502 errors from nginx.
This PR introduces a feature that will detect a date inside the post that a user is bookmarking, and offer that date as an option in the bookmark modal.
The logic is that we get the first date/time detected in the post. If it does not have a time, just a date, then we default to 8:00am for the time.
Our Email::Sender class accepts an optional user argument, which is used to create a PostReplyKey record when present. This record is used to sub out the %{reply_key} placeholder in the Reply-To mail header, so if we do not pass in the user we get a broken Reply-To header.
This is especially problematic in the IMAP group SMTP situation, because these emails go to customers that we are replying to, and when they reply to us the email bounces! This fixes the issue by passing user to the Email::Sender when sending a group_smtp email but there is still more to do in another PR.
This Email::Sender optional user is a bit of a footgun IMO, especially because most of the time we use it there is a user we can source. I would like to do another PR for this after this one to make the parameter not optional, so we don't end up with these reply issues down the line again.
If RAILS_DISABLE_ACTIVERECORD_LOGS=1 is passed when starting Rails, none of the query log output will show. This is very useful for debugging with breakpoints because logs are not flooding in constantly.
When posts or topics are deleted we don't want to immediately delete associated bookmarks, so we have a grace period to recover them and their reminders if the post or topic is un-deleted. This PR adds a task to the Weekly scheduled job to go and delete bookmarks attached to posts or topics deleted > 3 days ago.
See https://meta.discourse.org/t/email-address-change-confirmation-email-not-sent-but-every-other-notification-emails-are/165358
In short: with disable emails set to non-staff, email address change confirmation emails (those sent to the new address) are not sent for staff or admin members.
This was happening because we were looking up the staff user with the to_address of the email, but the to address was the new email address because we are sending a confirm email change email, and thus the user could not be found. We didn't need to do this anyway because we are passing the user into the Email::Sender class anyway.
See https://meta.discourse.org/t/changing-a-users-email/164512 for additional context.
Previously when an admin user changed a user's email we assumed that they would need a password reset too because they likely did not have access to their account. This proved to be incorrect, as there are other reasons a user needs admin to change their email. This PR:
* Changes the admin change email for user flow so the user is sent an email to confirm the change
* We now record who the email change request was requested by
* If the requested by user is admin and not the user we note this in the email sent to the user
* We also make the confirm change email route open to anonymous users, so it can be clicked by the user even if they do not have access to their account. If there is a logged in user we make sure the confirmation matches the current user.
Upload.secure_media_url? raised an exceptions when the URL was invalid,
which was a issue in some situations where secure media URLs must be
removed.
For example, sending digests used PrettyText.strip_secure_media,
which used Upload.secure_media_url? to replace secure media with
placeholders. If the URL was invalid, then an exception would be raised
and left unhandled.
Now instead in UrlHelper.rails_route_from_url we return nil if there is something wrong with the URL.
Co-authored-by: Bianca Nenciu <nenciu.bianca@gmail.com>
The download link on the lightbox for images was not downloading the image if the upload was marked secure, because the code in the upload controller route was not respecting the dl=1 param for force download.
This PR fixes this so the download link works for secure images as well as regular ligthboxed images.
See https://meta.discourse.org/t/changing-a-users-email/164512 for context.
When admin changes an email for a user, we were incorrectly sending the password reset email to the user's old address. Also the new email does not come into effect until the reset password process is done, so this PR adds some notes to the admin to make this clearer.
With secure media and the UploadSecurity class, we need a nice way for plugins to register custom upload types that should be considered public and never secure.
We are making the changes from the PR #10563 the default behaviour. Now, if secure media is enabled, secure images will be embedded in emails by default instead of redacting them and displaying a message. This will be a nicer overall experience by default, and for forums that want to be super strict with redaction this setting can always be disabled.
Martin Brennan
