This commit allows group SMTP emails to be sent with a
different from email address that has been set up as an
alias in the email provider. Emails from the alias will
be grouped correctly using Message-IDs in the mail client,
and replies to the alias go into the correct group inbox.
This commit adds a requestCustomMarkdownCookFunction function
to the `helper` that is provided to custom markdown rules
via their `setup` function.
The way this works is that once the default markdown engine that
we use for cooking posts has been set up, we loop through all
of the callbacks registered by `requestCustomMarkdownCookFunction`
and call `_buildCustomMarkdownCookFunction`. This creates
a new markdown engine using many of the same settings as the
default one, but will allow for the following options to be
changed by the markdown rule requesting the custom function:
* featuresOverride - The markdown-it features to allow for the engine
* markdownItRules - The markdown-it rules to allow for the engine
After this engine is set up a render function which renders + sanitizes
the output is returned for use by the markdown rule.
The use case for this API is mainly for block BBCode markdown rules
which want to render their content with a limited subset of the
markdown features/rules. Our initial use case for this is chat message
quoting.
This commit also does some minor refactoring of discourse-markdown-it
to accommodate this new engine building.
When changing to uppy for file uploads we forgot to add
these conditions to the paste event from 9c96511ec4
Basically, if you are pasting more than just a file (e.g. text,
html, rtf), then we should not handle the file and upload it, and
instead just paste in the text. This causes issues with spreadsheet
tools, that will copy the text representation and also an image
representation of cells to the user's clipboard.
This also moves the paste event for composer-upload-uppy to the
element found by the `editorClass` property, so it shares the paste
event with d-editor (via TextareaTextManipulation), which makes testing
this possible as the ember paste bindings are not picked up unless both
paste events are on the same element.
When creating a direct message to a group with group SMTP
set up, and adding another person to that message in the OP,
we send an email to the second person in the OP via the group_smtp
job. This in turn creates an IncomingEmail record to guard against
IMAP double sync.
The issue with this was that this IncomingEmail (which is essentialy
a placeholder/dummy one) was having its Message-ID used as the canonical
References Message-ID for subsequent emails sent out to user_private_message
recipients (such as members of the group), causing threading issues in
the mail client. The canonical <topic/ID@HOST> format should be used
instead for these cases.
This commit fixes the issue by only using the IncomingEmail for the
OP's Message-ID if the OP was created via our handle_mail email receiver
pipeline. It does not make sense to use it in other cases.
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.
```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```
These are passed down to markdown rules on opts.discourse.additionalOptions.
The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
If a model class calls preload_custom_fields twice then
we have to clear this otherwise the fields are cached inside the
already existing proxy and no new ones are added, so when we check
for custom_fields[KEY] an error is likely to occur
This commit makes a more specific N1NotPreLoadedError from
StandardError to raise when a custom field is loaded before
being preloaded, so it is easier to test that this does
not happen from plugins. Also adds the name of the class
trying to load the custom field to the error message.
This commit adds a hover effect for drag and drop in
the admin emoji uploader. It also changes the "Add New
Emoji" button to open the file selector; previously it
was useless because it was disabled unless a name was
entered (which is not even a requirement for the emoji)
and also it didn't actually do anything on click even
if it wasn't disabled.
Now we have a way of adding files without having to drag
and drop them, which is nice.
Also in this PR, there was no indication before that the upload was
complete apart from the button becoming enabled again.
This commit adds the highlight class to the emoji list
and removes it once the highlight fade animation is done,
like we do for new posts.
Follow up to 48f70dcd5f. The group
_appeared_ to be saved in the UI until a refresh when it became
clear that the group wasn't actually sent to the DB. This is because
of the way the per-file data was being set with a computed property.
This commit fixes the computed property by changing it to a regular
function and also makes sure the name resetting after the first upload
in multiple uploads works too.
When uploading multiple emoji in Admin/Customize/Emojis
with an emoji Group selected, the group was cleared between
each file uploaded, making bulk uploading of emojis a chore
if anything but the default group was needed.
This commit fixes the issue, introduces tests for emoji-uploader,
and also adds `add-files` appEvents for uppy-upload mixin, same
as the composer-upload-uppy mixin, for interop with tests and so
we don't have to rely on a file upload element's change event.
This reverts commit 2c7906999a.
The changes break some things in local development (putting JS files
into minified files, not allowing debugger, and others)
It is too close to release of 2.8 for incomplete
feature shenanigans. Ignores and drops the columns and drops
the trigger/function introduced in
e21c640a3c.
Will pick this feature back up post-release.
We are planning on attaching bookmarks to more and
more other models, so it makes sense to make a polymorphic
relationship to handle this. This commit adds the new
columns and backfills them in the bookmark table, and
makes sure that any new bookmark changes fill in the columns
via DB triggers.
This way we can gradually change the frontend and backend
to use these new columns, and eventually delete the
old post_id and for_topic columns in `bookmarks`.
The rake task deleted here was added back in Feb 2020
when bookmarks were first converted from PostAction
records, it is no longer needed. The ignored columns
were removed in ed83d7573e.
This commit adds a check that runs regularly as per
2d68e5d942 which tests the
credentials of groups with SMTP or IMAP enabled. If any issues
are found with those credentials a high priority problem is added to the
admin dashboard.
This commit also formats the admin dashboard differently if
there are high priority problems, bringing them to the top of
the list and highlighting them.
The problem will be cleared if the issue is fixed before the next
problem check, or if the group's settings are updated with a valid
credential.
This commit adds API documentation for the new upload
endpoints related to direct + multipart external uploads.
Also included is a rake task which watches the files in
the spec/requests/api directory and calls a script file
(spec/regenerate_swagger_docs) whenever one changes. This
script runs rake rswag:specs:swaggerize and then copies
the openapi.yml file over to the discourse_api_docs repo
directory, and hits a script there to convert the YML to
JSON so the API docs are refreshed while the server is
still running. This makes the loop of making a doc change
and seeing it in the local server much faster.
The rake task is rake autospec:swagger
This commit removes jQuery file uploader from Discourse,
completing the transition to Uppy. The image-uploader
and UploadMixin components are also removed in this commit
as they have already been replaced and are the only things
using jQuery file upload.
.-'~~~`-.
.' `.
| R I P |
| jquery |
| file |
| upload |
| |
\\| 2013-2021 |//
-----------------
Now that d5e380e5c1 has been
committed there is nothing in the codebase that uses either
resumable.js or the old backup-uploader component.
R.I.P resumable.js
Occasionally there will be a misconfigured CORS rule or a different
network failure when loading one of the media optimization WASM scripts.
This commit handles load failures and sends a new installFailed message
from the service worker, so that we don't error and hold up the rest
of the uploads if this occurs; the worker will just not process anything
and will keep trying to install itself with subsequent uploads until it succeeds.
This commit also removes the redundant useUppy variable in the worker
this should have been removed a while ago in f70e6c302f
This commit removes the enable_experimental_backup_uploader site
setting and the flags in backups-index.hbs to make the uppy
backup uploader the main one from now on.
A follow-up commit will delete the old backup uploader code and
also remove resumable.js from the project.
This commit introduces scheduled problem checks for the admin dashboard, which are long running or otherwise cumbersome problem checks that will be run every 10 minutes rather than every time the dashboard is loaded. If these scheduled checks add a problem, the problem will remain until it is cleared or until the scheduled job runs again.
An example of a check that should be scheduled is validating credentials against an external provider.
This commit also introduces the concept of a `priority` to the problems generated by `AdminDashboardData` and the scheduled checks. This is `low` by default, and can be set to `high`, but this commit does not change any part of the UI with this information, only adds a CSS class.
I will be making a follow up PR to check group SMTP credentials.
This is so the target element for file drag + drop is
not always just this.element for the component, and
provides a way to hook into onDragOver and onDragLeave.
By default also adds a .uppy-is-drag-over class to the target
element.
In the composer, we already only allow for S3 multipart uploads
if enable_direct_s3_uploads is true, so in the backups uploader
that is based on Uppy we want to do the same thing. In future
if self-hosters need some way to not use S3 multipart in these
scenarios for whatever reason we can revisit this then (which
should be as simple as adding a enable_multipart_s3_uploads site
setting).
We cannot use any of the uppy mixins or core code, because
the code there is not shared with the wizard, and to move
it all to discourse-common would be a task almost equal
difficulty to taking the ring to Mordor.
Therefore, we can just use the uppy vendor libraries in the
wizard, and do a quick-n-dirty version of the uppy upload
code for the wizard-field-image uploader.
This commit allows for using Tab and Shift+Tab to indent
and de-indent selected text in the composer. The selected
text is searched for the most occurrences of either tabs (\t)
or spaces at the start of each line, and that character is
used for indentation of all lines.
Since 3b13f1146b the email threading
in mail clients has been broken, because the random suffix meant
that the References header would always be different for non-group
SMTP email notifications sent out.
This commit fixes the issue by always using the "canonical" topic
reference ID inside the References header in the format:
topic/TOPIC_ID@HOST
Which was the old format. We also add the References header to
notifications sent for the first post arriving, so the threading
works for subsequent emails. The Message-ID header is still random
as per the previous change.
Currently the Message-IDs we send out for outbound email
are not unique; for a post they look like:
topic/TOPIC_ID/POST_ID@HOST
And for a topic they look like:
topic/TOPIC_ID@HOST
This commit changes the outbound Message-IDs to also have
a random suffix before the host, so the new format is
like this:
topic/TOPIC_ID/POST_ID.RANDOM_SUFFIX@HOST
Or:
topic/TOPIC_ID.RANDOM_SUFFIX@HOST
This should help with email deliverability. This change
is backwards-compatible, the old Message-ID format will
still be recognized in the mail receiver flow, so people
will still be able to reply using Message-IDs, In-Reply-To,
and References headers that have already been sent.
This commit also refactors Message-ID related logic
to a central location, and adds judicious amounts of
tests and documentation.
Part of overall strategy to remove jQuery file uploader
from the codebase. Also added some helper functionality to
the uppy mixin to allow for non-autostart uploads (all
previous upload changes have been for auto start components.)
The commit 20b2a42f49 broke
upload handlers, because previously we passed through the
native File object to the handler, not the uppy-wrapped
File object.
The inProgressUploads is meant to be used to display these uploads
in a UI, and Ember will only update the array in the UI if pushObject
is used to notify it.
This is a big change to change over to using the uppy
upload mixin in the composer by default. This gets rid
of the temporary composer-editor-uppy component, as well
as removing the old ComposerUpload mixin and copying over
any missing functions that were not yet implemented by
ComposerUploadUppy. This has been working well on our
hosting for some time now and has led us to several
bug fixes.
This commit also deletes the old plugin API for adding
preprocessors for the uploads. The accepted method of doing
this now is via an uppy preprocessor plugin, which we have
several examples of in the core codebase.
Leaving the `enable_experimental_composer_uploader` site setting
intact for now because some plugins still rely on it, this
will be removed at a later date.
One step closer to ending the jQuery file uploader saga...
In jQuery file upload land, we were sending a single file through
at a time to matching upload handlers. This in turn required plugin
authors to marshal the files as they came through one by one if they
wanted to group them together to do something with them. Now that
we are using uppy, files come through in the groups they are added
in (for example dropping multiple, selecting multiple from the system
file dialogue).
This commit changes the matching upload handlers to send through
all matching files at once instead of piecemeal.
In f6528afa01 I added parity support
for composer upload handlers to the uppy-ized composer. However the
way I assumed that it was only possible to handle a single file
upload at a time was false; it only appeared this way in the old
jQuery file upload composer because jQuery file upload sent through
files one at a time even if multiple were added at once. This caused
issues in certain plugins and themes by third parties.
This commit fixes the issue by making the uppy upload handler work
the same as the old one, by capturing all of the added files that
have matching handlers then going through them one by one and passing
them to the handler function.
The theme creator endpoints return JSON with a 201 CREATED
status code. With the recent changes to bootstrap-json/index.js
for the Ember CLI proxy, these endpoints were broken because
201 was not an accepted status code. This commit simply
adds 201 to the array, but prettier forced a reformat as well!
This commit adds handlers for the composer uppy mixin to allow
for cancelling individual file uploads, not just all of them
at once. This is also combined with better tracking of in progress
uploads along with their progress percentage, for UI that needs
to be able to display the progress for individual files and
also cancel individual files.
To use this, a cancel button in the UI should call a function like this:
```javascript
cancelSingleUpload(fileId) {
this.appEvents.trigger(`${this.eventPrefix}:cancel-upload`, {
fileId,
});
},
```
Additionally, the `inProgressUploads` can be shown in the UI. It is an array of objects with the file name, ID, and the progress percentage. We can add more data to this if needed down the line.
This takes the uppy chunking algorithm and combines it with some
form submission from resumable.js for parity with the current
backup controller to make local backup uploads work with uppy.
We can then use this to replace the resumable-upload component
and the resumable.js library from our codebase, once stable.
This is disabled by default, so people using local backups will not
be affected. The enable_experimental_backup_uploader site setting
must be enabled for this to work.
We are pushing /notification-alert/#{user_id} and /notification/#{user_id}
messages to MessageBus from both PostAlerter and User#publish_notification_state.
This can cause memory issues on large sites with many users. This commit
stems the bleeding by only sending these alert messages if the user
in question has been seen in the last 30 days, which eliminates a large
chunk of users on some sites.
We are linting SCSS on the GitHub actions CI but not
on pre-commit, which can lead to lint failures in CI.
Better to warn developers about this locally like
our other lints.
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.
The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:
U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
This commit refactors the direct external upload routes (get presigned
put, complete external, create/abort/complete multipart) into a
helper which is then included in both BackupController and the
UploadController. This is done so UploadController doesn't need
strange backup logic added to it, and so each controller implementing
this helper can do their own validation/error handling nicely.
This is a follow up to e4350bb966
Uppy adds the file name as the "name" parameter in the
payload by default, which means that for things like the
emoji uploader which have a name param used by the controller,
that param will be passed as the file name. We already use
the existing file name if the name param is null, so this
commit just does further cleanup of the name param, removing
the extension if it is a filename so we don't end up with
emoji names like blah_png.
When uploading emoji with the new uppy upload mixin, we were
not sending the name of the emoji in the payload, or more
accurately uppy was already using the file name as the name
value and we were not overriding it from data. This commit
changes the behaviour for single files uploaded via the uppy
upload mixin, by merging the file's meta object with this.data
from the parent component.
When there are multiple groups on a topic, we were selecting
the first from the topic allowed groups to act as the sender
email address when sending group SMTP replies via PostAlerter.
However, this was not ordered, and since there is no created_at
column on TopicAllowedGroup we cannot order this nicely, which
caused just a random group to be used (based on whatever postgres
decided it felt like that morning).
This commit changes the group used for SMTP sending to be the
group using the email_username of the to address of the first
incoming email for the topic, if there are more than one allowed
groups on the topic. Otherwise it just uses the only SMTP enabled
group.
Sometimes, a user may have a malformed email such as
`test@test.com<mailto:test@test.com` their email address,
and as a topic participant will be included as a CC email
when sending a GroupSmtpEmail. This causes the CC parsing to
fail and further down the line in Email::Sender the code
to check the CC addresses expects an array but gets a string
instead because of the parse failure.
Instead, we can just check if the CC addresses are valid
and drop them if they are not in the GroupSmtpEmail job.
Uppy and Resumable slice up their chunks differently, which causes a difference
in this algorithm. Let's take a 131.6MB file (137951695 bytes) with a 5MB (5242880 bytes)
chunk size. For resumable, there are 26 chunks, and uppy there are 27. This is
controlled by forceChunkSize in resumable which is false by default. The final
chunk size is 6879695 (chunk size + remainder) whereas in uppy it is 1636815 (just remainder).
This means that the current condition of uploaded_file_size + current_chunk_size >= total_size
is hit twice by uppy, because it uses a more correct number of chunks. This
can be solved for both uppy and resumable by checking the _previous_ chunk
number * chunk_size as the uploaded_file_size.
An example of what is happening before that change, using the current
chunk number to calculate uploaded_file_size.
chunk 26: resumable: uploaded_file_size (26 * 5242880) + current_chunk_size (6879695) = 143194575 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (5242880) = 141557760 >= total_size (137951695) ? YES
chunk 27: uppy: uploaded_file_size (27 * 5242880) + current_chunk_size (1636815) = 143194575 >= total_size (137951695) ? YES
An example of what this looks like after the change, using the previous
chunk number to calculate uploaded_file_size:
chunk 26: resumable: uploaded_file_size (25 * 5242880) + current_chunk_size (6879695) = 137951695 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (25 * 5242880) + current_chunk_size (5242880) = 136314880 >= total_size (137951695) ? NO
chunk 27: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (1636815) = 137951695 >= total_size (137951695) ? YES
Same issue as 28b00dc6fc, the
Mocha::ExpectationError inherits from Exception instead
of StandardError so RspecErrorTracker does not show the
actual failed expectation in request specs, the status of
the response is just 500 with no further detail.
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.
To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:
* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts
Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.
Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.
This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
We are only using list_multipart_parts right now in the
uploads controller for multipart uploads to check if the
upload exists; thus we don't need up to 1000 parts.
Also adding a note for future explorers that list_multipart_parts
only gets 1000 parts max, and adding params for max parts
and starting parts.
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:
* assets
* direct S3 backups
* direct S3 uploads
This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).
As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.
This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
Since db7be947df the functionality
to not save a bookmark when pressing ESC on the modal has been
broken, because a new initiatedByESC event was introduced there
in d-modal instead of clicking on the modal close button. This
commit fixes the issue by adding an initiatedByESC property
to onClose inside application.js, and checking for that OR
initiatedByCloseButton in the bookmark modal.
This commit, while changing the watched word uploader to use
uppy, also fixes a minor bug with the UppyUploadMixin where
the file input's value was not cleared after reset, which
prevented subsequent file uploads. The composer mixin already
has this fix.
This commit changes the emoji uploader to use the UppyUploadMixin,
and makes some minor changes to the emoji uploader (tightening the
copy for drag and drop and adding a percentage for the upload).
Since no other uppy upload mixin components have needed to upload
multiple files so far, this necessitated adding a tracker for the
in progress uploads so we know when to reset the uploader once all
uploads are complete.
At the moment, the emoji uploader cannot be used for direct S3 uploads
because the admin emoji controller creates other records and does other
magic with the emojis. At some point we need to factor this kind of thing
into the ExternalUploadManager.transform! action to complete external
uploads.
* DEV: Output webmock errors in request specs
In request specs, if you had not properly mocked an external
HTTP call, you would end up with a 500 error with no further
information instead of your expected response code, with an
rspec output like this:
```
Failures:
1) UploadsController#generate_presigned_put when the store is external generates a presigned URL and creates an external upload stub
Failure/Error: expect(response.status).to eq(200)
expected: 200
got: 500
(compared using ==)
# ./spec/requests/uploads_controller_spec.rb:727:in `block (4 levels) in <top (required)>'
# ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
```
This is not helpful at all when you want to find what you actually
failed to mock, which is shown straight away in non-request specs.
This commit introduces a rescue_from block in the application
controller to log this error, so we have a much nicer output that
helps the developer find the issue:
```
Failures:
1) UploadsController#generate_presigned_put when the store is external generates a presigned URL and creates an external upload stub
Failure/Error: expect(response.status).to eq(200)
expected: 200
got: 500
(compared using ==)
# ./spec/requests/uploads_controller_spec.rb:727:in `block (4 levels) in <top (required)>'
# ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# WebMock::NetConnectNotAllowedError:
# Real HTTP connections are disabled. Unregistered request: GET https://s3-upload-bucket.s3.us-west-1.amazonaws.com/?cors with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'', 'Authorization'=>'AWS4-HMAC-SHA256 Credential=some key/20211101/us-west-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=test', 'Host'=>'s3-upload-bucket.s3.us-west-1.amazonaws.com', 'User-Agent'=>'aws-sdk-ruby3/3.121.2 ruby/2.7.1 x86_64-linux aws-sdk-s3/1.96.1', 'X-Amz-Content-Sha256'=>'test', 'X-Amz-Date'=>'20211101T035113Z'}
#
# You can stub this request with the following snippet:
#
# stub_request(:get, "https://s3-upload-bucket.s3.us-west-1.amazonaws.com/?cors").
# with(
# headers: {
# 'Accept'=>'*/*',
# 'Accept-Encoding'=>'',
# 'Authorization'=>'AWS4-HMAC-SHA256 Credential=some key/20211101/us-west-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=test',
# 'Host'=>'s3-upload-bucket.s3.us-west-1.amazonaws.com',
# 'User-Agent'=>'aws-sdk-ruby3/3.121.2 ruby/2.7.1 x86_64-linux aws-sdk-s3/1.96.1',
# 'X-Amz-Content-Sha256'=>'test',
# 'X-Amz-Date'=>'20211101T035113Z'
# }).
# to_return(status: 200, body: "", headers: {})
#
# registered request stubs:
#
# stub_request(:head, "https://s3-upload-bucket.s3.us-west-1.amazonaws.com/")
#
# ============================================================
```
* DEV: Require webmock in application controller if rails.env.test
* DEV: Rescue from StandardError and NetConnectNotAllowedError
When deleting a for_topic bookmark, we were calling
bookmark.attachedTo() for the bookmarks:changed event,
but the bookmark was not always a Bookmark model instance,
so sometimes that call would error. Now we make sure that
the bookmarks in the topic.bookmarks JS array are all
bookmark model instances, and added a test to cover this
deleting for_topic bookmark case as well.
In preparation for adding automatic CORS rules creation
for direct S3 uploads, I am adding tests here and moving the
CORS rule definitions into a dedicated class so they are all
in the one place.
There is a problem with ensure_cors! as well -- if there is
already a CORS rule defined (presumably the asset one) then
we do nothing and do not apply the new rule. This means that
the S3BackupStore.ensure_cors method does nothing right now
if the assets rule is already defined, and it will mean the
same for any direct S3 upload rules I add for uppy. We need
to be able to add more rules, not just one.
This is not a problem on our hosting because we define the
rules at an infra level.
This commit adds uploadHandler support to composer uploads using
uppy. The only things we have that are using this are discourse-brightcove and
discourse-video, which both pop modal windows to handle the file upload and
completely leave out all the composer-type flows. This implementation simply
follows the existing one, where if a single file is uploaded and there
is a matching upload handler we take control away from uppy and hand
it off to the upload handler.
Trying to get this kind of thing working within uppy would require a few
changes because they have no way to restrict uploaders to certain file types
and with the way their uploaders are run it doesn't look like it would be easy
to add this either, so I don't think this is worth the work unless at some
point in the future we plan to have more upload handler integrations.
I also fixed an issue with `cleanUpComposerUploadHandler` which is used
in tests to reset the state of `uploadHandlers` in the composer. This
was doing `uploadHandlers = []` to clear that array, but that creates
a brand new array so anything else referencing the original array will
lose that reference. Better to set `uploadHandlers.length = 0` to
clear it. This was breaking the tests I added to see if upload handlers
were working.
Previously, `loadLibs` was called inside the `optimize` function of
the media-optimization-worker, which meant that it could be hit
multiple times causing load errors (as seen in b69c2f7311)
This commit moves that call to a specific message handler (the `install` message)
for the service worker, and refactors the service for the media-optimization-worker
to wait for this installation to complete before continuing with processing
image optimizations.
This way, we know for sure based on promises and worker messages
that the worker is installed and has all required libraries
loaded before we continue on with attempting any processing. The
change made in b69c2f7311 is no
longer needed with this commit.
This new app event will fire whenever a bookmark is created,
edited, or deleted for a post or topic, and replaces these old
app events which had inconsistent APIs:
* page:bookmark-post-toggled
* topic:bookmark-toggled
When the event is triggered, the arguments are in this order:
1. bookmark - The bookmark record created or changed. Will be null
if the bookmark was deleted.
2. target - Object with target (post or topic) and targetId (post ID
or topic ID)
When inviting a group to a topic, there may be members of
the group already in the topic as topic allowed users. These
can be safely removed from the topic, because they are implicitly
allowed in the topic based on their group membership.
Also, this prevents issues with group SMTP emails, which rely
on the topic_allowed_users of the topic to send to and cc's
for emails, and if there are members of the group as topic_allowed_users
then that complicates things and causes odd behaviour.
We also ensure that the OP of the topic is not removed from
the topic_allowed_users when a group they belong to is added,
as it will make it harder to add them back later.
We had code to open the bookmark modal in two places -- the bookmark
list and also from within a topic. This caused the two code paths to
drift, as in the bookmark list we were not passing in the forTopic or
autoDeletePreferences data into the modal, and we were also not refreshing
the bookmark list when the bookmark was deleted from within the modal.
This commit moves the modal opening code into an importable
function from the controllers/bookmark module, and all callers
have to do is pass it an instance of Bookmark and also options
for what to do for the following:
* onAfterSave
* onAfterDelete
* onCloseWithoutSaving
Previously when clicking the Delete button for small action posts
there was no way to recover this post if the action was accidental.
Now if canRecover is true on the post, which it is just after it
is deleted and the post is fetched from the server again, we show
an undo button which calls the recover endpoint for the post.
We also now disallow the editing of the post if it is deleted, and
show the proper deleted red CSS on the small action post when deleted.
This commit removes the recipient's username from the
respond to / participants list that is shown at the bottom
of user notification emails. For example if the recipient's
username was jsmith, and there were participants ljones and
bmiller, we currently show this:
> "reply to this email to respond to jsmith, ljones, bmiller"
or
> "Participants: jsmith, ljones, bmiller"
However this is a bit redundant, as you are not replying to
yourself here if you are the recipient user. So we omit the
recipient user's username from this list, which is only used
in the text of the email and not elsewhere.
Instead of using image-uploader, which relies on the old
UploadMixin, we can now use the uppy-image-uploader which
uses the new UppyUploadMixin which is stable enough and
supports both regular XHR uploads and direct S3 uploads,
controlled by a site setting (default to XHR).
At some point it may make sense to rename uppy-image-uploader
back to image-uploader, once we have gone through plugins
etc. and given a bit of deprecation time period.
This commit also fixes `for_private_message`, `for_site_setting`,
and `pasted` flags not being sent via uppy uploads onto the
UploadCreator, both via regular XHR uploads and also through
external/multipart uploads.
The uploaders changed are:
* site setting images
* badge images
* category logo
* category background
* group flair
* profile background
* profile card background
This commit makes the following change to the Edit Bookmark
modal window for clarity:
* If the user is editing an existing bookmark without a reminder set,
hide the "none needed" option. This will draw more attention to the
delete button.
* If the user is editing an existing bookmark with a reminder set for the
future, change the "none needed" option to say "remove reminder, keep bookmark"
To do this, I needed to provide an option to override the labels
for time shortcuts in certain cases, so I could keep the NONE shortcut
but have the different wording.
When we are calling the loadLibs function, which in turn calls:
importScripts(settings.mozjpeg_script);
importScripts(settings.resize_script);
For the media-optimization-worker service worker, we are getting
an error in Firefox, which balks at wasm_bindgen, a global
variable defined with let, being redefined when the module loads.
This causes image processing to fail in Firefox when more than one
image is uploaded at a time.
The solution to this is to just check whether the scripts are
already imported, and if so do not import them again.
Chrome doesn't seem to care about this variable redefinition
and does not error, and it seems to be expected behaviour that
the script can be loaded multiple times (see https://github.com/w3c/ServiceWorker/issues/1041)
This commit bumps the following uppy modules:
* @uppy/aws-s3
* @uppy/aws-s3-multipart
* @uppy/core
* @uppy/drop-target
* @uppy/xhr-upload
This is done so we can use the new functionality for retrying
failed prepareUploadParts calls, introduced in
https://github.com/transloadit/uppy/commit/e435f4a91747db8116e1c9b5dd37de25bf847f43.
I also needed to make some changes to composer-upload-uppy to
support this retrying, while at the same time being able to
throw a bootbox with the error message if the number of retries
are exceeded.
Firefox does not return a PerformanceMeasure object when using
performance.mark and performance.measure, even though MDN says it
should https://developer.mozilla.org/en-US/docs/Web/API/Performance/measure#return_value
So for now, we disable the upload instrumentation with a test
to see if a PerformanceMeasure (or anything really) is returned.
When creating a reply after already navigating out of the
topic (e.g. open the reply composer, go to a different topic,
then create the post), the _removeDeleteOnOwnerReplyBookmarks
function was erroring because it relied on the topic model
being present.
We can skip this function altogether if the topic model is _not_
present, because the PostCreator already takes care of deleting
bookmarks with the on_owner_reply auto_delete_preference. The
_removeDeleteOnOwnerReplyBookmarks function just cleans up the
in-memory post stream and topic model.
In the user bookmark list, when we show the excerpt of the bookmark
(which is usually just the bookmarked post excerpt), we want to show
the first unread post's excerpt instead for for_topic bookmarks. This
is because when the user clicks on that bookmark link, they are taken
to the first unread post in the topic, not the OP, as per:
27699648ef
Martin Brennan