## Overview
This PR introduces comprehensive search functionality for chat messages,
enabling users to search through their chat history both globally across
all accessible channels and within specific channels.
### Search Capabilities
**All-Channel Search**: When no channel is specified, users can search
across all channels they have access to. The search respects channel
permissions through `ChannelFetcher.all_secured_channel_ids`, ensuring
users only see results from channels they can view.
**Per-Channel Search**: Users can scope their search to a specific
channel by providing a `channel_id` parameter, useful for finding
messages within a particular conversation context.
**Search Features**:
- Full-text search using PostgreSQL's tsvector/tsquery
- Advanced filters: `@username` to filter by author, `#channel` to
filter by channel slug
- Sort options: relevance (default) or latest
- Pagination support
- Search data weighted by relevance
## Site Setting: `chat_search_enabled`
This feature is gated behind the `chat_search_enabled` site setting,
which is currently:
- **Default**: `false`
- **Hidden**: `true`
- **Client-accessible**: `true`
### Deployment Strategy
Due to the need for chat messages to be indexed before search becomes
useful, we're implementing a two-phase deployment:
**Phase 1 (Initial Merge)**:
- `chat_search_enabled` remains `false` and hidden
- The `register_search_index` uses default (true) instead of `chat_search_enabled` value
- This allows the reindexing infrastructure to begin indexing existing
chat messages even if we don't show the UI yet
**Wait Period**:
- Wait at least one week after Phase 1 deployment
- `Jobs::ReindexSearch` runs every 2 hours and will progressively index
all chat messages
- This ensures most sites have a significant part of their chat history indexed
**Phase 2 (Follow-up Merge)**:
- Set `chat_search_enabled` default to `true` and unhide it
- Update the `register_search_index` enabled proc uses the default
(true) instead of using the `chat_search_enabled` setting
- Users can now access search with pre-indexed data
**Rationale**: Without this phased approach, users would see the search
UI immediately but receive no results until the reindexing job runs,
creating a confusing experience. By pre-indexing while the UI is hidden,
we ensure search works immediately when enabled.
## New Plugin API: `register_search_index`
This PR introduces a new plugin API that allows plugins to register
custom search indexes that integrate seamlessly with Discourse's search
infrastructure.
### API Signature
```ruby
register_search_index(
model_class:, # The ActiveRecord model to index
search_data_class:, # The model for storing search data
index_version:, # Version number for re-indexing
search_data:, # Proc that returns weighted search data
load_unindexed_record_ids:,# Proc that finds records needing indexing
enabled: # Optional proc to enable/disable (default: -> { true })
)
```
### How It Works
**Integration with SearchIndexer**: When `SearchIndexer.index(obj)` is
called, it checks registered search handlers for the object's type. If a
handler matches, it:
1. Calls the `search_data` proc with the object and an `IndexerHelper`
instance
2. Receives weighted search data (`:a_weight`, `:b_weight`, `:c_weight`,
`:d_weight`)
3. Updates the corresponding search data table with PostgreSQL's
tsvector
**Integration with Jobs::ReindexSearch**: The scheduled job (runs every
2 hours) calls `rebuild_registered_search_handlers`, which:
1. Iterates through all registered search handlers
2. Skips handlers where `enabled` proc returns `false`
3. Calls `load_unindexed_record_ids` to find records needing indexing
4. Indexes up to `limit` records per handler (default: 10,000)
### Chat Implementation Example
```ruby
register_search_index(
model_class: Chat::Message,
search_data_class: Chat::MessageSearchData,
index_version: 1,
search_data: proc { |message, indexer_helper|
{
a_weight: message.message,
d_weight: indexer_helper.scrub_html(message.cooked)[0..600_000]
}
},
load_unindexed_record_ids: proc { |limit:, index_version:|
Chat::Message
.joins("LEFT JOIN chat_message_search_data ON chat_message_id = chat_messages.id")
.where(
"chat_message_search_data.locale IS NULL OR
chat_message_search_data.locale != ? OR
chat_message_search_data.version != ?",
SiteSetting.default_locale,
index_version
)
.order("chat_messages.id ASC")
.limit(limit)
.pluck(:id)
}
)
```
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
Co-authored-by: Loïc Guitaut <5648+Flink@users.noreply.github.com>
In `Chat::ChannelFetcher.secured_direct_message_channels_search`,
`User.preload_custom_fields` is called with `channels.flat_map {
_1.chatable.users }`. However,
the `Chat::DirectMessageSerializer` was getting the users via
`object.direct_message_users.map(&:user)` which uses the
`Chat::DirectMessage.direct_message_users` scope instead of the
`Chat::DirectMessage.users` scope resulting in ActiveRecord returning
new `User` objects that do not have user custom fields preloaded.
This fixes the appearance of AI generated gists in the topic list on
/filter, and also includes the gist toggle via a new plugin outlet on
/filter called `after-filter-navigation-menu`.
This shares the state with the discovery route toggle (which is separate
from PM toggle state). I've also added tests to cover gist appearance on
/filter. Before the /filter state was shared with PM state, which was
incorrect.
Before:
<img width="2232" height="424" alt="image"
src="https://github.com/user-attachments/assets/b22ed7a6-388e-4b88-99d4-e9e10af6275a"
/>
After:
<img width="2272" height="506" alt="image"
src="https://github.com/user-attachments/assets/cae03f12-9153-4bcb-a9d3-52712fb2d945"
/>
Having it in d-ai's plugin.rb file solves it when running plugin tests.
But when running core tests, plugins are not loaded, but the tables
still exist in the database.
Followup to 6247fdc255
This is a follow up to cf4193e6e1.
When chat is being initialized, we initiate an async request to
`/chat/api/me/channels` via `this.chat.loadChannels` but do not await on the request to be completed.
This is fine when a user is not visiting a chat channel route directly.
However, not awaiting on `this.chat.loadChannels` can cause problems
like a user's thread list or drafts to not be displayed if the
`/chat/api/me/channels` request does not return before rendering
happens. To resolve this, we will now wait for the promise in
`this.chat.loadChannels` to resolve before allowing rendering to happen on the `ChatChannelRoute`.
This change adds a new `ReviewableActionBuilder#build_bundle` helper for
quickly defining action bundles that can be performed on reviewables.
`ReviewableActionBuilder#build_action` has also been updated to allow
plugin-defined actions to appear correctly.
The core reviewable types have been updated to use this new method, and
I've also added support for reviewable chat messages, to demonstrate
plugin support.
Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
Since the new modifier has been added, some specs were not using using
the focus check on the composer (included in `fill_composer` method) and
we were actually not focused which was causing these specs to fail.
Adds a safe inset bottom for lightbox captions on mobile and removes
image padding on small screens. This change also applies bottom
depending on whether caption is being set or not.
How it looks:
<img width="225" height="487" alt="IMG_9334"
src="https://github.com/user-attachments/assets/d92d379a-ad63-44d0-8320-a0861b2a10e4"
/>
When we updated the list of available/supported holidays regions in
eabbac18cf and in
1983a44812 we left some records with
invalid regions.
This adds a migration to correct old records, as well as adding an error
handling to log an error when a region is invalid, rather than throwing
an exception and breaking the background job.
Ref - https://meta.discourse.org/t/-/384873
## Summary
- Adds estimated completion time (ETA) display for AI translation
backfill progress
- ETA is calculated based on the configured hourly rate and remaining
untranslated posts
- Display format automatically adjusts based on time remaining (minutes,
hours, or days)
- ETA appears inline with the post count for better UX
## Implementation Details
- Backend sends `hourly_rate` from site settings to frontend
- Frontend calculates total remaining posts across all locales
- ETA calculation: `remaining_posts / hourly_rate`
- Format logic:
- < 1 hour: shows minutes
- 1-24 hours: shows hours
- \> 24 hours: shows days
- Only displays when backfill is enabled and there are posts remaining
<img width="1243" height="242" alt="image"
src="https://github.com/user-attachments/assets/314bc399-9694-43ec-8591-0237c1b902f6"
/>
1. Update `register_preloaded_category_custom_fields` API to be
reload-safe
2. Update post-voting to use this in all cases, and remove the
`respond_to?` checks since this plugin is now bundled with core
Adds missing data attributes that PhotoSwipe requires for lightboxing
quoted images. This is a follow up to #35442.
In core we handle this within the cooked post processor, but in chat our
message processor is more lightweight and doesn't have the same handling
for lightbox images.
---------
Co-authored-by: Yuriy Kurant <yuriy@discourse.org>
... you don't have access to.
Before creating the notification for an event, we weren't checking
whether the invitee had access to the topic, thus generating phantom
notification (aka. a number in the hamburger menu, but no notification
item in any lists).
Also made sure that creating a private event without any invitees, does
not generate invites for ... everyone 😅
This will sort the bars of the AI translation chart in descending order
based on completion %.
I've also made an adjustment to show a single decimal of precision
between 99 and 100% so we're not rounding up to 100% when there are
still a small number of remaining posts to translate.
Follow-up to be8b2b3, which wasn't quite enough adjustment to get the
desired outcome. At the moment it's correctly showing how many posts
need to be translated (1203), but it's still showing the total of posts
in the default locale (190284), rather than distinguishing the total
number of posts that need translating in that language.
<img width="2222" height="482" alt="image"
src="https://github.com/user-attachments/assets/0e4e1bee-0033-4b98-8f0c-b90d95845203"
/>
This PR further adjusts the chart to show "remaining work" for each
locale, rather than showing the total number of posts. So the outcome
should be more like
<img width="2210" height="638" alt="image"
src="https://github.com/user-attachments/assets/baa099d4-20f0-40bc-8fbb-3d2d5308e97a"
/>
I've also updated the tooltip on the chart to clarify that this is the
number of posts not in the language that need to be translated into it
We need to define #targets, or the problem check code will create a
tracker without a target when the check passes. This bug causes the
check to ignore the perform_every constraint.
The chart for AI translations included all posts for the default locale,
which would make the rest of the chart fairly disproportionate
<img width="1766" height="1228" alt="image"
src="https://github.com/user-attachments/assets/63298222-ca41-4874-8103-d5c0dae05795"
/>
In this PR I'm only showing the posts in the default locale that need
work done, which will be a much smaller number, resulting in a chart
that better represents actual translation work needed. Worst case is
that there's a very small number of posts to be translated to the
default locale, but overall this should better represent the work to be
done.
(placeholder data shown here)
<img width="2210" height="730" alt="image"
src="https://github.com/user-attachments/assets/66e9ccb7-19f4-4aa7-addb-1b3f8f6a6dfc"
/>
I've also added a note to explain that we're not showing all the posts
in the default locale, only the ones needing translation.
Our current implementation of lightbox uses Magnific Popup, which is now
deprecated and will only receive critical/security bug fixes. Magnific
also relies on jQuery, which we would like to remove where possible
throughout our codebase.
After looking at various options, the general consensus was that
PhotoSwipe was a close match to what we need.
Regular image types that are supported with our current lightbox (jpg,
png etc) will work with their existing cooked markup (no need to rebake
posts). This PR also adds support for SVG images and markup from various
theme components (ie. Discourse Mermaid).
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
This creates an overriding `rswag:specs:swaggerize` rake task that also
adds plugin paths, and updates spec helpers to handle plugin paths.
Also adds the spec files for the discourse-calendar events index
endpoint.
### Testing
Running `rake rswag:specs:swaggerize` now generates the same
`openapi/openapi.yaml` file, with `/discourse-post-event/events.ics` and
`/discourse-post-event/events.json` GET documentation.
Previously manual user connect/revoke in preferences were both enabled
permanently based on the default values in the ManagedAuthenticator
class. This adds settings so either one can be explicitly disabled.
Enforces the gap cursor using the new `createGapCursor: true` node spec
prop to the following block nodes:
- `blockquote`
- `quote`
- `code_block`
- `html_block`
- `spoiler`
The validity checking for parameter values being passed in expects them
all to be strings, then casts them to their appropriate type after
confirming that they're valid.
Rather than bypassing these checks, it's safer to convert the values to
strings, then continue with the existing validity checks.
ActiveRecord prints these warnings when loading the db schema into its
internal cache. We don't need to do this for the AI embeddings tables,
since we don't use ActiveRecord models for them.
Reverts the system-spec-specific workaround from 446ff04e
Chat::MessageProcessor includes CookedProcessorMixin but only sets
@model, leaving @post as nil. The process_hotlinked_image method was
attempting to access @post.post_hotlinked_media, causing undefined
method errors when processing chat messages with images.
With out AI credit system being rolled out, we no longer rely on
allowing/disallowing seeded models in the enumerator. This update allows
seeded models to be selected and used in the `LlmEnumerator`.
We markup video thumbnail `div`s in posts with `tabindex="0"` which
makes them tabbable, but without a role it's unclear they're
interactive.
Since on click or keypress these play the video, it should have
`role="button"` as well as an an accessible label.
I also noticed that `onKeyPress` was added here but isn't wired up to
anything... so I've included that and also added the space key as a
trigger in addition to enter.
David Taylor
Joffrey JAFFEUX
Jarek Radosz
Alan Guo Xiang Tan
Kris
Krzysztof Kotlarek
Gary PendergastandKrzysztof Kotlarek
chapoiandawesomerobot
David Battersby
Roman Rizzi
Régis Hanol
Keegan George
Rafael dos Santos Silva
Ted Johansson
Kelv
Chris Alberti
Renato Atilio
Discourse Translator Bot
Arpit Jalan