FEATURE: Generic hashtag autocomplete lookup and markdown cooking (#18937)

This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.

**Serverside**

We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.

Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.

The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.

When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.

**Markdown**

The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.

This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.

**Chat Channels**

This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.

------

**Known Rough Edges**

- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil

spec/services/hashtag_autocomplete_service_spec.rb

@@ -10,8 +10,26 @@ RSpec.describe HashtagAutocompleteService do
 
   before { Site.clear_cache }
 
-  def register_bookmark_data_source
-    HashtagAutocompleteService.register_data_source("bookmark") do |guardian_scoped, term, limit|
+  class BookmarkDataSource
+    def self.icon
+      "bookmark"
+    end
+
+    def self.lookup(guardian_scoped, slugs)
+      guardian_scoped
+        .user
+        .bookmarks
+        .where("LOWER(name) IN (:slugs)", slugs: slugs)
+        .map do |bm|
+          HashtagAutocompleteService::HashtagItem.new.tap do |item|
+            item.text = bm.name
+            item.slug = bm.name.gsub(" ", "-")
+            item.icon = icon
+          end
+        end
+    end
+
+    def self.search(guardian_scoped, term, limit)
       guardian_scoped
         .user
         .bookmarks
@@ -21,12 +39,31 @@ RSpec.describe HashtagAutocompleteService do
           HashtagAutocompleteService::HashtagItem.new.tap do |item|
             item.text = bm.name
             item.slug = bm.name.gsub(" ", "-")
-            item.icon = "bookmark"
+            item.icon = icon
           end
         end
     end
   end
 
+  describe ".contexts_with_ordered_types" do
+    it "returns a hash of all the registrered search contexts and their types in the defined priority order" do
+      expect(HashtagAutocompleteService.contexts_with_ordered_types).to eq(
+        { "topic-composer" => %w[category tag] },
+      )
+      HashtagAutocompleteService.register_type_in_context("category", "awesome-composer", 50)
+      HashtagAutocompleteService.register_type_in_context("tag", "awesome-composer", 100)
+      expect(HashtagAutocompleteService.contexts_with_ordered_types).to eq(
+        { "topic-composer" => %w[category tag], "awesome-composer" => %w[tag category] },
+      )
+    end
+  end
+
+  describe ".data_source_icons" do
+    it "gets an array for all icons defined by data sources so they can be used for markdown allowlisting" do
+      expect(HashtagAutocompleteService.data_source_icons).to eq(%w[folder tag])
+    end
+  end
+
   describe "#search" do
     it "returns search results for tags and categories by default" do
       expect(subject.search("book", %w[category tag]).map(&:text)).to eq(
@@ -41,7 +78,9 @@ RSpec.describe HashtagAutocompleteService do
     end
 
     it "respects the limit param" do
-      expect(subject.search("book", %w[tag category], limit: 1).map(&:text)).to eq(["great-books x 0"])
+      expect(subject.search("book", %w[tag category], limit: 1).map(&:text)).to eq(
+        ["great-books x 0"],
+      )
     end
 
     it "does not allow more than SEARCH_MAX_LIMIT results to be specified by the limit param" do
@@ -59,7 +98,9 @@ RSpec.describe HashtagAutocompleteService do
 
     it "includes the tag count" do
       tag1.update!(topic_count: 78)
-      expect(subject.search("book", %w[tag category]).map(&:text)).to eq(["great-books x 78", "Book Club"])
+      expect(subject.search("book", %w[tag category]).map(&:text)).to eq(
+        ["great-books x 78", "Book Club"],
+      )
     end
 
     it "does case-insensitive search" do
@@ -71,6 +112,11 @@ RSpec.describe HashtagAutocompleteService do
       )
     end
 
+    it "can search categories by name or slug" do
+      expect(subject.search("book-club", %w[category]).map(&:text)).to eq(["Book Club"])
+      expect(subject.search("Book C", %w[category]).map(&:text)).to eq(["Book Club"])
+    end
+
     it "does not include categories the user cannot access" do
       category1.update!(read_restricted: true)
       expect(subject.search("book", %w[tag category]).map(&:text)).to eq(["great-books x 0"])
@@ -86,20 +132,7 @@ RSpec.describe HashtagAutocompleteService do
       Fabricate(:bookmark, user: user, name: "cool rock song")
       guardian.user.reload
 
-      HashtagAutocompleteService.register_data_source("bookmark") do |guardian_scoped, term, limit|
-        guardian_scoped
-          .user
-          .bookmarks
-          .where("name ILIKE ?", "%#{term}%")
-          .limit(limit)
-          .map do |bm|
-            HashtagAutocompleteService::HashtagItem.new.tap do |item|
-              item.text = bm.name
-              item.slug = bm.name.dasherize
-              item.icon = "bookmark"
-            end
-          end
-      end
+      HashtagAutocompleteService.register_data_source("bookmark", BookmarkDataSource)
 
       expect(subject.search("book", %w[category tag bookmark]).map(&:text)).to eq(
         ["Book Club", "great-books x 0", "read review of this fantasy book"],
@@ -112,6 +145,7 @@ RSpec.describe HashtagAutocompleteService do
       expect(subject.search("book", %w[category tag]).map(&:ref)).to eq(
         %w[hobbies:book-club great-books],
       )
+      category1.update!(parent_category: nil)
     end
 
     it "appends type suffixes for the ref on conflicting slugs on items that are not the top priority type" do
@@ -123,7 +157,7 @@ RSpec.describe HashtagAutocompleteService do
       Fabricate(:bookmark, user: user, name: "book club")
       guardian.user.reload
 
-      register_bookmark_data_source
+      HashtagAutocompleteService.register_data_source("bookmark", BookmarkDataSource)
 
       expect(subject.search("book", %w[category tag bookmark]).map(&:ref)).to eq(
         %w[book-club book-club::tag great-books book-club::bookmark],
@@ -151,4 +185,156 @@ RSpec.describe HashtagAutocompleteService do
       end
     end
   end
+
+  describe "#lookup_old" do
+    fab!(:tag2) { Fabricate(:tag, name: "fiction-books") }
+
+    it "returns categories and tags in a hash format with the slug and url" do
+      result = subject.lookup_old(%w[book-club great-books fiction-books])
+      expect(result[:categories]).to eq({ "book-club" => "/c/book-club/#{category1.id}" })
+      expect(result[:tags]).to eq(
+        {
+          "fiction-books" => "http://test.localhost/tag/fiction-books",
+          "great-books" => "http://test.localhost/tag/great-books",
+        },
+      )
+    end
+
+    it "does not include categories the user cannot access" do
+      category1.update!(read_restricted: true)
+      result = subject.lookup_old(%w[book-club great-books fiction-books])
+      expect(result[:categories]).to eq({})
+    end
+
+    it "does not include tags the user cannot access" do
+      Fabricate(:tag_group, permissions: { "staff" => 1 }, tag_names: ["great-books"])
+      result = subject.lookup_old(%w[book-club great-books fiction-books])
+      expect(result[:tags]).to eq({ "fiction-books" => "http://test.localhost/tag/fiction-books" })
+    end
+
+    it "handles tags which have the ::tag suffix" do
+      result = subject.lookup_old(%w[book-club great-books::tag fiction-books])
+      expect(result[:tags]).to eq(
+        {
+          "fiction-books" => "http://test.localhost/tag/fiction-books",
+          "great-books" => "http://test.localhost/tag/great-books",
+        },
+      )
+    end
+
+    context "when not tagging_enabled" do
+      before { SiteSetting.tagging_enabled = false }
+
+      it "does not return tags" do
+        result = subject.lookup_old(%w[book-club great-books fiction-books])
+        expect(result[:categories]).to eq({ "book-club" => "/c/book-club/#{category1.id}" })
+        expect(result[:tags]).to eq({})
+      end
+    end
+  end
+
+  describe "#lookup" do
+    fab!(:tag2) { Fabricate(:tag, name: "fiction-books") }
+
+    it "returns category and tag in a hash format with the slug and url" do
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(["book-club"])
+      expect(result[:category].map(&:relative_url)).to eq(["/c/book-club/#{category1.id}"])
+      expect(result[:tag].map(&:slug)).to eq(%w[fiction-books great-books])
+      expect(result[:tag].map(&:relative_url)).to eq(%w[/tag/fiction-books /tag/great-books])
+    end
+
+    it "does not include category the user cannot access" do
+      category1.update!(read_restricted: true)
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:category]).to eq([])
+    end
+
+    it "does not include tag the user cannot access" do
+      Fabricate(:tag_group, permissions: { "staff" => 1 }, tag_names: ["great-books"])
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:tag].map(&:slug)).to eq(%w[fiction-books])
+      expect(result[:tag].map(&:relative_url)).to eq(["/tag/fiction-books"])
+    end
+
+    it "handles type suffixes for slugs" do
+      result =
+        subject.lookup(%w[book-club::category great-books::tag fiction-books], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(["book-club"])
+      expect(result[:category].map(&:relative_url)).to eq(["/c/book-club/#{category1.id}"])
+      expect(result[:tag].map(&:slug)).to eq(%w[fiction-books great-books])
+      expect(result[:tag].map(&:relative_url)).to eq(%w[/tag/fiction-books /tag/great-books])
+    end
+
+    it "handles parent:child category lookups" do
+      parent_category = Fabricate(:category, name: "Media", slug: "media")
+      category1.update!(parent_category: parent_category)
+      result = subject.lookup(%w[media:book-club], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(["book-club"])
+      expect(result[:category].map(&:ref)).to eq(["media:book-club"])
+      expect(result[:category].map(&:relative_url)).to eq(["/c/media/book-club/#{category1.id}"])
+      category1.update!(parent_category: nil)
+    end
+
+    it "does not return the category if the parent does not match the child" do
+      parent_category = Fabricate(:category, name: "Media", slug: "media")
+      category1.update!(parent_category: parent_category)
+      result = subject.lookup(%w[bad-parent:book-club], %w[category tag])
+      expect(result[:category]).to be_empty
+    end
+
+    it "for slugs without a type suffix it falls back in type order until a result is found or types are exhausted" do
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(["book-club"])
+      expect(result[:category].map(&:relative_url)).to eq(["/c/book-club/#{category1.id}"])
+      expect(result[:tag].map(&:slug)).to eq(%w[fiction-books great-books])
+      expect(result[:tag].map(&:relative_url)).to eq(%w[/tag/fiction-books /tag/great-books])
+
+      category2 = Fabricate(:category, name: "Great Books", slug: "great-books")
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(%w[book-club great-books])
+      expect(result[:category].map(&:relative_url)).to eq(
+        ["/c/book-club/#{category1.id}", "/c/great-books/#{category2.id}"],
+      )
+      expect(result[:tag].map(&:slug)).to eq(%w[fiction-books])
+      expect(result[:tag].map(&:relative_url)).to eq(%w[/tag/fiction-books])
+
+      category1.destroy!
+      Fabricate(:tag, name: "book-club")
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+      expect(result[:category].map(&:slug)).to eq(["great-books"])
+      expect(result[:category].map(&:relative_url)).to eq(["/c/great-books/#{category2.id}"])
+      expect(result[:tag].map(&:slug)).to eq(%w[book-club fiction-books])
+      expect(result[:tag].map(&:relative_url)).to eq(%w[/tag/book-club /tag/fiction-books])
+
+      result = subject.lookup(%w[book-club great-books fiction-books], %w[tag category])
+      expect(result[:category]).to eq([])
+      expect(result[:tag].map(&:slug)).to eq(%w[book-club fiction-books great-books])
+      expect(result[:tag].map(&:relative_url)).to eq(
+        %w[/tag/book-club /tag/fiction-books /tag/great-books],
+      )
+    end
+
+    it "includes other data sources" do
+      Fabricate(:bookmark, user: user, name: "read review of this fantasy book")
+      Fabricate(:bookmark, user: user, name: "coolrock")
+      guardian.user.reload
+
+      HashtagAutocompleteService.register_data_source("bookmark", BookmarkDataSource)
+
+      result = subject.lookup(["coolrock"], %w[category tag bookmark])
+      expect(result[:bookmark].map(&:slug)).to eq(["coolrock"])
+    end
+
+    context "when not tagging_enabled" do
+      before { SiteSetting.tagging_enabled = false }
+
+      it "does not return tag" do
+        result = subject.lookup(%w[book-club great-books fiction-books], %w[category tag])
+        expect(result[:category].map(&:slug)).to eq(["book-club"])
+        expect(result[:category].map(&:relative_url)).to eq(["/c/book-club/#{category1.id}"])
+        expect(result[:tag]).to eq([])
+      end
+    end
+  end
 end