Files
discourse/app/models
Guo Xiang Tan cfd507822f PERF: Improve quality of PostSearchData#raw_data. (#7275)
This commit fixes the follow quality issue with `PostSearchData#raw_data`:

1. URLs are being tokenized and links with similar href and characters
are being duplicated in the raw data.

`Post#cooked`:

```
<p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org
```

2. Ligthbox being included in search pollutes the
`PostSearchData#raw_data` unncessarily.

From 28 March 2018 to 28 March 2019, searches for the term `image` on
`meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for
uploads that were pasted.

`Post#cooked`

```
<p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image
```

In terms of indexing performance, we now have to parse the given HTML
through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms
to scrub plus the indexing takes place in a background job.
2019-04-01 10:14:29 +08:00
..
2018-07-16 14:19:07 +08:00
2018-07-16 14:19:07 +08:00
2018-12-04 10:48:16 +01:00
2017-03-22 14:26:53 +08:00
2016-08-15 17:59:36 +10:00
2019-01-11 14:30:19 -05:00
2019-02-08 12:12:38 +01:00
2017-07-28 10:20:09 +09:00
2016-12-12 17:29:54 +08:00
2015-09-18 10:41:10 +10:00
2016-03-28 15:21:45 +11:00
2016-05-30 10:45:32 +10:00
2018-07-16 14:19:07 +08:00
2016-11-24 10:13:03 +08:00
2018-07-16 14:19:07 +08:00
2019-03-29 12:15:39 +08:00
2018-07-16 14:19:07 +08:00
2018-07-16 14:19:07 +08:00
2018-07-16 14:19:07 +08:00
2019-01-11 14:30:19 -05:00
2014-04-08 17:35:44 +02:00
2018-07-16 14:19:07 +08:00
2019-01-07 18:11:43 +01:00
2019-04-01 10:11:08 +08:00
2019-02-08 12:12:38 +01:00
2018-07-16 14:19:07 +08:00
2018-10-08 18:01:21 +02:00
2018-07-16 14:19:07 +08:00