Problem: #25826 added a (duplicate) sign comparison function, which was
modified and strayed from the original in #27418.
Solution: Merge the two functions and add a display test that actually
tests for this order in addition to the legacy tests.
The "priority" field of extmarks can be used to set priorities of
extmarks which dictates which highlight group a range will actually have
when there are multiple extmarks applied. However, when multiple
extmarks have the same priority, the only way to enforce an actual
priority is through the order in which the extmarks are set.
It is not always possible or desirable to set extmarks in a specific
order, however, so we add a new "subpriority" field that explicitly
enforces the ordering of extmarks that have the same priority.
For now this will be used only to enforce priority of treesitter
highlights. A single node in a treesitter tree may match multiple
captures, in which case that node will have multiple extmarks set. The
order in which captures are returned from the treesitter API is not
_necessarily_ in the same order they are defined in a query file, so we
use the new subpriority field to force that ordering.
For now subpriorites are not documented and are not meant to be used by
external code, and it only applies to ephemeral extmarks. We indicate
the "private" nature of subpriorities by prefixing the field name with
an "_".
Problem: Marks moved by undo may be lost to "b_signcols.count".
Solution: Count signs for each undo object separately instead of
once for the entire undo.
Extmarks can contain URLs which can then be drawn in any supporting UI.
In the TUI, for example, URLs are "drawn" by emitting the OSC 8 control
sequence to the TTY. On terminals which support the OSC 8 sequence this
will create clickable hyperlinks.
URLs are treated as inline highlights in the decoration subsystem, so
are included in the `DecorSignHighlight` structure. However, unlike
other inline highlights they use allocated memory which must be freed,
so they set the `ext` flag in `DecorInline` so that their lifetimes are
managed along with other allocated memory like virtual text.
The decoration subsystem then adds the URLs as a new highlight
attribute. The highlight subsystem maintains a set of unique URLs to
avoid duplicating allocations for the same string. To attach a URL to an
existing highlight attribute we call `hl_add_url` which finds the URL in
the set (allocating and adding it if it does not exist) and sets the
`url` highlight attribute to the index of the URL in the set (using an
index helps keep the size of the `HlAttrs` struct small).
This has the potential to lead to an increase in highlight attributes
if a URL is used over a range that contains many different highlight
attributes, because now each existing attribute must be combined with
the URL. In practice, however, URLs typically span a range containing a
single highlight (e.g. link text in Markdown), so this is likely just a
pathological edge case.
When a new highlight attribute is defined with a URL it is copied to all
attached UIs with the `hl_attr_define` UI event. The TUI manages its own
set of URLs (just like the highlight subsystem) to minimize allocations.
The TUI keeps track of which URL is "active" for the cell it is
printing. If no URL is active and a cell containing a URL is printed,
the opening OSC 8 sequence is emitted and that URL becomes the actively
tracked URL. If the cursor is moved while in the middle of a URL span,
we emit the terminating OSC sequence to prevent the hyperlink from
spanning multiple lines.
This does not support nested hyperlinks, but that is a rare (and,
frankly, bizarre) use case. If a valid use case for nested hyperlinks
ever presents itself we can address that issue then.
Problem: Some edge cases to the old (pre-#26406) and current "b_signcols"
structure result in an incorrectly sized "auto" 'signcolumn'.
Solution: * Implement a simpler 'signcolumn' validation strategy by immediately
counting the number of signs in a range upon sign insertion and
deletion. Decrease in performance here but there is a clear path
forward to decreasing this performance hit by moving signs to a
dedicated marktree, or by adding meta-data to the existing
marktree which may be queried more efficiently?
* Also replace "max_count" and keep track of the number of lines with
a certain number of signs. This makes it so that it is no longer
necessary to scan the entire buffer when the maximum number of signs
decreases. This likely makes the commit a net increase in performance.
* To ensure correctness we also have re-initialize the count for an
edited region that spans multiple lines. Such an edit may move the
signs within it. Thus we count and decrement before splicing the
marktree and count and increment after.
Problem:
Many decoration providers (treesitter injection highlighting, semantic
token highlighting, inlay hint) rely on the correctness of the `botline`
argument of `on_win` callback. However, `botline` can be smaller than
the actual line number of the last displayed line if some lines are
folded. In such cases, some decorations will be missing in the lines not
covered by `botline`.
Solution:
Validate `botline` when invoking `on_win`.
NOTE:
It seems that the old code was deliberately avoiding this presumably due
to performance reasons. However, I haven't experienced noticeable lag
after this change, and I believe the cost of botline computation would
be much smaller than the cost of decoration providers.
Problem: Unable to predict which byte-offset to place virtual text to
make it repeat visually in the wrapped part of a line.
Solution: Add a flag to nvim_buf_set_extmark() that causes virtual
text to repeat in wrapped lines.
This is the command invoked repeatedly to make the changes:
:%s/^\(.*\)|\%(\*\(\d\+\)\)\?$\n\1|\%(\*\(\d\+\)\)\?$/\=submatch(1)..'|*'..(max([str2nr(submatch(2)),1])+max([str2nr(submatch(3)),1]))/g
Problem: The entire marktree needs to be traversed each time a sign is
removed from the sentinel line.
Solution: Remove sentinel line and instead keep track of the number of
lines that hold up the 'signcolumn' in "max_count". Adjust this
number for added/removed signs, and set it to 0 when the
maximum number of signs on a line changes. Only when
"max_count" is decremented to 0 due to sign removal do we need
to check the entire buffer.
Also replace "invalid_top" and "invalid_bot" with a map of
invalid ranges, further reducing the number of lines to be
checked.
Also improve tree traversal when counting the number of signs.
Instead of looping over the to be checked range and counting
the overlap for each row, keep track of the overlap in an
array and add this to the count.
Problem: Signcolumn width does not increase when ranged sign does not
start at sentinel line.
Solution: Handle paired range of added sign when checking signcols.
decor->text.str pointer must go. This removes it for conceal char,
in preparation for a larger PR which will also handle the sign case.
By actually allowing composing chars for a conceal chars, this
becomes a feature and not just a refactor, as a bonus.
Remove the monolithic Decoration struct. Before this change, each extmark
could either represent just a hl_id + priority value as a inline
decoration, or it would take a pointer to this monolitic 112 byte struct
which has to be allocated.
This change separates the decorations into two pieces: DecorSignHighlight
for signs, highlights and simple set-flag decorations (like spell,
ui-watched), and DecorVirtText for virtual text and lines.
The main separation here is whether they are expected to allocate more
memory. Currently this is not really true as sign text has to be an
allocated string, but the plan is to get rid of this eventually (it can
just be an array of two schar_T:s). Further refactors are expected to
improve the representation of each decoration kind individually. The
goal of this particular PR is to get things started by cutting the
Gordian knot which was the monolithic struct Decoration.
Now, each extmark can either contain chained indicies/pointers to
these kinds of objects, or it can fit a subset of DecorSignHighlight
inline.
The point of this change is not only to make decorations smaller in
memory. In fact, the main motivation is to later allow them to grow
_larger_, but on a dynamic, on demand fashion. As a simple example, it
would be possible to augment highlights to take a list of multiple
`hl_group`:s, which then would trivially map to a chain of multiple
DecorSignHighlight entries.
One small feature improvement included with this refactor itself, is
that the restriction that extmarks cannot be removed inside a decoration
provider has been lifted. These are instead safely lifetime extended
on a "to free" list until the current iteration of screen drawing is done.
NB: flags is a mess. but DecorLevel is useless, this slightly less so
Problem: The legacy signlist data structures and associated functions are
redundant since the introduction of extmark signs.
Solution: Store signs defined through the legacy commands in a hashmap, placed
signs in the extmark tree. Replace signlist associated functions.
Usage of the legacy sign commands should yield no change in behavior with the
exception of:
- "orphaned signs" are now always removed when the line it is placed on is
deleted. This used to depend on the value of 'signcolumn'.
- It is no longer possible to place multiple signs with the same identifier
in a single group on multiple lines. This will now move the sign instead.
Moreover, both signs placed through the legacy sign commands and through
|nvim_buf_set_extmark()|:
- Will show up in both |sign-place| and |nvim_buf_get_extmarks()|.
- Are displayed by increasing sign identifier, left to right.
Extmark signs used to be ordered decreasingly as opposed to legacy signs.
It is a design goal of extmarks that they allow precise tracking
of changes across undo/redo, including restore the exact positions
after a do/undo or undo/redo cycle. However this behavior is not useful
for all usecases. Many plugins won't keep marks around for long after
text changes, but uses them more like a cache until some external source
(like LSP semantic highlights) has fully updated to changed text and
then will explicitly readjust/replace extmarks as needed.
Add a "undo_restore" flag which is true by default (matches existing
behavior) but can be set to false to opt-out of this behavior.
Delete dead u_extmark_set() code.
Problem: Peeking and flushing output slows down execution.
Solution: Do not update the mode message when global_busy is set. Do not
flush when only peeking for a character. (Ken Takata)
cb574f4154
The removes the previous restriction that nvim_buf_set_extmark()
could not be used to highlight arbitrary multi-line regions
The problem can be summarized as follows: let's assume an extmark with a
hl_group is placed covering the region (5,0) to (50,0) Now, consider
what happens if nvim needs to redraw a window covering the lines 20-30.
It needs to be able to ask the marktree what extmarks cover this region,
even if they don't begin or end here.
Therefore the marktree needs to be augmented with the information covers
a point, not just what marks begin or end there. To do this, we augment
each node with a field "intersect" which is a set the ids of the
marks which overlap this node, but only if it is not part of the set of
any parent. This ensures the number of nodes that need to be explicitly
marked grows only logarithmically with the total number of explicitly
nodes (and thus the number of of overlapping marks).
Thus we can quickly iterate all marks which overlaps any query position
by looking up what leaf node contains that position. Then we only need
to consider all "start" marks within that leaf node, and the "intersect"
set of that node and all its parents.
Now, and the major source of complexity is that the tree restructuring
operations (to ensure that each node has T-1 <= size <= 2*T-1) also need
to update these sets. If a full inner node is split in two, one of the
new parents might start to completely overlap some ranges and its ids
will need to be moved from its children's sets to its own set.
Similarly, if two undersized nodes gets joined into one, it might no
longer completely overlap some ranges, and now the children which do
needs to have the have the ids in its set instead. And then there are
the pivots! Yes the pivot operations when a child gets moved from one
parent to another.
Problem:
* The guessed botline might be smaller than the actual botline e.g. when
there are folds and the user is typing in insert mode. This may result
in incorrect treesitter highlights for injections.
* botline can be larger than the last line number of the buffer, which
results in errors when placing extmarks.
Solution:
* Take a more conservative approximation. I am not sure if it is
sufficient to guarantee correctness, but it seems to be good enough
for the case mentioned above.
* Clamp it to the last line number.
Co-authored-by: Lewis Russell <me@lewisr.dev>
Problem: Wrong cursor position with virtual text before double-width
char at window edge.
Solution: Check for double-width char before adding virtual text size.
closes: vim/vim#12977ac2d8815ae
Problem: Wrong cursor position with virtual text before a whitespace
character and 'linebreak'.
Solution: Always set "col_adj" to "size - 1" and apply 'linebreak' after
adding the size of 'breakindent' and 'showbreak'.
closes: vim/vim#129566e55e85f92
N/A patches:
vim-patch:9.0.1826: keytrans() doesn't translate recorded key typed in a GUI
Problem: Multiline regex with Visual selection fails when Visual
selection contains virtual text after last char.
Solution: Only include virtual text after last char when getting full
line length.
closes: vim/vim#12908e3daa06be1
Problem: Cursor position still wrong with 'showbreak' and virtual text
after last character or 'listchars' "eol".
Solution: Remove unnecessary w_wcol adjustment in curs_columns(). Also
fix first char of virtual text not shown at the start of a screen
line.
closes: vim/vim#12478closes: vim/vim#12532closes: vim/vim#129046a3897232a
