Merge pull request #1817 from bfredl/bufmatch

support buffer-associated highlighting by external plugins

runtime/doc/api.txt

@@ -0,0 +1,99 @@
+*api.txt*    For Nvim.							 {Nvim}
+
+
+		 NVIM REFERENCE MANUAL    by Thiago de Arruda
+
+The C API of Nvim						     *nvim-api*
+
+1. Introduction			|nvim-api-intro|
+2. API Types			|nvim-api-types|
+3. API metadata			|nvim-api-metadata|
+4. Buffer highlighting		|nvim-api-highlights|
+
+==============================================================================
+1. Introduction			*nvim-api-intro*
+
+Nvim defines a C API as the primary way for external code to interact with
+the NVim core. In the present version of Nvim the API is primarily used by
+external processes to interact with Nvim using the msgpack-rpc protocol, see
+|msgpack-rpc|. The API will also be used from vimscript to access new Nvim core
+features, but this is not implemented yet. Later on, Nvim might be embeddable
+in C applications as libnvim, and the application will then control the
+embedded instance by calling the C API directly.
+
+==============================================================================
+2. API Types						       *nvim-api-types*
+
+Nvim's C API uses custom types for all functions. Some are just typedefs
+around C99 standard types, and some are Nvim defined data structures.
+
+Boolean				  -> bool
+Integer (signed 64-bit integer)	  -> int64_t
+Float (IEEE 754 double precision) -> double
+String				  -> {char* data, size_t size} struct
+
+Additionally, the following data structures are defined:
+
+Array
+Dictionary
+Object
+
+The following handle types are defined as integer typedefs, but are
+discriminated as separate types in an Object:
+
+Buffer				  -> enum value kObjectTypeBuffer
+Window				  -> enum value kObjectTypeWindow
+Tabpage				  -> enum value kObjectTypeTabpage
+
+==============================================================================
+3. API metadata						    *nvim-api-metadata*
+
+Nvim exposes metadata about the API as a Dictionary with the following keys:
+
+functions  	calling signature of the API functions
+types		The custom handle types defined by Nvim
+error_types	The possible kinds of errors an API function can exit with.
+
+This metadata is mostly useful for external programs accessing the api over
+msgpack-api, see |msgpack-rpc-api|.
+
+==============================================================================
+4. Buffer highlighting					  *nvim-api-highlights*
+
+Nvim allows plugins to add position-based highlights to buffers. This is
+similar to |matchaddpos()| but with some key differences. The added highlights
+are associated with a buffer and adapts to line insertions and deletions,
+similar to signs. It is also possible to manage a set of highlights as a group
+and delete or replace all at once.
+
+The intended use case are linter or semantic highlighter plugins that monitor
+a buffer for changes, and in the background compute highlights to the buffer.
+Another use case are plugins that show output in an append-only buffer, and
+want to add highlights to the outputs. Highlight data cannot be preserved
+on writing and loading a buffer to file, nor in undo/redo cycles.
+
+Highlights are registered using the |buffer_add_highlight| function, see the
+generated API documentation for details. If an external highlighter plugin is
+adding a large number of highlights in a batch, performance can be improved by
+calling |buffer_add_highlight| as an asynchronous notification, after first
+(synchronously) reqesting a source id. Here is an example using wrapper
+functions in the python client:
+>
+    src = vim.new_highlight_source()
+
+    buf = vim.current.buffer
+    for i in range(5):
+        buf.add_highlight("String",i,0,-1,src_id=src)
+
+    # some time later
+
+    buf.clear_highlight(src)
+<
+If the highlights don't need to be deleted or updated, just pass -1 as
+src_id (this is the default in python). |buffer_clear_highlight| can be used
+to clear highligts from a specific source, in a specific line range or the
+entire buffer by passing in the line range 0, -1 (the later is the default
+in python as used above).
+
+==============================================================================
+ vim:tw=78:ts=8:noet:ft=help:norl:

runtime/doc/msgpack_rpc.txt

@@ -7,7 +7,7 @@
 The Msgpack-RPC Interface to Nvim				  *msgpack-rpc*
 
 1. Introduction			|msgpack-rpc-intro|
-2. API				|msgpack-rpc-api|
+2. API mapping			|msgpack-rpc-api|
 3. Connecting			|msgpack-rpc-connecting|
 4. Clients			|msgpack-rpc-clients|
 5. Types			|msgpack-rpc-types|
@@ -36,13 +36,13 @@ Nvim's msgpack-rpc interface is like a more powerful version of Vim's
 `clientserver` feature.
 
 ==============================================================================
-2. API							      *msgpack-rpc-api*
+2. API mapping						      *msgpack-rpc-api*
 
-The Nvim C API is automatically exposed to the msgpack-rpc interface by the
+The Nvim C API, see |nvim-api|, is automatically exposed to the msgpack-rpc
-build system, which parses headers at src/nvim/api from the project root. A
+interface by the build system, which parses headers at src/nvim/api from the
-dispatch function is generated, which matches msgpack-rpc method names with
+project root. A dispatch function is generated, which matches msgpack-rpc method
-non-static API functions, converting/validating arguments and return values
+names with non-static API functions, converting/validating arguments and return
-back to msgpack.
+values back to msgpack.
 
 Client libraries will normally provide wrappers that hide msgpack-rpc details
 from programmers. The wrappers can be automatically generated by reading
@@ -63,7 +63,7 @@ Here's a simple way to get human-readable description of the API (requires
 Python and the `pyyaml`/`msgpack-python` pip packages):
 >
     nvim --api-info | python -c 'import msgpack, sys, yaml; print yaml.dump(msgpack.unpackb(sys.stdin.read()))' > api.yaml
-
+<
 ==============================================================================
 3. Connecting					       *msgpack-rpc-connecting*
 
@@ -162,8 +162,8 @@ https://github.com/msgpack-rpc/msgpack-rpc-ruby/blob/master/lib/msgpack/rpc/tran
 ==============================================================================
 5. Types						    *msgpack-rpc-types*
 
-Nvim's C API uses custom types for all functions (some are just typedefs
+Nvim's C API uses custom types for all functions, se |nvim-api-types|.
-around C99 standard types). The types can be split into two groups:
+For the purpose of mapping to msgpack, he types can be split into two groups:
 
 - Basic types that map natively to msgpack (and probably have a default
   representation in msgpack-supported programming languages)

src/nvim/api/buffer.c

@@ -19,6 +19,7 @@
 #include "nvim/mark.h"
 #include "nvim/fileio.h"
 #include "nvim/move.h"
+#include "nvim/syntax.h"
 #include "nvim/window.h"
 #include "nvim/undo.h"
 
@@ -514,6 +515,99 @@ ArrayOf(Integer, 2) buffer_get_mark(Buffer buffer, String name, Error *err)
   return rv;
 }
 
+/// Adds a highlight to buffer.
+///
+/// This can be used for plugins which dynamically generate highlights to a
+/// buffer (like a semantic highlighter or linter). The function adds a single
+/// highlight to a buffer. Unlike matchaddpos() highlights follow changes to
+/// line numbering (as lines are inserted/removed above the highlighted line),
+/// like signs and marks do.
+///
+/// "src_id" is useful for batch deletion/updating of a set of highlights. When
+/// called with src_id = 0, an unique source id is generated and returned.
+/// Succesive calls can pass in it as "src_id" to add new highlights to the same
+/// source group. All highlights in the same group can then be cleared with
+/// buffer_clear_highlight. If the highlight never will be manually deleted
+/// pass in -1 for "src_id".
+///
+/// If "hl_group" is the empty string no highlight is added, but a new src_id
+/// is still returned. This is useful for an external plugin to synchrounously
+/// request an unique src_id at initialization, and later asynchronously add and
+/// clear highlights in response to buffer changes.
+///
+/// @param buffer The buffer handle
+/// @param src_id Source group to use or 0 to use a new group,
+///               or -1 for ungrouped highlight
+/// @param hl_group Name of the highlight group to use
+/// @param line The line to highlight
+/// @param col_start Start of range of columns to highlight
+/// @param col_end End of range of columns to highlight,
+///                or -1 to highlight to end of line
+/// @param[out] err Details of an error that may have occurred
+/// @return The src_id that was used
+Integer buffer_add_highlight(Buffer buffer,
+                             Integer src_id,
+                             String hl_group,
+                             Integer line,
+                             Integer col_start,
+                             Integer col_end,
+                             Error *err)
+{
+  buf_T *buf = find_buffer_by_handle(buffer, err);
+  if (!buf) {
+    return 0;
+  }
+
+  if (line < 0 || line >= MAXLNUM) {
+    api_set_error(err, Validation, _("Line number outside range"));
+    return 0;
+  }
+  if (col_start < 0 || col_start > MAXCOL) {
+    api_set_error(err, Validation, _("Column value outside range"));
+    return 0;
+  }
+  if (col_end < 0 || col_end > MAXCOL) {
+    col_end = MAXCOL;
+  }
+
+  int hlg_id = syn_name2id((char_u*)hl_group.data);
+  src_id = bufhl_add_hl(buf, (int)src_id, hlg_id, (linenr_T)line+1,
+                        (colnr_T)col_start+1, (colnr_T)col_end);
+  return src_id;
+}
+
+/// Clears highlights from a given source group and a range of lines
+///
+/// To clear a source group in the entire buffer, pass in 1 and -1 to
+/// line_start and line_end respectively.
+///
+/// @param buffer The buffer handle
+/// @param src_id Highlight source group to clear, or -1 to clear all groups.
+/// @param line_start Start of range of lines to clear
+/// @param line_end End of range of lines to clear (exclusive)
+///                 or -1 to clear to end of file.
+/// @param[out] err Details of an error that may have occurred
+void buffer_clear_highlight(Buffer buffer,
+                            Integer src_id,
+                            Integer line_start,
+                            Integer line_end,
+                            Error *err)
+{
+  buf_T *buf = find_buffer_by_handle(buffer, err);
+  if (!buf) {
+    return;
+  }
+
+  if (line_start < 0 || line_start >= MAXLNUM) {
+    api_set_error(err, Validation, _("Line number outside range"));
+    return;
+  }
+  if (line_end < 0 || line_end > MAXLNUM) {
+    line_end = MAXLNUM;
+  }
+
+  bufhl_clear_line_range(buf, (int)src_id, (int)line_start+1, (int)line_end);
+}
 
 // Check if deleting lines made the cursor position invalid.
 // Changed the lines from "lo" to "hi" and added "extra" lines (negative if

src/nvim/buffer.c

@@ -580,16 +580,17 @@ free_buffer_stuff (
 )
 {
   if (free_options) {
-    clear_wininfo(buf);                 /* including window-local options */
+    clear_wininfo(buf);                 // including window-local options
-    free_buf_options(buf, TRUE);
+    free_buf_options(buf, true);
     ga_clear(&buf->b_s.b_langp);
   }
-  vars_clear(&buf->b_vars->dv_hashtab);   /* free all internal variables */
+  vars_clear(&buf->b_vars->dv_hashtab);   // free all internal variables
   hash_init(&buf->b_vars->dv_hashtab);
-  uc_clear(&buf->b_ucmds);              /* clear local user commands */
+  uc_clear(&buf->b_ucmds);              // clear local user commands
-  buf_delete_signs(buf);                /* delete any signs */
+  buf_delete_signs(buf);                // delete any signs
-  map_clear_int(buf, MAP_ALL_MODES, TRUE, FALSE);    /* clear local mappings */
+  bufhl_clear_all(buf);                // delete any highligts
-  map_clear_int(buf, MAP_ALL_MODES, TRUE, TRUE);     /* clear local abbrevs */
+  map_clear_int(buf, MAP_ALL_MODES, true, false);    // clear local mappings
+  map_clear_int(buf, MAP_ALL_MODES, true, true);     // clear local abbrevs
   xfree(buf->b_start_fenc);
   buf->b_start_fenc = NULL;
 }
@@ -4870,6 +4871,224 @@ void sign_mark_adjust(linenr_T line1, linenr_T line2, long amount, long amount_a
     }
 }
 
+// bufhl: plugin highlights associated with a buffer
+
+/// Adds a highlight to buffer.
+///
+/// Unlike matchaddpos() highlights follow changes to line numbering (as lines
+/// are inserted/removed above the highlighted line), like signs and marks do.
+///
+/// When called with "src_id" set to 0, a unique source id is generated and
+/// returned. Succesive calls can pass it in as "src_id" to add new highlights
+/// to the same source group. All highlights in the same group can be cleared
+/// at once. If the highlight never will be manually deleted pass in -1 for
+/// "src_id"
+///
+/// if "hl_id" or "lnum" is invalid no highlight is added, but a new src_id
+/// is still returned.
+///
+/// @param buf The buffer to add highlights to
+/// @param src_id src_id to use or 0 to use a new src_id group,
+///               or -1 for ungrouped highlight.
+/// @param hl_id Id of the highlight group to use
+/// @param lnum The line to highlight
+/// @param col_start First column to highlight
+/// @param col_end The last column to highlight,
+///                or -1 to highlight to end of line
+/// @return The src_id that was used
+int bufhl_add_hl(buf_T *buf,
+                 int src_id,
+                 int hl_id,
+                 linenr_T lnum,
+                 colnr_T col_start,
+                 colnr_T col_end) {
+  static int next_src_id = 1;
+  if (src_id == 0) {
+    src_id = next_src_id++;
+  }
+  if (hl_id <= 0) {
+      // no highlight group or invalid line, just return src_id
+      return src_id;
+  }
+  if (!buf->b_bufhl_info) {
+    buf->b_bufhl_info = map_new(linenr_T, bufhl_vec_T)();
+  }
+  bufhl_vec_T* lineinfo = map_ref(linenr_T, bufhl_vec_T)(buf->b_bufhl_info,
+                                                         lnum, true);
+
+  bufhl_hl_item_T *hlentry = kv_pushp(bufhl_hl_item_T, *lineinfo);
+  hlentry->src_id = src_id;
+  hlentry->hl_id = hl_id;
+  hlentry->start = col_start;
+  hlentry->stop = col_end;
+
+  if (0 < lnum && lnum <= buf->b_ml.ml_line_count) {
+    changed_lines_buf(buf, lnum, lnum+1, 0);
+    redraw_buf_later(buf, VALID);
+  }
+  return src_id;
+}
+
+/// Clear bufhl highlights from a given source group and range of lines.
+///
+/// @param buf The buffer to remove highlights from
+/// @param src_id highlight source group to clear, or -1 to clear all groups.
+/// @param line_start first line to clear
+/// @param line_end last line to clear or MAXLNUM to clear to end of file.
+void bufhl_clear_line_range(buf_T *buf,
+                            int src_id,
+                            linenr_T line_start,
+                            linenr_T line_end) {
+  if (!buf->b_bufhl_info) {
+    return;
+  }
+  linenr_T line;
+  linenr_T first_changed = MAXLNUM, last_changed = -1;
+  // In the case line_start - line_end << bufhl_info->size
+  // it might be better to reverse this, i e loop over the lines
+  // to clear on.
+  bufhl_vec_T unused;
+  map_foreach(buf->b_bufhl_info, line, unused, {
+    (void)unused;
+    if (line_start <= line && line <= line_end) {
+      if (bufhl_clear_line(buf->b_bufhl_info, src_id, line)) {
+        if (line > last_changed) {
+          last_changed = line;
+        }
+        if (line < first_changed) {
+          first_changed = line;
+        }
+      }
+    }
+  })
+
+  if (last_changed != -1) {
+    changed_lines_buf(buf, first_changed, last_changed+1, 0);
+    redraw_buf_later(buf, VALID);
+  }
+}
+
+/// Clear bufhl highlights from a given source group and given line
+///
+/// @param bufhl_info The highlight info for the buffer
+/// @param src_id Highlight source group to clear, or -1 to clear all groups.
+/// @param lnum Linenr where the highlight should be cleared
+static bool bufhl_clear_line(bufhl_info_T *bufhl_info, int src_id, int lnum) {
+  bufhl_vec_T* lineinfo = map_ref(linenr_T, bufhl_vec_T)(bufhl_info,
+                                                         lnum, false);
+  size_t oldsize = kv_size(*lineinfo);
+  if (src_id < 0) {
+    kv_size(*lineinfo) = 0;
+  } else {
+    size_t newind = 0;
+    for (size_t i = 0; i < kv_size(*lineinfo); i++) {
+      if (kv_A(*lineinfo, i).src_id != src_id) {
+        if (i != newind) {
+          kv_A(*lineinfo, newind) = kv_A(*lineinfo, i);
+        }
+        newind++;
+      }
+    }
+    kv_size(*lineinfo) = newind;
+  }
+
+  if (kv_size(*lineinfo) == 0) {
+    kv_destroy(*lineinfo);
+    map_del(linenr_T, bufhl_vec_T)(bufhl_info, lnum);
+  }
+  return kv_size(*lineinfo) != oldsize;
+}
+
+/// Remove all highlights and free the highlight data
+void bufhl_clear_all(buf_T* buf) {
+  if (!buf->b_bufhl_info) {
+    return;
+  }
+  bufhl_clear_line_range(buf, -1, 1, MAXLNUM);
+  map_free(linenr_T, bufhl_vec_T)(buf->b_bufhl_info);
+  buf->b_bufhl_info = NULL;
+}
+
+/// Adjust a placed highlight for inserted/deleted lines.
+void bufhl_mark_adjust(buf_T* buf,
+                       linenr_T line1,
+                       linenr_T line2,
+                       long amount,
+                       long amount_after) {
+  if (!buf->b_bufhl_info) {
+    return;
+  }
+
+  bufhl_info_T *newmap = map_new(linenr_T, bufhl_vec_T)();
+  linenr_T line;
+  bufhl_vec_T lineinfo;
+  map_foreach(buf->b_bufhl_info, line, lineinfo, {
+    if (line >= line1 && line <= line2) {
+      if (amount == MAXLNUM) {
+        bufhl_clear_line(buf->b_bufhl_info, -1, line);
+        continue;
+      } else {
+        line += amount;
+      }
+    } else if (line > line2) {
+      line += amount_after;
+    }
+    map_put(linenr_T, bufhl_vec_T)(newmap, line, lineinfo);
+  });
+  map_free(linenr_T, bufhl_vec_T)(buf->b_bufhl_info);
+  buf->b_bufhl_info = newmap;
+}
+
+
+/// Get highlights to display at a specific line
+///
+/// @param buf The buffer handle
+/// @param lnum The line number
+/// @param[out] info The highligts for the line
+/// @return true if there was highlights to display
+bool bufhl_start_line(buf_T *buf, linenr_T lnum, bufhl_lineinfo_T *info) {
+  if (!buf->b_bufhl_info) {
+    return false;
+  }
+
+  info->valid_to = -1;
+  info->entries = map_get(linenr_T, bufhl_vec_T)(buf->b_bufhl_info, lnum);
+  return kv_size(info->entries) > 0;
+}
+
+/// get highlighting at column col
+///
+/// It is is assumed this will be called with
+/// non-decreasing column nrs, so that it is
+/// possible to only recalculate highlights
+/// at endpoints.
+///
+/// @param info The info returned by bufhl_start_line
+/// @param col The column to get the attr for
+/// @return The highilight attr to display at the column
+int bufhl_get_attr(bufhl_lineinfo_T *info, colnr_T col) {
+  if (col <= info->valid_to) {
+    return info->current;
+  }
+  int attr = 0;
+  info->valid_to = MAXCOL;
+  for (size_t i = 0; i < kv_size(info->entries); i++) {
+    bufhl_hl_item_T entry = kv_A(info->entries, i);
+    if (entry.start <= col && col <= entry.stop) {
+      int entry_attr = syn_id2attr(entry.hl_id);
+      attr = hl_combine_attr(attr, entry_attr);
+      if (entry.stop < info->valid_to) {
+        info->valid_to = entry.stop;
+      }
+    } else if (col < entry.start && entry.start-1 < info->valid_to) {
+      info->valid_to = entry.start-1;
+    }
+  }
+  info->current = attr;
+  return attr;
+}
+
+
 /*
  * Set 'buflisted' for curbuf to "on" and trigger autocommands if it changed.
  */

src/nvim/buffer_defs.h

@@ -28,6 +28,8 @@ typedef struct file_buffer buf_T; // Forward declaration
 #include "nvim/profile.h"
 // for String
 #include "nvim/api/private/defs.h"
+// for Map(K, V)
+#include "nvim/map.h"
 
 #define MODIFIABLE(buf) (!buf->terminal && buf->b_p_ma)
 
@@ -59,21 +61,21 @@ typedef struct file_buffer buf_T; // Forward declaration
 #define VALID_BOTLINE_AP 0x40   /* w_botine is approximated */
 #define VALID_TOPLINE   0x80    /* w_topline is valid (for cursor position) */
 
-/* flags for b_flags */
+// flags for b_flags
-#define BF_RECOVERED    0x01    /* buffer has been recovered */
+#define BF_RECOVERED    0x01    // buffer has been recovered
-#define BF_CHECK_RO     0x02    /* need to check readonly when loading file
+#define BF_CHECK_RO     0x02    // need to check readonly when loading file
-                                   into buffer (set by ":e", may be reset by
+                                // into buffer (set by ":e", may be reset by
-                                   ":buf" */
+                                // ":buf")
-#define BF_NEVERLOADED  0x04    /* file has never been loaded into buffer,
+#define BF_NEVERLOADED  0x04    // file has never been loaded into buffer,
-                                   many variables still need to be set */
+                                // many variables still need to be set
-#define BF_NOTEDITED    0x08    /* Set when file name is changed after
+#define BF_NOTEDITED    0x08    // Set when file name is changed after
-                                   starting to edit, reset when file is
+                                // starting to edit, reset when file is
-                                   written out. */
+                                // written out.
-#define BF_NEW          0x10    /* file didn't exist when editing started */
+#define BF_NEW          0x10    // file didn't exist when editing started
-#define BF_NEW_W        0x20    /* Warned for BF_NEW and file created */
+#define BF_NEW_W        0x20    // Warned for BF_NEW and file created
-#define BF_READERR      0x40    /* got errors while reading the file */
+#define BF_READERR      0x40    // got errors while reading the file
-#define BF_DUMMY        0x80    /* dummy buffer, only used internally */
+#define BF_DUMMY        0x80    // dummy buffer, only used internally
-#define BF_PRESERVED    0x100   /* ":preserve" was used */
+#define BF_PRESERVED    0x100   // ":preserve" was used
 
 /* Mask to check for flags that prevent normal writing */
 #define BF_WRITE_MASK   (BF_NOTEDITED + BF_NEW + BF_READERR)
@@ -101,6 +103,11 @@ typedef int scid_T;                     /* script ID */
 // for signlist_T
 #include "nvim/sign_defs.h"
 
+// for bufhl_*_T
+#include "nvim/bufhl_defs.h"
+
+typedef Map(linenr_T, bufhl_vec_T) bufhl_info_T;
+
 // for FileID
 #include "nvim/os/fs_defs.h"
 
@@ -754,6 +761,8 @@ struct file_buffer {
   dict_T *additional_data;      // Additional data from shada file if any.
 
   int b_mapped_ctrl_c;          // modes where CTRL-C is mapped
+
+  bufhl_info_T *b_bufhl_info;   // buffer stored highlights
 };
 
 /*

src/nvim/bufhl_defs.h

@@ -0,0 +1,25 @@
+#ifndef NVIM_BUFHL_DEFS_H
+#define NVIM_BUFHL_DEFS_H
+
+#include "nvim/pos.h"
+#include "nvim/lib/kvec.h"
+// bufhl: buffer specific highlighting
+
+struct bufhl_hl_item
+{
+  int src_id;
+  int hl_id;  // highlight group
+  colnr_T start;  // first column to highlight
+  colnr_T stop;  // last column to highlight
+};
+typedef struct bufhl_hl_item bufhl_hl_item_T;
+
+typedef kvec_t(struct bufhl_hl_item) bufhl_vec_T;
+
+typedef struct {
+  bufhl_vec_T entries;
+  int current;
+  colnr_T valid_to;
+} bufhl_lineinfo_T;
+
+#endif  // NVIM_BUFHL_DEFS_H

src/nvim/ex_docmd.c

@@ -4320,7 +4320,7 @@ static void ex_unmap(exarg_T *eap)
  */
 static void ex_mapclear(exarg_T *eap)
 {
-  map_clear(eap->cmd, eap->arg, eap->forceit, FALSE);
+  map_clear_mode(eap->cmd, eap->arg, eap->forceit, false);
 }
 
 /*
@@ -4328,7 +4328,7 @@ static void ex_mapclear(exarg_T *eap)
  */
 static void ex_abclear(exarg_T *eap)
 {
-  map_clear(eap->cmd, eap->arg, TRUE, TRUE);
+  map_clear_mode(eap->cmd, eap->arg, true, true);
 }
 
 static void ex_autocmd(exarg_T *eap)

src/nvim/getchar.c

@@ -2915,9 +2915,9 @@ do_map (
                 did_it = TRUE;
               }
             }
-            if (mp->m_mode == 0) {              /* entry can be deleted */
+            if (mp->m_mode == 0) {              // entry can be deleted
-              map_free(mpp);
+              mapblock_free(mpp);
-              continue;                         /* continue with *mpp */
+              continue;                         // continue with *mpp
             }
 
             /*
@@ -3012,7 +3012,7 @@ theend:
  * Delete one entry from the abbrlist or maphash[].
  * "mpp" is a pointer to the m_next field of the PREVIOUS entry!
  */
-static void map_free(mapblock_T **mpp)
+static void mapblock_free(mapblock_T **mpp)
 {
   mapblock_T  *mp;
 
@@ -3080,7 +3080,7 @@ int get_map_mode(char_u **cmdp, int forceit)
  * Clear all mappings or abbreviations.
  * 'abbr' should be FALSE for mappings, TRUE for abbreviations.
  */
-void map_clear(char_u *cmdp, char_u *arg, int forceit, int abbr)
+void map_clear_mode(char_u *cmdp, char_u *arg, int forceit, int abbr)
 {
   int mode;
   int local;
@@ -3132,8 +3132,8 @@ map_clear_int (
       mp = *mpp;
       if (mp->m_mode & mode) {
         mp->m_mode &= ~mode;
-        if (mp->m_mode == 0) {       /* entry can be deleted */
+        if (mp->m_mode == 0) {       // entry can be deleted
-          map_free(mpp);
+          mapblock_free(mpp);
           continue;
         }
         /*

src/nvim/lib/khash.h

@@ -184,7 +184,7 @@ typedef khint_t khiter_t;
 #define kfree(P) xfree(P)
 #endif
 
-static const double __ac_HASH_UPPER = 0.77;
+#define __ac_HASH_UPPER 0.77
 
 #define __KHASH_TYPE(name, khkey_t, khval_t) \
 	typedef struct { \

src/nvim/lib/kvec.h

@@ -77,10 +77,10 @@ int main() {
 		(v).items[(v).size++] = (x);										\
 	} while (0)
 
-#define kv_pushp(type, v) (((v).size == (v).capacity)?							\
+#define kv_pushp(type, v) ((((v).size == (v).capacity)?							\
 						   ((v).capacity = ((v).capacity? (v).capacity<<1 : 8),				\
 							(v).items = (type*)xrealloc((v).items, sizeof(type) * (v).capacity), 0)	\
-						   : 0), ((v).items + ((v).size++))
+						   : 0), ((v).items + ((v).size++)))
 
 #define kv_a(type, v, i) (((v).capacity <= (size_t)(i)? \
 						  ((v).capacity = (v).size = (i) + 1, kv_roundup32((v).capacity), \

src/nvim/map.c

@@ -18,6 +18,9 @@
 #define uint32_t_eq kh_int_hash_equal
 #define int_hash kh_int_hash_func
 #define int_eq kh_int_hash_equal
+#define linenr_T_hash kh_int_hash_func
+#define linenr_T_eq kh_int_hash_equal
+
 
 #if defined(ARCH_64)
 #define ptr_t_hash(key) uint64_t_hash((uint64_t)key)
@@ -78,6 +81,25 @@
     return rv;                                                                \
   }                                                                           \
                                                                               \
+  U *map_##T##_##U##_ref(Map(T, U) *map, T key, bool put)                     \
+  {                                                                           \
+    int ret;                                                                  \
+    khiter_t k;                                                               \
+    if (put) {                                                                \
+      k = kh_put(T##_##U##_map, map->table, key, &ret);                       \
+      if (ret) {                                                              \
+        kh_val(map->table, k) = INITIALIZER(T, U);                            \
+      }                                                                       \
+    } else {                                                                  \
+      k = kh_get(T##_##U##_map, map->table, key);                             \
+      if (k == kh_end(map->table)) {                                          \
+        return NULL;                                                          \
+      }                                                                       \
+    }                                                                         \
+                                                                              \
+    return &kh_val(map->table, k);                                            \
+  }                                                                           \
+                                                                              \
   U map_##T##_##U##_del(Map(T, U) *map, T key)                                \
   {                                                                           \
     U rv = INITIALIZER(T, U);                                                 \
@@ -118,3 +140,5 @@ MAP_IMPL(ptr_t, ptr_t, DEFAULT_INITIALIZER)
 MAP_IMPL(uint64_t, ptr_t, DEFAULT_INITIALIZER)
 #define MSGPACK_HANDLER_INITIALIZER {.fn = NULL, .async = false}
 MAP_IMPL(String, MsgpackRpcRequestHandler, MSGPACK_HANDLER_INITIALIZER)
+#define KVEC_INITIALIZER { .size = 0, .capacity = 0, .items = NULL }
+MAP_IMPL(linenr_T, bufhl_vec_T, KVEC_INITIALIZER)

src/nvim/map.h

@@ -6,6 +6,7 @@
 #include "nvim/map_defs.h"
 #include "nvim/api/private/defs.h"
 #include "nvim/msgpack_rpc/defs.h"
+#include "nvim/bufhl_defs.h"
 
 #define MAP_DECLS(T, U)                                                       \
   KHASH_DECLARE(T##_##U##_map, T, U)                                          \
@@ -19,6 +20,7 @@
   U map_##T##_##U##_get(Map(T, U) *map, T key);                               \
   bool map_##T##_##U##_has(Map(T, U) *map, T key);                            \
   U map_##T##_##U##_put(Map(T, U) *map, T key, U value);                      \
+  U *map_##T##_##U##_ref(Map(T, U) *map, T key, bool put);                    \
   U map_##T##_##U##_del(Map(T, U) *map, T key);                               \
   void map_##T##_##U##_clear(Map(T, U) *map);
 
@@ -28,12 +30,14 @@ MAP_DECLS(cstr_t, ptr_t)
 MAP_DECLS(ptr_t, ptr_t)
 MAP_DECLS(uint64_t, ptr_t)
 MAP_DECLS(String, MsgpackRpcRequestHandler)
+MAP_DECLS(linenr_T, bufhl_vec_T)
 
 #define map_new(T, U) map_##T##_##U##_new
 #define map_free(T, U) map_##T##_##U##_free
 #define map_get(T, U) map_##T##_##U##_get
 #define map_has(T, U) map_##T##_##U##_has
 #define map_put(T, U) map_##T##_##U##_put
+#define map_ref(T, U) map_##T##_##U##_ref
 #define map_del(T, U) map_##T##_##U##_del
 #define map_clear(T, U) map_##T##_##U##_clear
 

src/nvim/mark.c

@@ -922,6 +922,7 @@ void mark_adjust(linenr_T line1, linenr_T line2, long amount, long amount_after)
     }
 
     sign_mark_adjust(line1, line2, amount, amount_after);
+    bufhl_mark_adjust(curbuf, line1, line2, amount, amount_after);
   }
 
   /* previous context mark */

src/nvim/misc1.c

@@ -1985,13 +1985,13 @@ changed_lines (
   changed_common(lnum, col, lnume, xtra);
 }
 
-static void 
+/// Mark line range in buffer as changed.
-changed_lines_buf (
+///
-    buf_T *buf,
+/// @param buf the buffer where lines were changed
-    linenr_T lnum,              /* first line with change */
+/// @param lnum first line with change
-    linenr_T lnume,             /* line below last changed line */
+/// @param lnume line below last changed line
-    long xtra                  /* number of extra lines (negative when deleting) */
+/// @param xtra number of extra lines (negative when deleting)
-)
+void changed_lines_buf(buf_T *buf, linenr_T lnum, linenr_T lnume, long xtra)
 {
   if (buf->b_mod_set) {
     /* find the maximum area that must be redisplayed */

src/nvim/msgpack_rpc/defs.h

@@ -1,8 +1,6 @@
 #ifndef NVIM_MSGPACK_RPC_DEFS_H
 #define NVIM_MSGPACK_RPC_DEFS_H
 
-#include <msgpack.h>
-
 
 /// The rpc_method_handlers table, used in msgpack_rpc_dispatch(), stores
 /// functions of this type.
@@ -24,22 +22,6 @@ void msgpack_rpc_add_method_handler(String method,
 
 void msgpack_rpc_init_function_metadata(Dictionary *metadata);
 
-/// Dispatches to the actual API function after basic payload validation by
-/// `msgpack_rpc_call`. It is responsible for validating/converting arguments
-/// to C types, and converting the return value back to msgpack types.
-/// The implementation is generated at compile time with metadata extracted
-/// from the api/*.h headers,
-///
-/// @param channel_id The channel id
-/// @param method_id The method id
-/// @param req The parsed request object
-/// @param error Pointer to error structure
-/// @return Some object
-Object msgpack_rpc_dispatch(uint64_t channel_id,
-                            msgpack_object *req,
-                            Error *error)
-  FUNC_ATTR_NONNULL_ARG(2) FUNC_ATTR_NONNULL_ARG(3);
-
 MsgpackRpcRequestHandler msgpack_rpc_get_handler_for(const char *name,
                                                      size_t name_len)
   FUNC_ATTR_NONNULL_ARG(1);

src/nvim/screen.c

@@ -2184,6 +2184,10 @@ win_line (
   int prev_c1 = 0;                      /* first composing char for prev_c */
   int did_line_attr = 0;
 
+  bool has_bufhl = false;                // this buffer has highlight matches
+  int bufhl_attr = 0;                   // attributes desired by bufhl
+  bufhl_lineinfo_T bufhl_info;          // bufhl data for this line
+
   /* draw_state: items that are drawn in sequence: */
 #define WL_START        0               /* nothing done yet */
 # define WL_CMDLINE     WL_START + 1    /* cmdline window column */
@@ -2244,6 +2248,11 @@ win_line (
     }
   }
 
+  if (bufhl_start_line(wp->w_buffer, lnum, &bufhl_info)) {
+    has_bufhl = true;
+    extra_check = true;
+  }
+
   /* Check for columns to display for 'colorcolumn'. */
   color_cols = wp->w_buffer->terminal ? NULL : wp->w_p_cc_cols;
   if (color_cols != NULL)
@@ -3335,6 +3344,17 @@ win_line (
             char_attr = hl_combine_attr(spell_attr, char_attr);
         }
 
+        if (has_bufhl && v > 0) {
+          bufhl_attr = bufhl_get_attr(&bufhl_info, (colnr_T)v);
+          if (bufhl_attr != 0) {
+            if (!attr_pri) {
+              char_attr = hl_combine_attr(char_attr, bufhl_attr);
+            } else {
+              char_attr = hl_combine_attr(bufhl_attr, char_attr);
+            }
+          }
+        }
+
         if (wp->w_buffer->terminal) {
           char_attr = hl_combine_attr(char_attr, term_attrs[vcol]);
         }

test/functional/ui/bufhl_spec.lua

@@ -0,0 +1,261 @@
+local helpers = require('test.functional.helpers')
+local Screen = require('test.functional.ui.screen')
+local clear, feed, nvim, insert = helpers.clear, helpers.feed, helpers.nvim, helpers.insert
+local execute, request, eq, neq = helpers.execute, helpers.request, helpers.eq, helpers.neq
+
+
+describe('Buffer highlighting', function()
+  local screen
+  local curbuf
+
+  local hl_colors = {
+    NonText = Screen.colors.Blue,
+    Question = Screen.colors.SeaGreen,
+    String = Screen.colors.Fuchsia,
+    Statement = Screen.colors.Brown,
+    Special = Screen.colors.SlateBlue,
+    Identifier = Screen.colors.DarkCyan
+  }
+
+  before_each(function()
+    clear()
+    execute("syntax on")
+    screen = Screen.new(40, 8)
+    screen:attach()
+    screen:set_default_attr_ignore( {{bold=true, foreground=hl_colors.NonText}} )
+    screen:set_default_attr_ids({
+      [1] = {foreground = hl_colors.String},
+      [2] = {foreground = hl_colors.Statement, bold = true},
+      [3] = {foreground = hl_colors.Special},
+      [4] = {bold = true, foreground = hl_colors.Special},
+      [5] = {foreground = hl_colors.Identifier},
+      [6] = {bold = true},
+      [7] = {underline = true, bold = true, foreground = hl_colors.Special},
+      [8] = {foreground = hl_colors.Special, underline = true}
+    })
+    curbuf = request('vim_get_current_buffer')
+  end)
+
+  after_each(function()
+    screen:detach()
+  end)
+
+  local function add_hl(...)
+    return request('buffer_add_highlight', curbuf, ...)
+  end
+
+  local function clear_hl(...)
+    return request('buffer_clear_highlight', curbuf, ...)
+  end
+
+
+  it('works', function()
+    insert([[
+      these are some lines
+      with colorful text]])
+    feed('+')
+
+    screen:expect([[
+      these are some lines                    |
+      with colorful tex^t                      |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+
+    add_hl(-1, "String", 0 , 10, 14)
+    add_hl(-1, "Statement", 1 , 5, -1)
+
+    screen:expect([[
+      these are {1:some} lines                    |
+      with {2:colorful tex^t}                      |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+
+    feed("ggo<esc>")
+    screen:expect([[
+      these are {1:some} lines                    |
+      ^                                        |
+      with {2:colorful text}                      |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+
+    clear_hl(-1, 0 , -1)
+    screen:expect([[
+      these are some lines                    |
+      ^                                        |
+      with colorful text                      |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+  end)
+
+  describe('support adding multiple sources', function()
+    local id1, id2
+    before_each(function()
+      insert([[
+        a longer example
+        in order to demonstrate
+        combining highlights
+        from different sources]])
+
+      execute("hi ImportantWord gui=bold cterm=bold")
+      id1 = add_hl(0, "ImportantWord", 0, 2, 8)
+      add_hl(id1, "ImportantWord", 1, 12, -1)
+      add_hl(id1, "ImportantWord", 2, 0, 9)
+      add_hl(id1, "ImportantWord", 3, 5, 14)
+
+      id2 = add_hl(0, "Special", 0, 2, 8)
+      add_hl(id2, "Identifier", 1, 3, 8)
+      add_hl(id2, "Special", 1, 14, 20)
+      add_hl(id2, "Underlined", 2, 6, 12)
+      add_hl(id2, "Underlined", 3, 0, 9)
+      neq(id1, id2)
+
+      screen:expect([[
+        a {4:longer} example                        |
+        in {5:order} to {6:de}{4:monstr}{6:ate}                 |
+        {6:combin}{7:ing}{8: hi}ghlights                    |
+        {8:from }{7:diff}{6:erent} source^s                  |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+        :hi ImportantWord gui=bold cterm=bold   |
+      ]])
+    end)
+
+    it('and clearing the first added', function()
+      clear_hl(id1, 0, -1)
+      screen:expect([[
+        a {3:longer} example                        |
+        in {5:order} to de{3:monstr}ate                 |
+        combin{8:ing hi}ghlights                    |
+        {8:from diff}erent source^s                  |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+        :hi ImportantWord gui=bold cterm=bold   |
+      ]])
+    end)
+
+    it('and clearing the second added', function()
+      clear_hl(id2, 0, -1)
+      screen:expect([[
+        a {6:longer} example                        |
+        in order to {6:demonstrate}                 |
+        {6:combining} highlights                    |
+        from {6:different} source^s                  |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+        :hi ImportantWord gui=bold cterm=bold   |
+      ]])
+    end)
+
+    it('and clearing line ranges', function()
+      clear_hl(-1, 0, 1)
+      clear_hl(id1, 1, 2)
+      clear_hl(id2, 2, -1)
+      screen:expect([[
+        a longer example                        |
+        in {5:order} to de{3:monstr}ate                 |
+        {6:combining} highlights                    |
+        from {6:different} source^s                  |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+        :hi ImportantWord gui=bold cterm=bold   |
+      ]])
+    end)
+
+    it('and renumbering lines', function()
+      feed('3Gddggo<esc>')
+      screen:expect([[
+        a {4:longer} example                        |
+        ^                                        |
+        in {5:order} to {6:de}{4:monstr}{6:ate}                 |
+        {8:from }{7:diff}{6:erent} sources                  |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+                                                |
+      ]])
+
+      execute(':3move 4')
+      screen:expect([[
+        a {4:longer} example                        |
+                                                |
+        {8:from }{7:diff}{6:erent} sources                  |
+        ^in {5:order} to {6:de}{4:monstr}{6:ate}                 |
+        ~                                       |
+        ~                                       |
+        ~                                       |
+        ::3move 4                               |
+      ]])
+    end)
+  end)
+
+  it('prioritizes latest added highlight', function()
+    insert([[
+      three overlapping colors]])
+    id1 = add_hl(0, "Identifier", 0, 6, 17)
+    id2 = add_hl(0, "Special", 0, 0, 9)
+    id3 = add_hl(0, "String", 0, 14, 23)
+
+    screen:expect([[
+      {3:three ove}{5:rlapp}{1:ing color}^s                |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+
+    clear_hl(id2, 0, 1)
+    screen:expect([[
+      three {5:overlapp}{1:ing color}^s                |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+  end)
+
+  it('works with multibyte text', function()
+    insert([[
+      Ta båten över sjön!]])
+    add_hl(-1, "Identifier", 0, 3, 9)
+    add_hl(-1, "String", 0, 16, 21)
+
+    screen:expect([[
+      Ta {5:båten} över {1:sjön}^!                     |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+      ~                                       |
+                                              |
+    ]])
+  end)
+end)