Merge PR #3413 'Refactor Neovim to remove the side effects of K_EVENT'

src/nvim/README.md

@@ -0,0 +1,190 @@
+## Source code overview
+
+Since Neovim has inherited most code from Vim, some information in [its
+README](https://raw.githubusercontent.com/vim/vim/master/src/README.txt) still
+applies.
+
+This document aims to give a high level overview of how Neovim works internally,
+focusing on parts that are different from Vim. Currently this is still a work in
+progress, especially because I have avoided adding too many details about parts
+that are constantly changing. As the code becomes more organized and stable,
+this document will be updated to reflect the changes.
+
+If you are looking for module-specific details, it is best to read the source
+code. Some files are extensively commented at the top(eg: terminal.c,
+screen.c).
+
+### Top-level program loops
+
+First let's understand what a Vim-like program does by analyzing the workflow of
+a typical editing session:
+
+01. Vim dispays the welcome screen
+02. User types: `:`
+03. Vim enters command-line mode
+04. User types: `edit README.txt<CR>`
+05. Vim opens the file and returns to normal mode
+06. User types: `G`
+07. Vim navigates to the end of the file
+09. User types: `5`
+10. Vim enters count-pending mode
+11. User types: `d`
+12. Vim enters operator-pending mode
+13. User types: `w`
+14. Vim deletes 5 words
+15. User types: `g`
+16. Vim enters the "g command mode"
+17. User types: `g`
+18. Vim goes to the beginning of the file
+19. User types: `i`
+20. Vim enters insert mode
+21. User types: `word<ESC>`
+22. Vim inserts "word" at the beginning and returns to normal mode
+
+Note that we have split user actions into sequences of inputs that change the
+state of the editor. While there's no documentation about a "g command
+mode"(step 16), internally it is implemented similarly to "operator-pending
+mode".
+
+From this we can see that Vim has the behavior of a input-driven state
+machine(more specifically, a pushdown automaton since it requires a stack for
+transitioning back from states). Assuming each state has a callback responsible
+for handling keys, this pseudocode(a python-like language) shows a good
+representation of the main program loop:
+
+```py
+def state_enter(state_callback, data):
+  do
+    key = readkey()                 # read a key from the user
+  while state_callback(data, key)   # invoke the callback for the current state
+```
+
+That is, each state is entered by calling `state_enter` and passing a
+state-specific callback and data. Here is a high-level pseudocode for a program
+that implements something like the workflow described above:
+
+```py
+def main()
+  state_enter(normal_state, {}):
+
+def normal_state(data, key):
+  if key == ':':
+    state_enter(command_line_state, {})
+  elif key == 'i':
+    state_enter(insert_state, {})
+  elif key == 'd':
+    state_enter(delete_operator_state, {})
+  elif key == 'g':
+    state_enter(g_command_state, {})
+  elif is_number(key):
+    state_enter(get_operator_count_state, {'count': key})
+  elif key == 'G'
+    jump_to_eof()
+  return true
+
+def command_line_state(data, key):
+  if key == '<cr>':
+    if data['input']:
+      execute_ex_command(data['input'])
+    return false
+  elif key == '<esc>'
+    return false
+
+  if not data['input']:
+    data['input'] = ''
+
+  data['input'] += key
+  return true
+
+def delete_operator_state(data, key):
+  count = data['count'] or 1
+  if key == 'w':
+    delete_word(count)
+  elif key == '$':
+    delete_to_eol(count)
+  return false  # return to normal mode
+
+def g_command_state(data, key):
+  if key == 'g':
+    go_top()
+  elif key == 'v':
+    reselect()
+  return false  # return to normal mode
+
+def get_operator_count_state(data, key):
+  if is_number(key):
+    data['count'] += key
+    return true
+  unshift_key(key)  # return key to the input buffer
+  state_enter(delete_operator_state, data)
+  return false
+
+def insert_state(data, key):
+  if key == '<esc>':
+    return false  # exit insert mode
+  self_insert(key)
+  return true
+```
+
+While the actual code is much more complicated, the above gives an idea of how
+Neovim is organized internally. Some states like the `g_command_state` or
+`get_operator_count_state` do not have a dedicated `state_enter` callback, but
+are implicitly embedded into other states(this will change later as we continue
+the refactoring effort). To start reading the actual code, here's the
+recommended order:
+
+1. `state_enter()` function(state.c). This is the actual program loop,
+   note that a `VimState` structure is used, which contains function pointers
+   for the callback and state data.
+2. `main()` function(main.c). After all startup, `normal_enter` is called
+   at the end of function to enter normal mode.
+3. `normal_enter()` function(normal.c) is a small wrapper for setting
+   up the NormalState structure and calling `state_enter`.
+4. `normal_check()` function(normal.c) is called before each iteration of
+   normal mode.
+5. `normal_execute()` function(normal.c) is called when a key is read in normal
+   mode.
+
+The basic structure described for normal mode in 3, 4 and 5 is used for other
+modes managed by the `state_enter` loop:
+
+- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`)
+- insert mode: `insert_{enter,check,execute}()`(`edit.c`)
+- terminal mode: `terminal_{enter,execute}()`(`terminal.c`)
+
+### Async event support
+
+One of the features Neovim added is the support for handling arbitrary
+asynchronous events, which can include:
+
+- msgpack-rpc requests
+- job control callbacks
+- timers(not implemented yet but the support code is already there)
+
+Neovim implements this functionality by entering another event loop while
+waiting for characters, so instead of:
+
+```py
+def state_enter(state_callback, data):
+  do
+    key = readkey()                 # read a key from the user
+  while state_callback(data, key)   # invoke the callback for the current state
+```
+
+Neovim program loop is more like:
+
+```py
+def state_enter(state_callback, data):
+  do
+    event = read_next_event()       # read an event from the operating system
+  while state_callback(data, event) # invoke the callback for the current state
+```
+
+where `event` is something the operating system delivers to us, including(but
+not limited to) user input. The `read_next_event()` part is internally
+implemented by libuv, the platform layer used by Neovim.
+
+Since Neovim inherited its code from Vim, the states are not prepared to receive
+"arbitrary events", so we use a special key to represent those(When a state
+receives an "arbitrary event", it normally doesn't do anything other update the
+screen).

src/nvim/ex_docmd.c

@@ -6544,7 +6544,7 @@ do_exedit (
         msg_scroll = 0;
         must_redraw = CLEAR;
 
-        main_loop(FALSE, TRUE);
+        normal_enter(false, true);
 
         RedrawingDisabled = rd;
         no_wait_return = nwr;

src/nvim/getchar.c

@@ -1131,7 +1131,7 @@ static void gotchars(char_u *chars, int len)
  * - While reading a script file.
  * - When no_u_sync is non-zero.
  */
-static void may_sync_undo(void)
+void may_sync_undo(void)
 {
   if ((!(State & (INSERT + CMDLINE)) || arrow_used)
       && scriptin[curscript] == NULL)

src/nvim/keymap.c

@@ -283,7 +283,6 @@ static struct key_name_entry {
   {K_ZERO,            (char_u *)"Nul"},
   {K_SNR,             (char_u *)"SNR"},
   {K_PLUG,            (char_u *)"Plug"},
-  {K_CURSORHOLD,      (char_u *)"CursorHold"},
   {K_PASTE,           (char_u *)"Paste"},
   {0,                 NULL}
 };

src/nvim/keymap.h

@@ -242,7 +242,6 @@ enum key_extra {
   , KE_X2RELEASE
 
   , KE_DROP             /* DnD data is available */
-  , KE_CURSORHOLD       /* CursorHold event */
   , KE_NOP              /* doesn't do something */
   , KE_FOCUSGAINED      /* focus gained */
   , KE_FOCUSLOST        /* focus lost */
@@ -437,7 +436,6 @@ enum key_extra {
 #define K_FOCUSGAINED   TERMCAP2KEY(KS_EXTRA, KE_FOCUSGAINED)
 #define K_FOCUSLOST     TERMCAP2KEY(KS_EXTRA, KE_FOCUSLOST)
 
-#define K_CURSORHOLD    TERMCAP2KEY(KS_EXTRA, KE_CURSORHOLD)
 #define K_EVENT         TERMCAP2KEY(KS_EXTRA, KE_EVENT)
 #define K_PASTE         TERMCAP2KEY(KS_EXTRA, KE_PASTE)
 

src/nvim/main.c

@@ -54,6 +54,7 @@
 #include "nvim/profile.h"
 #include "nvim/quickfix.h"
 #include "nvim/screen.h"
+#include "nvim/state.h"
 #include "nvim/strings.h"
 #include "nvim/syntax.h"
 #include "nvim/ui.h"
@@ -528,228 +529,16 @@ int main(int argc, char **argv)
   }
 
   TIME_MSG("before starting main loop");
+  ILOG("Starting Neovim main loop.");
 
   /*
    * Call the main command loop.  This never returns.
    */
-  main_loop(FALSE, FALSE);
+  normal_enter(false, false);
 
   return 0;
 }
 
-/*
- * Main loop: Execute Normal mode commands until exiting Vim.
- * Also used to handle commands in the command-line window, until the window
- * is closed.
- * Also used to handle ":visual" command after ":global": execute Normal mode
- * commands, return when entering Ex mode.  "noexmode" is TRUE then.
- */
-void
-main_loop (
-    int cmdwin,                 /* TRUE when working in the command-line window */
-    int noexmode               /* TRUE when return on entering Ex mode */
-)
-{
-  oparg_T oa;                                   /* operator arguments */
-  int previous_got_int = FALSE;                 /* "got_int" was TRUE */
-  linenr_T conceal_old_cursor_line = 0;
-  linenr_T conceal_new_cursor_line = 0;
-  int conceal_update_lines = FALSE;
-
-  ILOG("Starting Neovim main loop.");
-
-  clear_oparg(&oa);
-  while (!cmdwin
-      || cmdwin_result == 0
-      ) {
-    if (stuff_empty()) {
-      did_check_timestamps = FALSE;
-      if (need_check_timestamps)
-        check_timestamps(FALSE);
-      if (need_wait_return)             /* if wait_return still needed ... */
-        wait_return(FALSE);             /* ... call it now */
-      if (need_start_insertmode && goto_im()
-          && !VIsual_active
-         ) {
-        need_start_insertmode = FALSE;
-        stuffReadbuff((char_u *)"i");           /* start insert mode next */
-        /* skip the fileinfo message now, because it would be shown
-         * after insert mode finishes! */
-        need_fileinfo = FALSE;
-      }
-    }
-
-    /* Reset "got_int" now that we got back to the main loop.  Except when
-     * inside a ":g/pat/cmd" command, then the "got_int" needs to abort
-     * the ":g" command.
-     * For ":g/pat/vi" we reset "got_int" when used once.  When used
-     * a second time we go back to Ex mode and abort the ":g" command. */
-    if (got_int) {
-      if (noexmode && global_busy && !exmode_active && previous_got_int) {
-        /* Typed two CTRL-C in a row: go back to ex mode as if "Q" was
-         * used and keep "got_int" set, so that it aborts ":g". */
-        exmode_active = EXMODE_NORMAL;
-        State = NORMAL;
-      } else if (!global_busy || !exmode_active) {
-        if (!quit_more)
-          (void)vgetc();                        /* flush all buffers */
-        got_int = FALSE;
-      }
-      previous_got_int = TRUE;
-    } else
-      previous_got_int = FALSE;
-
-    if (!exmode_active)
-      msg_scroll = FALSE;
-    quit_more = FALSE;
-
-    /*
-     * If skip redraw is set (for ":" in wait_return()), don't redraw now.
-     * If there is nothing in the stuff_buffer or do_redraw is TRUE,
-     * update cursor and redraw.
-     */
-    if (skip_redraw || exmode_active)
-      skip_redraw = FALSE;
-    else if (do_redraw || stuff_empty()) {
-      /* Trigger CursorMoved if the cursor moved. */
-      if (!finish_op && (
-            has_cursormoved()
-            ||
-            curwin->w_p_cole > 0
-            )
-          && !equalpos(last_cursormoved, curwin->w_cursor)) {
-        if (has_cursormoved())
-          apply_autocmds(EVENT_CURSORMOVED, NULL, NULL,
-              FALSE, curbuf);
-        if (curwin->w_p_cole > 0) {
-          conceal_old_cursor_line = last_cursormoved.lnum;
-          conceal_new_cursor_line = curwin->w_cursor.lnum;
-          conceal_update_lines = TRUE;
-        }
-        last_cursormoved = curwin->w_cursor;
-      }
-
-      /* Trigger TextChanged if b_changedtick differs. */
-      if (!finish_op && has_textchanged()
-          && last_changedtick != curbuf->b_changedtick) {
-        if (last_changedtick_buf == curbuf)
-          apply_autocmds(EVENT_TEXTCHANGED, NULL, NULL,
-              FALSE, curbuf);
-        last_changedtick_buf = curbuf;
-        last_changedtick = curbuf->b_changedtick;
-      }
-
-      /* Scroll-binding for diff mode may have been postponed until
-       * here.  Avoids doing it for every change. */
-      if (diff_need_scrollbind) {
-        check_scrollbind((linenr_T)0, 0L);
-        diff_need_scrollbind = FALSE;
-      }
-      /* Include a closed fold completely in the Visual area. */
-      foldAdjustVisual();
-      /*
-       * When 'foldclose' is set, apply 'foldlevel' to folds that don't
-       * contain the cursor.
-       * When 'foldopen' is "all", open the fold(s) under the cursor.
-       * This may mark the window for redrawing.
-       */
-      if (hasAnyFolding(curwin) && !char_avail()) {
-        foldCheckClose();
-        if (fdo_flags & FDO_ALL)
-          foldOpenCursor();
-      }
-
-      /*
-       * Before redrawing, make sure w_topline is correct, and w_leftcol
-       * if lines don't wrap, and w_skipcol if lines wrap.
-       */
-      update_topline();
-      validate_cursor();
-
-      if (VIsual_active)
-        update_curbuf(INVERTED);        /* update inverted part */
-      else if (must_redraw)
-        update_screen(0);
-      else if (redraw_cmdline || clear_cmdline)
-        showmode();
-      redraw_statuslines();
-      if (need_maketitle)
-        maketitle();
-      /* display message after redraw */
-      if (keep_msg != NULL) {
-        char_u *p;
-
-        // msg_attr_keep() will set keep_msg to NULL, must free the string
-        // here. Don't reset keep_msg, msg_attr_keep() uses it to check for
-        // duplicates.
-        p = keep_msg;
-        msg_attr(p, keep_msg_attr);
-        xfree(p);
-      }
-      if (need_fileinfo) {              /* show file info after redraw */
-        fileinfo(FALSE, TRUE, FALSE);
-        need_fileinfo = FALSE;
-      }
-
-      emsg_on_display = FALSE;          /* can delete error message now */
-      did_emsg = FALSE;
-      msg_didany = FALSE;               /* reset lines_left in msg_start() */
-      may_clear_sb_text();              /* clear scroll-back text on next msg */
-      showruler(FALSE);
-
-      if (conceal_update_lines
-          && (conceal_old_cursor_line != conceal_new_cursor_line
-            || conceal_cursor_line(curwin)
-            || need_cursor_line_redraw)) {
-        if (conceal_old_cursor_line != conceal_new_cursor_line
-            && conceal_old_cursor_line
-            <= curbuf->b_ml.ml_line_count)
-          update_single_line(curwin, conceal_old_cursor_line);
-        update_single_line(curwin, conceal_new_cursor_line);
-        curwin->w_valid &= ~VALID_CROW;
-      }
-      setcursor();
-
-      do_redraw = FALSE;
-
-      /* Now that we have drawn the first screen all the startup stuff
-       * has been done, close any file for startup messages. */
-      if (time_fd != NULL) {
-        TIME_MSG("first screen update");
-        TIME_MSG("--- NVIM STARTED ---");
-        fclose(time_fd);
-        time_fd = NULL;
-      }
-    }
-
-    /*
-     * Update w_curswant if w_set_curswant has been set.
-     * Postponed until here to avoid computing w_virtcol too often.
-     */
-    update_curswant();
-
-    /*
-     * May perform garbage collection when waiting for a character, but
-     * only at the very toplevel.  Otherwise we may be using a List or
-     * Dict internally somewhere.
-     * "may_garbage_collect" is reset in vgetc() which is invoked through
-     * do_exmode() and normal_cmd().
-     */
-    may_garbage_collect = (!cmdwin && !noexmode);
-    /*
-     * If we're invoked as ex, do a round of ex commands.
-     * Otherwise, get and execute a normal mode command.
-     */
-    if (exmode_active) {
-      if (noexmode)         /* End of ":global/path/visual" commands */
-        return;
-      do_exmode(exmode_active == EXMODE_VIM);
-    } else
-      normal_cmd(&oa, TRUE);
-  }
-}
-
-
 /* Exit properly */
 void getout(int exitval)
 {
@@ -2075,3 +1864,5 @@ static void check_swap_exists_action(void)
     getout(1);
   handle_swap_exists(NULL);
 }
+
+

src/nvim/normal.h

@@ -36,8 +36,8 @@ typedef struct oparg_S {
   bool block_mode;              /* current operator is Visual block mode */
   colnr_T start_vcol;           /* start col for block mode operator */
   colnr_T end_vcol;             /* end col for block mode operator */
-  long prev_opcount;            /* ca.opcount saved for K_CURSORHOLD */
+  long prev_opcount;            // ca.opcount saved for K_EVENT
-  long prev_count0;             /* ca.count0 saved for K_CURSORHOLD */
+  long prev_count0;             // ca.count0 saved for K_EVENT
 } oparg_T;
 
 /*

src/nvim/os/input.c

@@ -73,10 +73,25 @@ void input_stop(void)
   stream_close(&read_stream, NULL);
 }
 
+static void cursorhold_event(void **argv)
+{
+  event_T event = State & INSERT ? EVENT_CURSORHOLDI : EVENT_CURSORHOLD;
+  apply_autocmds(event, NULL, NULL, false, curbuf);
+  did_cursorhold = true;
+}
+
+static void create_cursorhold_event(void)
+{
+  // If the queue had any items, this function should not have been
+  // called(inbuf_poll would return kInputAvail)
+  assert(queue_empty(loop.events));
+  queue_put(loop.events, cursorhold_event, 0);
+}
+
 // Low level input function
 int os_inchar(uint8_t *buf, int maxlen, int ms, int tb_change_cnt)
 {
-  if (rbuffer_size(input_buffer)) {
+  if (maxlen && rbuffer_size(input_buffer)) {
     return (int)rbuffer_read(input_buffer, (char *)buf, (size_t)maxlen);
   }
 
@@ -87,16 +102,12 @@ int os_inchar(uint8_t *buf, int maxlen, int ms, int tb_change_cnt)
     }
   } else {
     if ((result = inbuf_poll((int)p_ut)) == kInputNone) {
-      if (trigger_cursorhold() && maxlen >= 3
+      if (trigger_cursorhold() && !typebuf_changed(tb_change_cnt)) {
-          && !typebuf_changed(tb_change_cnt)) {
+        create_cursorhold_event();
-        buf[0] = K_SPECIAL;
+      } else {
-        buf[1] = KS_EXTRA;
+        before_blocking();
-        buf[2] = KE_CURSORHOLD;
+        result = inbuf_poll(-1);
-        return 3;
       }
-
-      before_blocking();
-      result = inbuf_poll(-1);
     }
   }
 
@@ -105,14 +116,14 @@ int os_inchar(uint8_t *buf, int maxlen, int ms, int tb_change_cnt)
     return 0;
   }
 
-  if (rbuffer_size(input_buffer)) {
+  if (maxlen && rbuffer_size(input_buffer)) {
     // Safe to convert rbuffer_read to int, it will never overflow since we use
     // relatively small buffers.
     return (int)rbuffer_read(input_buffer, (char *)buf, (size_t)maxlen);
   }
 
   // If there are events, return the keys directly
-  if (pending_events()) {
+  if (maxlen && pending_events()) {
     return push_event_key(buf, maxlen);
   }
 
@@ -313,6 +324,11 @@ void input_done(void)
   input_eof = true;
 }
 
+bool input_available(void)
+{
+  return rbuffer_size(input_buffer) != 0;
+}
+
 // This is a replacement for the old `WaitForChar` function in os_unix.c
 static InbufPollResult inbuf_poll(int ms)
 {

src/nvim/state.c

@@ -0,0 +1,62 @@
+#include <assert.h>
+
+#include "nvim/lib/kvec.h"
+
+#include "nvim/state.h"
+#include "nvim/vim.h"
+#include "nvim/getchar.h"
+#include "nvim/ui.h"
+#include "nvim/os/input.h"
+
+#ifdef INCLUDE_GENERATED_DECLARATIONS
+# include "state.c.generated.h"
+#endif
+
+
+void state_enter(VimState *s)
+{
+  for (;;) {
+    int check_result = s->check ? s->check(s) : 1;
+
+    if (!check_result) {
+      break;
+    } else if (check_result == -1) {
+      continue;
+    }
+
+    int key;
+
+getkey:
+    if (char_avail() || using_script() || input_available()) {
+      // Don't block for events if there's a character already available for
+      // processing. Characters can come from mappings, scripts and other
+      // sources, so this scenario is very common.
+      key = safe_vgetc();
+    } else if (!queue_empty(loop.events)) {
+      // Event was made available after the last queue_process_events call
+      key = K_EVENT;
+    } else {
+      input_enable_events();
+      // Flush screen updates before blocking
+      ui_flush();
+      // Call `os_inchar` directly to block for events or user input without
+      // consuming anything from `input_buffer`(os/input.c) or calling the
+      // mapping engine. If an event was put into the queue, we send K_EVENT
+      // directly.
+      (void)os_inchar(NULL, 0, -1, 0);
+      input_disable_events();
+      key = !queue_empty(loop.events) ? K_EVENT : safe_vgetc();
+    }
+
+    if (key == K_EVENT) {
+      may_sync_undo();
+    }
+
+    int execute_result = s->execute(s, key);
+    if (!execute_result) {
+      break;
+    } else if (execute_result == -1) {
+      goto getkey;
+    }
+  }
+}

src/nvim/state.h

@@ -0,0 +1,20 @@
+#ifndef NVIM_STATE_H
+#define NVIM_STATE_H
+
+#include <stddef.h>
+
+typedef struct vim_state VimState;
+
+typedef int(*state_check_callback)(VimState *state);
+typedef int(*state_execute_callback)(VimState *state, int key);
+
+struct vim_state {
+  state_check_callback check;
+  state_execute_callback execute;
+};
+
+#ifdef INCLUDE_GENERATED_DECLARATIONS
+# include "state.h.generated.h"
+#endif
+
+#endif  // NVIM_STATE_H

src/nvim/terminal.c

@@ -63,6 +63,7 @@
 #include "nvim/map.h"
 #include "nvim/misc1.h"
 #include "nvim/move.h"
+#include "nvim/state.h"
 #include "nvim/ex_docmd.h"
 #include "nvim/ex_cmds.h"
 #include "nvim/window.h"
@@ -73,6 +74,16 @@
 #include "nvim/api/private/helpers.h"
 #include "nvim/api/private/handle.h"
 
+typedef struct terminal_state {
+  VimState state;
+  Terminal *term;
+  int save_state;           // saved value of State
+  int save_rd;              // saved value of RedrawingDisabled
+  bool save_mapped_ctrl_c;  // saved value of mapped_ctrl_c;
+  bool close;
+  bool got_bs;              // if the last input was <C-\>
+} TerminalState;
+
 #ifdef INCLUDE_GENERATED_DECLARATIONS
 # include "terminal.c.generated.h"
 #endif
@@ -341,105 +352,106 @@ void terminal_resize(Terminal *term, uint16_t width, uint16_t height)
 void terminal_enter(void)
 {
   buf_T *buf = curbuf;
-  Terminal *term = buf->terminal;
+  TerminalState state, *s = &state;
-  assert(term && "should only be called when curbuf has a terminal");
+  memset(s, 0, sizeof(TerminalState));
+  s->term = buf->terminal;
+  assert(s->term && "should only be called when curbuf has a terminal");
 
   // Ensure the terminal is properly sized.
-  terminal_resize(term, 0, 0);
+  terminal_resize(s->term, 0, 0);
 
   checkpcmark();
   setpcmark();
-  int save_state = State;
+  s->save_state = State;
-  int save_rd = RedrawingDisabled;
+  s->save_rd = RedrawingDisabled;
   State = TERM_FOCUS;
   RedrawingDisabled = false;
-  bool save_mapped_ctrl_c = mapped_ctrl_c;
+  s->save_mapped_ctrl_c = mapped_ctrl_c;
   mapped_ctrl_c = true;
   // go to the bottom when the terminal is focused
-  adjust_topline(term, buf, false);
+  adjust_topline(s->term, buf, false);
   // erase the unfocused cursor
-  invalidate_terminal(term, term->cursor.row, term->cursor.row + 1);
+  invalidate_terminal(s->term, s->term->cursor.row, s->term->cursor.row + 1);
   showmode();
   ui_busy_start();
   redraw(false);
-  int c;
-  bool close = false;
 
-  bool got_bs = false;  // True if the last input was <C-\>
+  s->state.execute = terminal_execute;
+  state_enter(&s->state);
 
-  while (curbuf->handle == term->buf_handle) {
-    input_enable_events();
-    c = safe_vgetc();
-    input_disable_events();
-
-    switch (c) {
-      case K_LEFTMOUSE:
-      case K_LEFTDRAG:
-      case K_LEFTRELEASE:
-      case K_MIDDLEMOUSE:
-      case K_MIDDLEDRAG:
-      case K_MIDDLERELEASE:
-      case K_RIGHTMOUSE:
-      case K_RIGHTDRAG:
-      case K_RIGHTRELEASE:
-      case K_MOUSEDOWN:
-      case K_MOUSEUP:
-        if (send_mouse_event(term, c)) {
-          goto end;
-        }
-        break;
-
-      case K_EVENT:
-        // We cannot let an event free the terminal yet. It is still needed.
-        term->refcount++;
-        queue_process_events(loop.events);
-        term->refcount--;
-        if (term->buf_handle == 0) {
-          close = true;
-          goto end;
-        }
-        break;
-
-      case Ctrl_N:
-        if (got_bs) {
-          goto end;
-        }
-        // FALLTHROUGH
-
-      default:
-        if (c == Ctrl_BSL && !got_bs) {
-          got_bs = true;
-          break;
-        }
-        if (term->closed) {
-          close = true;
-          goto end;
-        }
-
-        got_bs = false;
-        terminal_send_key(term, c);
-    }
-  }
-
-end:
   restart_edit = 0;
-  State = save_state;
+  State = s->save_state;
-  RedrawingDisabled = save_rd;
+  RedrawingDisabled = s->save_rd;
   // draw the unfocused cursor
-  invalidate_terminal(term, term->cursor.row, term->cursor.row + 1);
+  invalidate_terminal(s->term, s->term->cursor.row, s->term->cursor.row + 1);
-  mapped_ctrl_c = save_mapped_ctrl_c;
+  mapped_ctrl_c = s->save_mapped_ctrl_c;
   unshowmode(true);
-  redraw(buf != curbuf);
+  redraw(curbuf->handle != s->term->buf_handle);
   ui_busy_stop();
-  if (close) {
+  if (s->close) {
-    bool wipe = term->buf_handle != 0;
+    bool wipe = s->term->buf_handle != 0;
-    term->opts.close_cb(term->opts.data);
+    s->term->opts.close_cb(s->term->opts.data);
     if (wipe) {
       do_cmdline_cmd("bwipeout!");
     }
   }
 }
 
+static int terminal_execute(VimState *state, int key)
+{
+  TerminalState *s = (TerminalState *)state;
+
+  switch (key) {
+    case K_LEFTMOUSE:
+    case K_LEFTDRAG:
+    case K_LEFTRELEASE:
+    case K_MIDDLEMOUSE:
+    case K_MIDDLEDRAG:
+    case K_MIDDLERELEASE:
+    case K_RIGHTMOUSE:
+    case K_RIGHTDRAG:
+    case K_RIGHTRELEASE:
+    case K_MOUSEDOWN:
+    case K_MOUSEUP:
+      if (send_mouse_event(s->term, key)) {
+        return 0;
+      }
+      break;
+
+    case K_EVENT:
+      // We cannot let an event free the terminal yet. It is still needed.
+      s->term->refcount++;
+      queue_process_events(loop.events);
+      s->term->refcount--;
+      if (s->term->buf_handle == 0) {
+        s->close = true;
+        return 0;
+      }
+      break;
+
+    case Ctrl_N:
+      if (s->got_bs) {
+        return 0;
+      }
+      // FALLTHROUGH
+
+    default:
+      if (key == Ctrl_BSL && !s->got_bs) {
+        s->got_bs = true;
+        break;
+      }
+      if (s->term->closed) {
+        s->close = true;
+        return 0;
+      }
+
+      s->got_bs = false;
+      terminal_send_key(s->term, key);
+  }
+
+  return curbuf->handle == s->term->buf_handle;
+}
+
 void terminal_destroy(Terminal *term)
 {
   buf_T *buf = handle_get_buffer(term->buf_handle);

test/functional/api/vim_spec.lua

@@ -247,33 +247,21 @@ describe('vim_* functions', function()
         ~                                       |
         {1:very fail}                               |
       ]])
+      helpers.wait()
 
       -- shows up to &cmdheight lines
-      nvim_async('err_write', 'more fail\n')
+      nvim_async('err_write', 'more fail\ntoo fail\n')
-      nvim_async('err_write', 'too fail\n')
       screen:expect([[
         ~                                       |
         ~                                       |
         ~                                       |
         ~                                       |
         ~                                       |
-        {1:very fail}                               |
         {1:more fail}                               |
-        {2:Press ENTER or type command to continue}^ |
-      ]])
-
-      -- shows the rest after return
-      feed('<cr>')
-      screen:expect([[
-        ~                                       |
-        ~                                       |
-        ~                                       |
-        {1:very fail}                               |
-        {1:more fail}                               |
-        {2:Press ENTER or type command to continue} |
         {1:too fail}                                |
         {2:Press ENTER or type command to continue}^ |
       ]])
+      feed('<cr>')  -- exit the press ENTER screen
     end)
   end)
 

test/functional/terminal/tui_spec.lua

@@ -12,7 +12,9 @@ describe('tui', function()
   before_each(function()
     helpers.clear()
     screen = thelpers.screen_setup(0, '["'..helpers.nvim_prog..'", "-u", "NONE", "-i", "NONE", "--cmd", "set noswapfile"]')
-    screen.timeout = 30000 -- pasting can be really slow in the TUI
+    -- right now pasting can be really slow in the TUI, especially in ASAN.
+    -- this will be fixed later but for now we require a high timeout.
+    screen.timeout = 60000
     screen:expect([[
       {1: }                                                 |
       ~                                                 |
@@ -51,6 +53,49 @@ describe('tui', function()
     ]])
   end)
 
+  it('interprets leading esc byte as the alt modifier', function()
+    local keys = 'dfghjkl'
+    for c in keys:gmatch('.') do
+      execute('nnoremap <a-'..c..'> ialt-'..c..'<cr><esc>')
+      feed('\x1b'..c)
+    end
+    screen:expect([[
+      alt-j                                             |
+      alt-k                                             |
+      alt-l                                             |
+      {1: }                                                 |
+      [No Name] [+]                                     |
+                                                        |
+      -- TERMINAL --                                    |
+    ]])
+    feed('gg')
+    screen:expect([[
+      {1:a}lt-d                                             |
+      alt-f                                             |
+      alt-g                                             |
+      alt-h                                             |
+      [No Name] [+]                                     |
+                                                        |
+      -- TERMINAL --                                    |
+    ]])
+  end)
+
+  it('accepts ascii control sequences', function()
+    feed('i')
+    feed('\x16\x07') -- ctrl+g
+    feed('\x16\x16') -- ctrl+v
+    feed('\x16\x0d') -- ctrl+m
+    screen:expect([[
+    {3:^G^V^M}{1: }                                           |
+    ~                                                 |
+    ~                                                 |
+    ~                                                 |
+    [No Name] [+]                                     |
+    -- INSERT --                                      |
+    -- TERMINAL --                                    |
+    ]], {[1] = {reverse = true}, [2] = {background = 11}, [3] = {foreground = 4}})
+  end)
+
   it('automatically sends <Paste> for bracketed paste sequences', function()
     feed('i\x1b[200~')
     screen:expect([[