The Windows command-line interface, part 2
How the evolution of the Console Window Host gave way to the Windows Terminal app.
Published in Articles on by Michiel van Oosterhout ~ 8 min read
In part 1 we learned about command lines, command-line shells, command-line programs, and how Windows has traditionally implemented a virtual console. We will now take a closer look at Console Window Host (conhost.exe) and the Windows Terminal app. The former is one of the earliest Windows NT applications and even though it got some significant updates between 2015 and 2018, and will remain an essential part of Windows, the Windows Terminal app, first released in 2019, should be considered the future of the command-line interface on Windows. How did Console Window Host evolve and give way to Windows Terminal, and what does it mean for the Windows command-line interface?
Console Window Host
Console Window Host is $Env:SystemRoot\System32\conhost.exe, a Windows application that is responsible for drawing a window, capturing and forwarding input to command-line programs, rendering output of command-line programs, and managing related settings in the Windows Registry and in shortcut files (.lnk). When starting a command-line program directly, Windows automatically starts an instance of Console Window Host and connects it to the command-line program's input and output. When starting Console Windows Host directly it will start and connect to Windows Command Processor (cmd.exe).
Console Window Host also provides the Console API to command-line programs.
The Console API can be used to write characters to a particular location, move the cursor, or change attributes like color or brightness of characters written to the output. Some functions allow for interaction with the Console Window Host, for example to set the title of its window, or to change its display mode to full screen.
However, this Console API also makes porting command-line programs to and from Windows more difficult, because other platforms use terminal sequences instead of an API.
General usability improvements
Microsoft introduced a legacy mode for Console Window Host, which allowed them to release several long overdue improvements to text selection, line wrapping, window resizing, and copy-paste behavior. The non-legacy mode even supports window transparency.
Terminal sequences
Terminal sequences are sequences of characters, embedded in the output of a command-line program, that are interpreted by a terminal as an instruction and not rendered. They can used to position the cursor, change the fore- and background color of text, and even change the title of a virtual terminal's window. Most terminal sequences start with the ESC control code, and are known as escape codes. These date back to the 70s when ANSI first defined a standard set of instructions and sequences 1. The current standard is ECMA-48 Control Functions for Coded Character Sets (1991).
When Microsoft started working on SSH support they quietly added support for terminal sequences to Console Window Host in Windows 10 November Update (version 1511). This feature can be enabled by command-line programs via the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag when calling SetConsoleMode:
When writing with
WriteFileorWriteConsole, characters are parsed for VT100 and similar control character sequences that control cursor movement, color/font mode, and other operations that can also be performed via the existing Console APIs. For more information, see Console Virtual Terminal Sequences.
You can test terminal sequences by entering the following command in Windows Command Processor (press Ctrl+[ to type the ESC control code, which will be rendered using caret notation): echo ^[[37;41mTEST^[[0m.
The example escape sequence that causes the following text to be rendered in white on a red background can be broken down like this:
^[[ 37 ; 41 m
--- -- - -- -
| | | | End of sequence marker
| | | Instruction: Set background color to red
| | Instruction separator
| Instruction: Set foreground color to white
Start of sequence
Terminal sequences can even be used to set the window title. For this test you should use Windows Powershell instead of Windows Command Processor, and enter the following command (press Ctrl+[ to type the ESC control code): echo "^[]0;TEST^[\". Alternatively end the sequence with BEL (press Ctrl+G) instead of ^[\: echo "^[]0;TEST^G".
Microsoft now recommends to use terminal sequences instead of the Console API 3.
24-bit colors
After adding support for terminal sequences Microsoft added support for 24-bit color. This was not implemented in the Console API, which to this day still only supports a 4-bit color pallette. The new 24-bit color support is actually support for 2 specific terminal sequences in the SGR category.
The terminal sequences ^[[38;2;<R>;<G>;<B>m and ^[[48;2;<R>;<G>;<B>m accept three decimal parameters separated by ; to specify the red, green, and blue component of the foreground and background color respectively 4.
Improved color palette
Although Microsoft did not change the nature of Console Window Host's color palette, in 2017 they did change the default colors assigned to each of the palette's entries. There are still only 16 entries in the palette, but the new color scheme (named Campbell) is suitable for modern display devices, improves legibility, and looks more modern.
New pseudo console
So far the Console Window Host and the command-line program were closely coupled, making it very difficult to implement 3rd-party consoles. But in 2018 Microsoft released the Windows pseudo console, a new mode for Console Window Host that allows 3rd-party consoles to take responsiblity for input and output.
In this mode Console Window Host receives (from stdin) UTF-8 encoded text with terminal sequences and converts it into a format suitable for its input buffer, and renders (to stdout) its output buffer as UTF-8 encoded text with terminal sequences. When running in this mode, a 3rd-party console is connected to stdin and stdout (the pseudo console) directly.
Improved Unicode support
Console Window Host uses a screen buffer to store the text that is to be rendered to the screen. This is a 2-dimensional (rows and columns) array of cells, each cell is a CHAR_INFO structure, representing a Unicode character and some text attributes like foreground and background color. The character is encoded using UCS-2 encoding (a single wchar_t value), limiting the buffer to characters from the Basic Multilingual Plane.
So in 2018 Microsoft released another update which removes this limitation, making it possible for a cell to contain this 😀 emoji for example, which is Unicode code point U+1F600 in the Supplementary Multilingual Plane 5, and thus requires 2 wchar_t values for a single cell.
When not running in pseudo console mode Console Window Host is responsible for rendering, and it uses GDI for rendering. GDI support for font-fallback, required to render missing glyphs (like emoji) from an alternative font, is inadequate, and support for surrogate pair, required to render Unicode code points from plains other than the BMP. So this update mainly benefits 3rd-party consoles that use more modern text rendering engines like DirectWrite and that connect to the pseudo console.
Experimental features
The final update for Console Window Host was the addition of experimental features in 2018. This improves the scrolling behavior of the window, adds additional cursor shapes, allows overriding the default 'inverse' color of the cursor with a specific RGB color, and allows overriding the foreground and background colors chosen from the color palette.
Windows Terminal
Finally in 2019 Microsoft announced the Windows Terminal app, a 1st-party console that uses the pseudo console and a DirectWrite-based text rendering engine, has support for tabbed windows, and provides a flexible configuration mechanism with multiple profiles that each can be associated with a specific command-line program.
Since the announcement multiple updates for Windows Terminal have been released and it has become the default command-line interface for Windows 11, and we should not expect more updates for the Console Window Host.
Summary
Windows Terminal is the default Windows command-line interface going forward, but it runs connected to the pseudo console provided by Console Window Host, one of the earliest Windows NT programs. Microsoft made significant improvements to Console Window Host to make this possible. Those improvements may make it feasible to run more command-line programs via Console Window Host, but for a truly modern command-line interface Windows Terminal is recommended.
Updates:
- Added a section about legacy console mode.
-
See ANSI escape code on Wikipedia, and ECMA-48 Control Functions for Character-Imaging I/O Devices. ↩︎
-
See C0 and C1 control codes and Caret notation on Wikipedia. ↩︎
-
See Classic Console APIs versus Virtual Terminal Sequences. ↩︎
-
See Extended Colors ↩︎