Building astroterm

Systems programming primer

Everything after this section assumes Unix systems vocabulary. This section supplies it — 42 concepts in seven clusters, each with commands you can run right now to watch the idea happen on your own machine. Nothing here is specific to terminals; it is the shared language of every program that talks to a Unix kernel.

A · The kernel boundary

strace -e trace=write echo hi
hi
write(1, "hi\n", 3)                     = 3

B · Files and descriptors

ls -l /proc/self/fd
tty

# now the same thing, redirected — watch entry 1 change
ls -l /proc/self/fd > /tmp/fds.txt ; cat /tmp/fds.txt

C · Processes and plumbing

strace -f -e trace=clone,execve,pipe2,dup2 bash -c 'ls | wc -l'

pipe2([3, 4], 0)              = 0        <- the pipe: read end 3, write end 4
clone(...)                    = 286668   <- fork #1
clone(...)                    = 286669   <- fork #2
[pid 286668] dup2(4, 1)       = 1        <- ls: stdout becomes the write end
[pid 286669] dup2(3, 0)       = 0        <- wc: stdin becomes the read end
[pid 286668] execve("/usr/bin/ls", ["ls"], ...)

D · Signals

kill -l

# start a pipeline in the background, then look at the groups
bash -c 'sleep 5 | cat & sleep 0.3; ps -o pid,pgid,sid,comm --ppid $$; wait'

    PID    PGID     SID COMMAND
 287289  287287  287275 sleep
 287290  287287  287275 cat

E · Terminals

stty -a

# turn off ECHO, type something, then turn it back on
stty -echo
# (type anything — nothing appears)
stty echo

F · Bytes, bits and text

printf '\033[1;31mred\033[0m\n' | xxd

00000000: 1b5b 313b 3331 6d72 6564 1b5b 306d 0a

G · When things go wrong

# install the syscall and library man pages — not present on this machine
sudo apt install -y manpages-dev
man 2 fork

# watch a syscall fail, and see errno by name
strace -e trace=openat cat /nonexistent

openat(AT_FDCWD, "/nonexistent", O_RDONLY) = -1 ENOENT (No such file or directory)

What happens when you type ls

One command, traced end to end through every component. If you read only one section of this document, read this one — it is the whole system working, before any of it is abstracted into crates.

What you are actually building

"A terminal" is three separate programs that happen to be adjacent. Getting this boundary right up front is the single decision the rest of the architecture hangs off.

The from-scratch line

From scratch does not mean from nothing. It means from specifications rather than from someone else's implementation — and that line has to be drawn explicitly, because there is a hard floor beneath it.

Which parts are actually ours

Module architecture

A Cargo workspace with nine members. Two organising rules: exactly one crate touches the kernel, and three crates touch nothing at all.

Core data structures

Three types carry almost all the state. Get their shape right and the rest of the emulator is mechanical.

astroterm-core/src/grid.rs

// 16 bytes packed. A 200-col x 10,000-line scrollback is then ~32 MB —
// acceptable. If it isn't, intern the attributes and store a u32 index.
#[derive(Clone, Copy, PartialEq)]
pub struct Cell {
    pub ch:    char,      // base char; see WIDE_TRAIL below
    pub fg:    Color,     // Indexed(u8) | Rgb(u8,u8,u8) | Default
    pub bg:    Color,
    pub attrs: Attrs,     // hand-rolled bitflags — 20 lines, no crate
}

pub struct Grid {
    cols:      usize,
    rows:      usize,
    cells:     Vec<Cell>,               // rows * cols, row-major, index = row * cols + col
    scrollback: VecDeque<Vec<Cell>>,    // ring, capped at SCROLLBACK_LINES (10_000)
    cursor:    Cursor,                  // row, col, visible, and the DECSC saved copy
    region:    (usize, usize),          // DECSTBM top/bottom margins, default (0, rows-1)
    tabstops:  Vec<bool>,               // per column; HTS sets, TBC clears
    alt:       Option<Box<Grid>>,       // ESC [ ? 1049 h swaps the whole grid out
    dirty:     Vec<bool>,               // one flag per row — the renderer's damage list
}

The escape-sequence parser

The part that sounds hardest and isn't. It is a byte-at-a-time state machine with eight states, specified precisely by Paul Williams' VT500 diagram, and it never needs to look ahead or back.

ESC [ 2 J          // ED   — erase display
ESC [ 12 ; 40 H    // CUP  — cursor to row 12, col 40
ESC [ 1 ; 31 m     // SGR  — bold, foreground red
ESC [ ? 1049 h     // DEC private mode set — switch to alt screen
ESC [ ? 25 l       // DEC private mode reset — hide cursor
ESC ] 0 ; title BEL // OSC  — set window title

The display stack

No SDL2, no Xlib, no FreeType. A unix socket, a documented binary protocol, and a bitmap font — which is, not coincidentally, exactly how a real VT100 put characters on a screen.

Talking to X11 directly

Glyphs from a bitmap font

astroterm-font/src/psf.rs — both headers, verified against real files on this machine

// PSF1 — 4-byte header. Width is ALWAYS 8; there is no width field.
const PSF1_MAGIC: u16 = 0x0436;         // on disk: 36 04
pub struct Psf1Header {
    magic:    u16,   // 0x0436
    mode:     u8,    // bit0: 512 glyphs (else 256) · bit1: unicode table follows
    charsize: u8,    // bytes per glyph == height, since width is 8
}

// PSF2 — 32-byte header.
const PSF2_MAGIC: u32 = 0x864a_b572;    // on disk: 72 b5 4a 86
pub struct Psf2Header {
    magic:         u32,
    version:       u32,   // 0
    headersize:    u32,   // 32 — offset of the first glyph
    flags:         u32,   // bit0: a unicode table follows the glyphs
    numglyph:      u32,
    bytesperglyph: u32,   // == height * ((width + 7) / 8)
    height:        u32,
    width:         u32,
}

# first four bytes tell you which format you have
zcat /usr/share/consolefonts/Uni3-TerminusBold16.psf.gz | head -c4 | xxd
# 3604 0310  -> PSF1, mode 3 (512 glyphs + unicode table), 16 rows tall, 8 wide

# list every PSF2 font you have
for f in /usr/share/consolefonts/*.psf.gz; do
  [ "$(zcat "$f" | head -c4 | xxd -p)" = "72b54a86" ] && echo "$f"
done

Concurrency: one thread, one poll

Dropping SDL2 removes the reason threads existed. X11 is a socket and the PTY master is a file descriptor, so a single poll() handles both — and that erases an entire class of bug.

astrosh/src/exec.rs — one stage of a pipeline

// Every allocation happens here, in the parent, before the fork.
let argv: Vec<CString> = build_argv(&cmd)?;
let (rd, wr) = sys::pipe()?;

match unsafe { sys::fork()? } {
    0 => {
        // Async-signal-safe territory. No println!, no String, no ?, no panic.
        sys::dup2(wr, 1);
        sys::close(rd);
        sys::close(wr);
        sys::setpgid(0, pgid);          // job control: join the group
        sys::execve(&argv[0], &argv, &envp);
        unsafe { sys::_exit(127) }       // command not found
    }
    child => {
        sys::setpgid(child, pgid).ok(); // race-free: both sides set it
        sys::close(wr)?;                // parent must close its copy or the
        Ok((child, rd))                 // reader never sees EOF
    }
}

The build plan

Nine phases, 84 numbered tasks. Each phase ends in something that runs, and each task is sized to fit an evening or two. Two estimate columns, because one number would be a lie: what this takes if it goes well, and what it takes the first time you meet process groups and a wire protocol.

1. 0

   Spike — prove the PTY works

   ~1 weekend·throwaway code·Low risk

   Before any architecture, make §2's trace real. One file, no crates but libc, run inside your existing terminal. Delete it afterwards — its only job is to convince you the PTY is not magic.

   Tasks

   1. 0.1
      Install rustup. Confirm rustc --version reports 1.8x or newer.
      20 min
   2. 0.2
      cargo new --bin spike, add libc = "0.2", print hello world.
      15 min
   3. 0.3
      Open /dev/ptmx with O_RDWR. Print the fd number you get back — it will be 3.
      30 min
   4. 0.4
      Unlock the pair: ioctl(fd, TIOCSPTLCK, &0), then ioctl(fd, TIOCGPTN, &n) to learn the slave number. Print /dev/pts/N.
      1 hr
   5. 0.5
      fork(). In the child: setsid(), open the slave, dup2 it onto fds 0, 1 and 2, then execvp("/bin/bash").
      2 hr
   6. 0.6
      In the parent: loop reading the master and writing to your own stdout. You should see bash's prompt appear. Checkpoint
      45 min
   7. 0.7
      Put your stdin into raw mode — tcgetattr, clear ECHO|ICANON, tcsetattr. Otherwise your outer terminal eats the keystrokes.
      1 hr
   8. 0.8
      poll() on both stdin and the master; copy bytes each way. You can now type into it. Checkpoint
      1 hr
   9. 0.9
      Restore the original termios on exit, including on panic. Skip this and every crash leaves your terminal unusable.
      30 min

   Files

   spike/src/main.rs, roughly 120 lines.

   Verify

   Type ls and see colours. Run sleep 100 and press Ctrl+C — it interrupts the sleep, not your program. Exit and your outer shell still behaves normally.

   Traps

   Forgetting setsid() leaves the child with no controlling terminal, so job control silently never works. A read returning 0 means the child exited, not an error. If you do wreck your terminal, reset fixes it.

   If you stall

   Use libc's forkpty() — one call that does the entire /dev/ptmx dance for you. Get the byte pump working, then come back and open /dev/ptmx by hand once you know what forkpty was hiding.

2. 1

   astroterm-sys — your own syscall layer

   1–2 weeks·replaces nix + signal-hook·Unsafe-heavy

   Turn the spike's scattered calls into one auditable crate. Every unsafe block in the whole project should end up living here.

   Tasks

   1. 1.1
      Create the workspace and all nine crates using the commands in §15.
      30 min
   2. 1.2
      errno.rs — one Result type. Every libc call returns -1 on failure and sets errno; wrap that pattern exactly once.
      2 hr
   3. 1.3
      fd.rs — an owned Fd newtype whose Drop calls close. Makes descriptor leaks impossible.
      2 hr
   4. 1.4
      io.rs — read, write, pipe, dup2, close. Handle short writes: write may transfer fewer bytes than asked.
      3 hr
   5. 1.5
      ioctl.rs — the wrapper plus constants: TIOCSPTLCK, TIOCGPTN, TCGETS (0x5401), TCSETS, TIOCSWINSZ (0x5414), TIOCGWINSZ.

      Verify every constant by printing it from a two-line C program. A wrong number does not error — it corrupts memory.
      4 hr
   6. 1.6
      termios.rs — the struct, tcgetattr/tcsetattr, and a raw_mode() helper. Add a test asserting size_of::<Termios>() equals the C value.
      4 hr
   7. 1.7
      process.rs — fork, execve, waitpid, setpgid, _exit. Read §9's callout before writing this.
      4 hr
   8. 1.8
      poll.rs — the pollfd struct and wrapper. Handle EINTR by retrying.
      2 hr
   9. 1.9
      signal.rs — the self-pipe. Handler writes one byte and nothing else; expose the read end as an Fd.
      3 hr
   10. 1.10
       pty.rs in astroterm-pty — allocate a pair, spawn a child with the slave as its controlling terminal, resize, reap.
       4 hr
   11. 1.11
       Port the Phase 0 spike onto these crates and delete every direct libc:: call from it.
       3 hr

   Files

   astroterm-sys/src/{errno,fd,io,ioctl,termios,process,poll,signal}.rs and astroterm-pty/src/lib.rs. Around 600 lines.

   Verify

   The spike still works, and grep -rn "libc::" --exclude-dir=astroterm-sys returns nothing.

   Traps

   In C, ioctl is variadic — the third argument's type varies by request, so a wrong pointer type compiles fine and misbehaves at runtime. Get the struct field order wrong and you read garbage rather than an error.

   If you stall

   Add nix and port module by module. The crate boundary is the point: you can replace astroterm-sys's internals later without touching a line above it.

3. 2

   astrosh v1 — REPL, pipes, redirection

   2–3 weeks·first genuinely usable program·Low risk

   The first thing you will actually use. Run it inside your normal terminal — none of this needs a window.

   Tasks

   1. 2.1
      REPL skeleton: print a prompt, read a line, echo it back, exit cleanly on Ctrl+D (read returns 0).
      2 hr
   2. 2.2
      Tokenizer: split on whitespace, honouring "double", 'single' and backslash escapes. Write the unit tests first — this is pure and trivially testable.
      6 hr
   3. 2.3
      AST types: Command { argv, redirects } and Pipeline(Vec<Command>).
      2 hr
   4. 2.4
      Parser: tokens into that AST, recognising |, >, >>, < and 2>.
      5 hr
   5. 2.5
      $PATH search — split on :, test each directory for an executable file.
      2 hr
   6. 2.6
      Run one command: fork, execve, waitpid. Report the exit status. Checkpoint
      4 hr
   7. 2.7
      Builtins — cd, exit, pwd, export. These must run in the parent; a forked child that changes directory then exits accomplishes nothing.
      3 hr
   8. 2.8
      Redirection: open the target file in the child and dup2 it onto the right descriptor before exec.
      4 hr
   9. 2.9
      Pipelines: N commands need N−1 pipes. Close every unused end in both parent and child. Checkpoint
      6 hr
   10. 2.10
       Track the last exit status and expose it as $?.
       1 hr

   Files

   astrosh/src/{main,lexer,ast,parser,exec,builtins}.rs. Around 700 lines.

   Verify

   cat < in.txt | tr a-z A-Z | sort -u > out.txt produces the right file. cd .. persists across commands. ps aux | grep defunct shows nothing after running a hundred commands.

   Traps

   The classic hang: the parent forgets to close its copy of a pipe's write end, so the reader never sees EOF and ls | wc -l waits forever. Missing waitpid leaks zombies steadily.

   If you stall

   Throw away the parser. Split on whitespace with no quoting and no pipes, and get fork/exec/waitpid rock solid first. Add the tokenizer once the process plumbing is boring.

4. 3

   astrosh v2 — expansion and job control

   3–4 weeks·Hardest phase in the project

   Expansion is fiddly but mechanical. Job control is where most hobby shells stop, and it is worth reading the GNU libc chapter end to end before writing any of it.

   Tasks

   1. 3.1
      Variable storage plus $VAR, ${VAR} and ${VAR:-default}.
      5 hr
   2. 3.2
      Tilde expansion — ~ and ~user.
      2 hr
   3. 3.3
      Globbing: read the directory and match *, ? and [abc] yourself. A recursive matcher is about 60 lines.
      6 hr
   4. 3.4
      Command substitution $(...) — recursive: run the inner command with stdout on a pipe, capture it, splice the result back in.
      5 hr
   5. 3.5
      Operators &&, || and ;, honouring exit status.
      3 hr
   6. 3.6
      Background execution with & — do not wait, record the job.
      2 hr
   7. 3.7
      Process groups: give every pipeline its own, calling setpgid in both parent and child so neither loses the race.
      4 hr
   8. 3.8
      Terminal ownership: tcsetpgrp to the foreground group before running, take it back after.

      Block SIGTTOU around the call or the shell sends itself a stop signal and freezes.
      5 hr
   9. 3.9
      waitpid with WUNTRACED and a job table recording state per pipeline.
      4 hr
   10. 3.10
       jobs, fg and bg, sending SIGCONT to resume. Checkpoint
       5 hr
   11. 3.11
       Signal disposition: the shell ignores SIGINT, SIGQUIT and SIGTSTP; children reset them to default before exec.
       3 hr

   Files

   Adds astrosh/src/{expand,glob,jobs,signals}.rs. Around 900 more lines.

   Verify

   Ctrl+Z inside vim returns you to the prompt; jobs lists it as Stopped; fg restores it with the screen intact; Ctrl+C kills a running child without touching the shell.

   Traps

   Symptoms here are bizarre rather than obvious — the shell suspending itself, or Ctrl+C killing everything including you. If something odd happens, suspect SIGTTOU or a process group that does not exist yet.

   If you stall

   Ship without job control. No Ctrl+Z, no fg, no bg. The shell stays fully usable for the overwhelming majority of interactive work. Come back after Phase 7 with more Unix behind you — this is the single most reasonable thing to defer in the whole plan.

5. 4

   astroterm-core — parser and grid, headless

   3–4 weeks·no graphics yet·Detail-heavy

   The emulator's brain, with no window anywhere. Everything here is pure functions over bytes, so correctness is cheap — there is nothing to look at and nothing to mock.

   Tasks

   1. 4.1
      Cell, Color and Attrs — hand-roll the bitflags, it is about 20 lines.
      3 hr
   2. 4.2
      Grid: allocation, indexing, clearing, resizing. Keep it one flat Vec.
      5 hr
   3. 4.3
      Cursor movement with bounds clamping in every direction.
      3 hr
   4. 4.4
      Scrolling inside a margin region, and pushing displaced lines into the scrollback ring.
      5 hr
   5. 4.5
      The Perform trait — print, execute, csi_dispatch, osc_dispatch, esc_dispatch.
      2 hr
   6. 4.6
      The state machine: Ground, Escape, CsiEntry, CsiParam, with parameter accumulation and defaults.
      6 hr
   7. 4.7
      CSI dispatch — CUP, CUU/CUD/CUF/CUB, CNL/CPL, ED, EL.
      5 hr
   8. 4.8
      SGR: the 16 base colours, 256-colour mode, 24-bit truecolour, bold, underline, reverse. Checkpoint
      6 hr
   9. 4.9
      DEC private modes ?25 (cursor), ?1049 (alt screen), ?7 (autowrap), ?2004 (bracketed paste).
      4 hr
   10. 4.10
       OSC handling — window title (0 and 2) and clipboard (52).
       3 hr
   11. 4.11
       The remaining editing sequences: ICH, DCH, IL, DL, DECSTBM, IND/RI, DECSC/DECRC, tab stops.
       6 hr
   12. 4.12
       Golden-test harness: capture with script -c "htop -n1" -f out.raw, replay the bytes, snapshot the grid as plain text.

       Build this now, not in Phase 8. Debugging a text dump is many times faster than debugging pixels.
       5 hr

   Files

   astroterm-core/src/{cell,grid,parser,dispatch,scrollback}.rs plus tests/corpus/. Around 1,200 lines.

   Verify

   Golden snapshots captured from vim, htop and less replay to byte-identical grids.

   Traps

   Escape sequence parameters default to 1, not 0 — ESC [ H means row 1 column 1. An unrecognised sequence must be discarded silently; printing it is the fastest way to make a terminal look broken.

   If you stall

   Stop implementing from the specification. Capture what ls --color, vim and htop actually emit on your machine, and implement only what appears. Grow the set from real captures rather than from XTerm ctlseqs top to bottom.

6. 5

   astroterm-unicode — tables from the UCD

   ~1 week·replaces two crates·Low risk

   Deriving the tables from Unicode's published data files is genuinely from scratch. Copying someone's pre-generated table is not.

   Tasks

   1. 5.1
      Download and commit EastAsianWidth.txt, GraphemeBreakProperty.txt and emoji-data.txt so builds stay offline and reproducible.
      1 hr
   2. 5.2
      build.rs: parse those files into sorted (start, end, value) ranges and emit a Rust source file.
      6 hr
   3. 5.3
      width(ch) by binary search — W and F are 2 cells, combining marks are 0, everything else is 1.
      3 hr
   4. 5.4
      Grapheme clustering so a ZWJ emoji sequence collapses into one cell.
      6 hr
   5. 5.5
      Swap it into astroterm-core in place of the Phase 4 stub, then re-run the golden tests and read the diff carefully. Checkpoint
      3 hr

   Files

   astroterm-unicode/{build.rs,src/lib.rs,data/*.txt}. Around 300 lines plus generated tables.

   Verify

   A line of CJK text and a ZWJ emoji each occupy the correct number of cells, asserted by a golden test. unicode-width appears nowhere in the tree.

   Traps

   The data files use ranges written 4E00..9FFF with comments after #; a sloppy parser silently truncates the table and you get wrong widths only for rare characters.

   If you stall

   Hardcode the main wide ranges — U+1100–115F, U+2E80–A4CF, U+AC00–D7A3, U+F900–FAFF, U+FF00–FF60 as width 2, everything else 1. That covers almost everything you will actually type. Generate the real tables later.

7. 6

   astroterm-x11 — a window on screen

   2–3 weeks·replaces SDL2·Highest risk

   Follow Figure 11 exactly. Get to a solid-coloured window before you even think about text — debugging the handshake, the keymap and the compositor simultaneously is how this phase goes wrong.

   Tasks

   1. 6.1
      Parse $DISPLAY and connect to /tmp/.X11-unix/X0 as a Unix domain socket.
      3 hr
   2. 6.2
      Read ~/.Xauthority — its own small binary format of family, address, display, name and the 16-byte cookie.
      4 hr
   3. 6.3
      Send the setup request carrying that cookie; parse the setup reply for root window, visual, depth, resource-id-base and resource-id-mask.

      Omit the cookie and the server just refuses you. This is where most people stall.
      6 hr
   4. 6.4
      XID allocator — mint your own ids as base | (counter & mask).
      1 hr
   5. 6.5
      CreateWindow (opcode 1) and MapWindow (opcode 8). A window appears. Checkpoint
      5 hr
   6. 6.6
      CreateGC (55) and PutImage (72) with a solid colour, then a gradient. Checkpoint
      5 hr
   7. 6.7
      Event loop: read the 32-byte event headers off the socket and decode by type.
      4 hr
   8. 6.8
      GetKeyboardMapping (101), then keycode → keysym → bytes, including Shift and Ctrl. Checkpoint
      8 hr
   9. 6.9
      ConfigureNotify (22) → recompute rows and columns → TIOCSWINSZ.
      3 hr
   10. 6.10
       InternAtom for WM_PROTOCOLS and WM_DELETE_WINDOW, then handle the ClientMessage so the title bar can close you.
       3 hr
   11. 6.11
       Split PutImage into row bands — a full 1920×1080 frame exceeds the maximum request length.
       3 hr

   Files

   astroterm-x11/src/{conn,auth,proto,window,events,keymap}.rs. Around 900 lines.

   Verify

   A window opens, fills with a gradient, prints the correct character to stdout for every key including modifiers, resizes cleanly, and closes from its own title bar.

   Traps

   Every X11 request is padded to a 4-byte boundary and its length field counts units of 4 bytes, not bytes. Miscount and the server desynchronises or ignores you silently. Compare your byte stream against xtrace on a working client.

   If you stall

   Use x11rb to get a window. Keep every part of Phase 7 moving, then swap your own client in behind the Surface trait afterwards. Of all the hatches here this is the one most likely to save the project — a window that exists beats a protocol you are still debugging.

8. 7

   astroterm-font and render — the first real terminal

   2–3 weeks·everything connects·Medium risk

   The phase where nine crates become one program. You already have 456 PSF fonts on disk, so there is nothing to download.

   Tasks

   1. 7.1
      PSF1 and PSF2 parser — 4-byte and 32-byte headers respectively. Half the fonts on your disk are each. Dispatch on the first two bytes.
      5 hr
   2. 7.2
      Font::glyph(ch) returning the packed bitmap for a character, falling back to U+FFFD.
      3 hr
   3. 7.3
      Framebuffer: a Vec<u32> plus the cell-to-pixel mapping from font width and height.
      2 hr
   4. 7.4
      Blit one cell — walk the glyph's bits, writing foreground where set and background where clear.
      3 hr
   5. 7.5
      Blit the whole grid and PutImage it. Text appears on screen for the first time. Checkpoint
      3 hr
   6. 7.6
      Damage tracking — redraw only rows flagged dirty, in contiguous bands.
      4 hr
   7. 7.7
      Wire the §9 poll loop over three descriptors: X11 socket, PTY master, signal pipe. Checkpoint
      5 hr
   8. 7.8
      Key encoding: arrows, Home/End, Page keys, function keys, Ctrl and Alt modifiers, and application cursor mode (arrows send ESC O A rather than ESC [ A when it is on).
      6 hr
   9. 7.9
      Cursor rendering — block shape, and blinking on a timer.
      3 hr
   10. 7.10
       Mouse selection and clipboard, using OSC 52 for copy.
       6 hr

   Files

   astroterm-font/src/psf.rs, astroterm-render/src/{fb,blit,damage}.rs, astroterm/src/main.rs. Around 800 lines.

   Verify

   vim, htop and less render correctly inside astroterm running bash, and resizing the window reflows them live.

   Traps

   Arrow keys breaking inside vim almost always means application cursor mode is being ignored. A wrong character width shifts an entire line — check §5's tables first.

   If you stall

   Skip font parsing entirely. Hardcode an 8×16 bitmap array for the 128 ASCII glyphs — mechanical, and a short script can generate it from an existing font. Get text on screen first; parse PSF afterwards.

9. 8

   Integration, conformance and polish

   Ongoing·the long tail·Incremental

   Point astroterm at astrosh instead of bash — the moment the project closes on itself and every piece is yours.

   Tasks

   1. 8.1
      Launch astrosh as astroterm's child. Your shell, in your terminal. Checkpoint
      1 hr
   2. 8.2
      Run vttest and work down menus 1, 2 and 11.
      ongoing
   3. 8.3
      Profile against the 16 ms frame budget; add MIT-SHM so frames skip the socket.
      1 wk
   4. 8.4
      TrueType rasterizer as a self-contained sub-project: sfnt container, cmap, loca, glyf, quadratic Béziers, scanline anti-aliasing.
      2 wk
   5. 8.5
      A config file for font, size and palette; publish your own terminfo entry.
      3 days

   Files

   astroterm-font/src/truetype.rs and assorted polish.

   Verify

   You use it as your daily driver for a full week without reaching for GNOME Terminal.

   Traps

   Resist starting 8.4 early. A TrueType rasterizer is genuinely fun and will absorb a month that Phase 6 needs more.

   If you stall

   Nothing here is load-bearing. 8a is worth finishing; 8b is a separate project you may quite reasonably never start.

What in this document has actually been verified

sudo apt install -y x11proto-dev
grep -E "define X_(CreateWindow|MapWindow|CreateGC|PutImage|InternAtom|GetKeyboardMapping) " \
  /usr/include/X11/Xproto.h
grep -E "define (KeyPress|Expose|ConfigureNotify|ClientMessage) " /usr/include/X11/X.h

Testing strategy

The three pure crates from Figure 7 are what make this tractable. Bytes in, grid out, no mocking required.

Risk register

The things most likely to cost you a frustrating weekend, and the mitigation for each.

Dependencies

One, and it contains no code.

Cargo.toml — every crate in the workspace

[dependencies]
libc = "0.2"   # astroterm-sys only. extern "C" declarations + constants.
               # Every other crate in the workspace has an empty [dependencies].

Explicitly out of scope

Named here so that skipping them is a decision rather than an omission.

Environment setup

A pleasing side effect of building from scratch: you need to install less than the dependency-using plan required. No SDL2, no X11 development headers, no font libraries.

Toolchain — Linux Mint 22.3

# Rust, via rustup (the apt package lags badly)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
rustc --version

# Linker only. No SDL2, no libx11-dev, no libfreetype-dev.
sudo apt install -y build-essential

# Debugging aids: vttest for conformance, xtrace to compare X11 byte streams
sudo apt install -y vttest xtrace

# Pick a font for Phase 7 — already on your disk. CHECK THE FORMAT FIRST:
zcat /usr/share/consolefonts/Uni3-TerminusBold16.psf.gz | head -c4 | xxd
# 3604 -> PSF1 (4-byte header)   72b54a86 -> PSF2 (32-byte header)
zcat /usr/share/consolefonts/Uni3-TerminusBold16.psf.gz > assets/font.psf

Workspace skeleton

cd ~/Desktop/"terminal project"
for c in sys pty unicode core font x11 render; do cargo new --lib astroterm-$c; done
cargo new --bin astroterm
cargo new --bin astrosh

cat > Cargo.toml <<'EOF'
[workspace]
resolver = "2"
members  = [
  "astroterm-sys", "astroterm-pty", "astroterm-unicode", "astroterm-core",
  "astroterm-font", "astroterm-x11", "astroterm-render",
  "astroterm", "astrosh",
]
EOF

git init && git add -A && git commit -m "workspace skeleton"

References

With no dependencies, specifications become the dependencies. These are the documents you are implementing.

• A parser for DEC's ANSI-compatible video terminals — Paul Williams The canonical state diagram behind §7. Read it before writing the parser, not after.
• X Window System Protocol, version 11 The specification you implement in Phase 6. Appendix B (the encoding) is the part you will live in — it gives the exact byte layout of every request and event.
• XTerm Control Sequences — Thomas Dickey The de-facto specification for what a modern terminal must support. The reference you keep open for the whole project.
• GNU libc manual — Job Control The one document that makes Phase 3 tractable. Its "Initializing the Shell" section is effectively a specification for the hardest code you will write.
• PSF font format — Andries Brouwer The complete PSF1 and PSF2 specification, in about two pages. Everything Phase 7's font parser needs.
• UAX #11 East Asian Width and UAX #29 Text Segmentation The rules behind Phase 5, plus the EastAsianWidth.txt and GraphemeBreakProperty.txt data files your build.rs parses.
• st — the suckless terminal A complete working emulator in ~5,000 lines of readable C, small enough to read end to end. Uses Xlib rather than raw protocol, but the grid and escape handling are the clearest reference there is.
• vttest The VT100/VT220 conformance suite from Phase 8's exit criteria. Brutal and extremely informative.
• POSIX Shell Command Language What astrosh is measured against in §11. Consult it for quoting and expansion order; do not read it front to back.

Glossary

Every term this document uses, defined in one or two sentences. 56 entries, alphabetical. §1 explains the same concepts in the order they build on each other; this is for looking one up mid-read.

alternate screen

The second, blank Grid that full-screen programs switch to with ESC [ ? 1049 h. Quitting vim restores your prompt because the original grid was never touched.

AST

Abstract syntax tree — the tree a parser builds. ls -l | wc becomes a pipeline node with two command children.

async-signal-safe

The short list of functions legal to call inside a signal handler. write qualifies; anything that allocates does not. Violating it deadlocks silently.

ARGB

A pixel packed into one u32 as alpha, red, green, blue — one byte each.

bitmask

Many true/false values packed one-per-bit into an integer. x & FLAG tests, x |= FLAG sets.

canonical mode

The terminal's default: the kernel buffers input until Enter and handles editing itself. Opposite of raw mode.

Cell

One character position on screen: a character plus foreground colour, background colour, and attributes. The grid is an array of these.

CSI

Control Sequence Introducer — the ESC [ that begins most escape sequences.

damage tracking

Redrawing only the rows that changed instead of the whole window. The difference between a fast terminal and a slow one.

DEC private mode

Escape sequences with a ? after the CSI, controlling terminal behaviour rather than drawing. ?25 is cursor visibility, ?1049 the alternate screen.

device file

A path that is really a driver, not stored bytes. Opening /dev/ptmx asks the kernel to create a new pseudoterminal.

dup2(old, new)

Makes descriptor new point at whatever old points at. The mechanism behind every redirect and pipe.

ECHO / ICANON

Two termios flags. ECHO makes the kernel print what you type; ICANON makes it buffer until Enter. Clearing both is most of what raw mode means.

escape sequence

A run of bytes starting with ESC (0x1B) that means a command rather than text — move the cursor, set a colour, clear the screen.

exec()

Replaces the program running in a process, keeping its PID and open descriptors. Never returns if it succeeds.

file descriptor

A small integer naming something a process has open. Index into a per-process kernel table. 0, 1 and 2 are stdin, stdout, stderr.

fork()

Clones the calling process. Returns twice — the child's PID in the parent, 0 in the child.

framebuffer

A plain array of pixels, one per screen position, laid out row by row. Drawing means writing integers into it.

glyph

The drawn shape of a character. A font maps characters to glyphs; the rasterizer turns a glyph into pixels.

Ground

The parser's resting state, where an incoming byte is ordinary text to be placed in a cell.

hex

Base 16, written 0x1B. One hex digit is exactly four bits, which is why byte values are written this way.

ioctl

"I/O control" — a numbered command sent to a driver for things that do not fit read or write, such as setting the terminal size.

job control

The machinery letting you suspend a program with Ctrl+Z and resume it with fg. Built on process groups and tcsetpgrp.

kernel

The core of the operating system. It owns the hardware; your program must ask it for everything via syscalls.

keycode / keysym

X11 gives you a keycode, a meaningless hardware number. Looking it up in the keyboard map gives a keysym, the actual symbol, which you then turn into bytes.

libc

The C standard library — ordinary functions wrapping syscalls, plus buffering and conveniences. Not the kernel.

line discipline

Kernel code between a PTY's master and slave. Echoes input, buffers lines, handles backspace, turns Ctrl+C into SIGINT.

master / slave

The two ends of a pseudoterminal. The emulator holds the master; the shell gets the slave as its stdin, stdout and stderr.

MIT-MAGIC-COOKIE-1

The X11 authentication scheme. A 16-byte secret stored in ~/.Xauthority that you must send in the setup request or the server refuses you.

OSC

Operating System Command — escape sequences of the form ESC ] ... BEL, used for the window title and clipboard access.

PID

Process identifier — the number the kernel uses to name a running process.

pipe()

Creates two joined descriptors; bytes written to one emerge from the other. The plumbing behind |.

poll()

Blocks until at least one of several descriptors has data available. Lets one thread wait on many sources at once.

process

A running program with its own memory and PID.

process group

A set of processes the kernel signals together. Every pipeline gets one, so Ctrl+C reaches all its stages.

PSF2

A bitmap font format: a 32-byte header then fixed-size packed glyph bitmaps, one bit per pixel. What console fonts use.

PTY

Pseudoterminal — a kernel-provided pair of descriptors that impersonates the serial terminal hardware Unix was designed for.

PutImage

The X11 request that copies a block of your pixels into a window. Opcode 72.

raw mode

Every keystroke delivered immediately, with no echo, no line buffering and no signal generation. What vim and htop use.

ring buffer

A fixed-size buffer that discards its oldest entry when full. Used for scrollback so memory has a hard ceiling.

scrollback

The lines that have scrolled off the top of the screen and are still held in memory.

self-pipe trick

A signal handler writes one byte to a pipe and does nothing else; the main loop notices the pipe is readable and does the real work. Sidesteps async-signal-safety entirely.

SGR

Select Graphic Rendition — the ESC [ ... m sequences that set colour, bold, underline and so on.

signal

An asynchronous notification from the kernel that interrupts a process. SIGINT is Ctrl+C, SIGWINCH means the window resized.

socket

A bidirectional channel between programs. A Unix domain socket lives as a path on disk, like /tmp/.X11-unix/X0.

state machine

A program always in exactly one named state, where each input causes a transition and perhaps an action.

syscall

A system call — the only way user-space code asks the kernel to do anything. A CPU instruction, not a function call.

termios

The struct holding every terminal setting: flags, special characters, speeds. Read with TCGETS, written with TCSETS.

TIOCSWINSZ

The ioctl that tells the kernel a terminal's new size in rows and columns. The kernel then sends SIGWINCH to the foreground program.

tty

Short for teletype. The general term for a terminal device.

UCD

Unicode Character Database — the published data files from which character width and grapheme-clustering tables are generated.

user space

Where ordinary programs run, walled off from hardware. The other side of the wall is kernel space.

VT100

The 1978 DEC video terminal whose escape sequences became the de-facto standard everything still implements.

waitpid()

Collects a finished child's exit status. Until called, the dead child remains a zombie.

XID

An X11 resource identifier. The client mints its own from a base and mask handed out in the setup reply.

zombie

A finished process still occupying a process-table slot because its parent has not called waitpid yet.