Rendering to non-terminal surfaces with CellSession

Copy Markdown View Source

ExRatatui.Session is the right primitive when whatever consumes the TUI speaks ANSI — a real terminal, an SSH channel, a TCP socket on the other end of a raw-mode client. It encodes the rendered frame as escape sequences and hands them to the transport.

ExRatatui.CellSession is the right primitive when the consumer is not a terminal: a Phoenix LiveView painting <span>s into the DOM, an embedded device rasterising glyphs to a 1bpp framebuffer, a screenshot tool dumping a frame to PNG/SVG, any future renderer for displays that don't accept ANSI. These consumers want the rendered cell buffer — (symbol, fg, bg, modifiers, skip) per cell — not bytes. CellSession exposes that buffer directly, skipping the ANSI encode/decode round-trip and the client-side terminal emulator that ANSI implies. Consumers that own real pixels can additionally receive bitmaps for the pixel widgets (see Pixel regions).

Session vs CellSession at a glance

SessionCellSession
BackendCrosstermBackend<SharedWriter>TestBackend
OutputANSI byte stream (take_output/1)Cell buffer (take_cells/1, take_cells_diff/1)
ConsumerTerminal emulator (real or virtual)Anything that walks structured cells
Touches OS tty?NoNo
Same draw API?Yes — draw/2 takes the same [{widget, rect}]
Same input parser?Yes — feed_input/2 is identical
Same lifecycle?Yes — new/2, resize/3, close/1, etc.

An ExRatatui.App can run on either without knowing the difference. Switching transports is a matter of which session type the transport constructs, not changes in the App.

Quick start

alias ExRatatui.CellSession
alias ExRatatui.CellSession.Snapshot
alias ExRatatui.Layout.Rect
alias ExRatatui.Style
alias ExRatatui.Widgets.{Block, Paragraph}

session = CellSession.new(40, 6)

paragraph =
  %Paragraph{
    text: "Hello from CellSession!",
    style: %Style{fg: :light_cyan, modifiers: [:bold]},
    alignment: :center,
    block: %Block{title: " demo ", borders: [:all], border_type: :rounded}
  }

:ok = CellSession.draw(session, [{paragraph, %Rect{x: 0, y: 0, width: 40, height: 6}}])

%Snapshot{width: 40, height: 6, cells: cells} = CellSession.take_cells(session)

CellSession.close(session)

cells is a list of %Cell{} structs in row-major order:

%ExRatatui.CellSession.Cell{
  row: 1, col: 14, symbol: "H",
  fg: :light_cyan, bg: :reset,
  modifiers: [:bold], skip: false
}

That's the whole API at the snapshot level. See the cell_dump.exs example for a tiny end-to-end script that paints a Paragraph, walks the snapshot, and prints it.

The cell shape

ExRatatui.CellSession.Cell documents every field. The conventions that matter when consuming cells: :reset colors mean "consumer's default" (terminal default, CSS default, "ink" on a 1-bit display), :modifiers come in a stable canonical order so == works without normalising, and renderers should treat skip: true as transparent (widgets that overlay, like Popup, use it).

Wide graphemes (CJK, emoji)

A two-cell-wide grapheme lands in its leading cell only. The following cell stays at its prior content (typically " "):

Cells for "中a" painted at (0, 0):
  (0, 0) symbol: "中"
  (1, 0) symbol: " "      # the continuation slot — left at default
  (2, 0) symbol: "a"

Snapshots vs diffs

take_cells/1 returns a full %Snapshot{} with every cell. Use it for the initial paint after a fresh client connects, one-off screenshots and exports, regression test fixtures, and debugging mid-stream — it is stateless and does not touch the diff baseline.

take_cells_diff/1 returns a %Diff{} with only the cells that changed since the previous diff call. Three cases produce a "full" payload (every cell appears as an op): the very first call after constructing the session (no prior baseline), a resize/3 between calls (the prior baseline is no longer a valid comparison reference), and a closed-then-reopened session (close/1 wipes the baseline). After that, ops typically cover only the small fraction of cells that actually changed.

session = CellSession.new(80, 24)

# Frame 0 — first call returns the full grid.
:ok = CellSession.draw(session, frame_0_widgets())
diff = CellSession.take_cells_diff(session)
length(diff.ops)  #=> 80 * 24 = 1920

# Frame 1 — typical small redraw.
:ok = CellSession.draw(session, frame_1_widgets())
diff = CellSession.take_cells_diff(session)
length(diff.ops)  #=> just the cells that changed

# No-op draw — diff is empty, no bytes to ship.
diff = CellSession.take_cells_diff(session)
diff.ops          #=> []

The op shape is identical to a snapshot's Cell. An op IS a cell at a position saying "set this position to this content." Clearing a cell is just setting it to a default-styled space.

Pixel regions for surfaces with real pixels

A cell grid is the right contract for a terminal-shaped surface, but some consumers own pixels: an e-ink panel, a browser canvas, an image export. On a plain CellSession the pixel widgets (ExRatatui.Widgets.Viewport3D, ExRatatui.Widgets.Image) fall back to half-block cells, which loses resolution and, on a 1-bit display, colour. A session that states its cell size in pixels gets the bitmaps instead:

session = CellSession.new(66, 37, font_size: {6, 8})

On such a session a Viewport3D in a pixel render mode (:auto, :kitty, :sixel, :iterm2) and an Image with any protocol but :halfblocks rasterise to RGB and ship the result out of band. Snapshots and diffs carry them in :regions, one ExRatatui.CellSession.Region per widget:

%CellSession.Region{
  x: 2, y: 3, width: 38, height: 33,      # the covered rect, in cells
  pixel_width: 228, pixel_height: 264,    # bitmap size in pixels
  format: :rgb8,                          # row-major, three bytes per pixel
  data: <<...>>
}

The contract is deliberately stateless. :regions is the complete list of regions on screen for that frame, never a delta; a region that is not listed is gone. Paint the cell ops first, then blit every region over its rect. The covered cells arrive as plain blanks, so the order is "background, then bitmap". The bitmap is normally width * font_width by height * font_height; the longest side is capped at 1280 px, so a very large rect gets a smaller bitmap to scale up uniformly. Image keeps its :resize semantics (:fit never upscales, :scale fills, :crop clips) and its :background fills the rest of the rect; without a background the region only covers the cells the picture touches. A consumer whose panel is mounted on its side turns the bitmap with ExRatatui.CellSession.Region.rotate/2, which rotates the pixels natively and leaves the cell rect alone — see Rendering to a Framebuffer. A consumer that needs the panel's own bytes resamples, clips, rotates and packs a region in one call with ExRatatui.Pixels.pack_rgb8/5 (Packing regions).

Sessions created without :font_size never carry regions, and the cell-mode render modes (:braille, :half_block, :ascii, :halfblocks) stay cell modes everywhere, so an app can still pick the cell look on purpose. Re-shipping unchanged bitmaps each frame is a conscious trade for a simple consumer; a transport that cares can compare data with the previous frame before pushing.

See headless_regions.exs for a script that renders a Viewport3D and an Image side by side through a font-size session and writes each region out as a PPM image, and Rendering to a Framebuffer for running an app on a pixel display.

Tight rects keep diffs small

Paragraph (and any styled widget) applies its :style to its entire rect, not just the painted text. A wide paragraph rect turns every cell in the rect into a styled cell, and the next diff correctly reports all of them as changed:

# Paint "X" red — but in a 5-wide rect.
paragraph = %Paragraph{text: "X", style: %Style{fg: :red}}
:ok = CellSession.draw(session, [{paragraph, %Rect{x: 0, y: 0, width: 5, height: 1}}])

diff = CellSession.take_cells_diff(session)
length(diff.ops)
#=> 5  — every cell in the rect is now `fg: :red`, even the empty spaces

For minimal diffs, keep styled rects tight to the content:

:ok = CellSession.draw(session, [{paragraph, %Rect{x: 0, y: 0, width: 1, height: 1}}])
length(diff.ops)
#=> 1  — only the "X" cell

This is the same behaviour an ANSI terminal would get — ratatui clears each cell in the rect to the paragraph's style. It just shows up directly in the diff payload here.

Driving an ExRatatui.App over a CellSession

The runtime server accepts a :cell_session transport tag in two shapes — a 3-tuple for transports that only need to ship rendered diffs, and a 4-tuple that adds an intent_writer_fn channel for transports that consume App-emitted intents (LiveView navigation, custom workflows, …).

# 3-tuple — frame-only transport. Intents from the App are silently dropped.
{:cell_session, %CellSession{} = cell_session, cell_writer_fn}

# 4-tuple — frame + intent transport.
{:cell_session, %CellSession{} = cell_session, cell_writer_fn, intent_writer_fn}

Both are 1-arity functions:

cell_writer_fn = fn %CellSession.Diff{} = diff ->
  # Ship the diff to wherever the renderer lives — a LiveView socket,
  # a websocket, an in-process callback, etc.
  send(target_pid, {:render, diff})
  :ok
end

intent_writer_fn = fn intent ->
  # Map App-emitted intents to consumer-side actions. Vocabulary is
  # entirely up to the consumer; the runtime forwards verbatim.
  send(target_pid, {:intent, intent})
  :ok
end

Then start the runtime:

{:ok, server} =
  ExRatatui.Transport.start_server(
    mod: MyTUI,
    transport: {:cell_session, cell_session, cell_writer_fn, intent_writer_fn}
  )

On every render the runtime server calls CellSession.draw/2, then CellSession.take_cells_diff/1, then hands the resulting %Diff{} to cell_writer_fn (with :regions filled in when the session was created with a :font_size). On every state transition where the App returned intents: [...], the runtime walks the list and calls intent_writer_fn once per intent, in emission order. Intents from a {:stop, state, intents: ...} transition fire before the server exits, so a TUI returning {:stop, state, intents: [{:redirect, "/login"}]} reliably reaches the consumer before the linked-server EXIT propagates.

Apps stay portable across transports: a TUI that emits {:redirect, path} from a callback runs unchanged over both a 4-tuple :cell_session (intent dispatched) and a :local tty (intent silently dropped — there's nothing to navigate).

See Runtime opts on ExRatatui.App for the App-side return-shape, and phoenix_ex_ratatui for a working consumer that turns intents into Phoenix.LiveView actions.

Performance notes

  • take_cells/1 allocates one %Cell{} per cell — ~1920 structs per call for an 80×24 grid. For tight loops, prefer the diff path.
  • take_cells_diff/1 clones the current ratatui buffer to use as next-call baseline; the diff comparison itself is O(width × height) of structural cell equality.
  • Cell encoding crosses the NIF boundary as a list of tuples, one per cell. The Elixir wrapper then maps tuples to %Cell{} structs. Both are linear in the number of cells emitted, which the diff path keeps small in steady state.
  • Always close unneeded sessions with close/1 — it deterministically drops the underlying ratatui terminal and the cached diff baseline rather than waiting for BEAM GC.