# Images

Render real images — PNG, JPEG, GIF, WebP, BMP — inside a TUI. The same model code displays Kitty graphics in a Kitty terminal, halfblocks in Livebook, and adapts gracefully when the audience's terminal support is unknown.

Built on [ratatui-image](https://github.com/ratatui/ratatui-image).

## Quick start

```elixir
def mount(_opts) do
  {:ok, picture} = ExRatatui.Image.new(File.read!("priv/slides/cover.png"))
  {:ok, %{picture: picture}}
end

def render(state, frame) do
  area = %Rect{x: 0, y: 0, width: frame.width, height: frame.height}
  [{state.picture, area}]
end
```

`ExRatatui.Image.new/2` decodes once and returns a stateful widget handle. Hold it in the model and reuse it across renders — re-encoding only happens when the protocol changes or the rect resizes. Image format is auto-detected from the bytes; no extension or content-type hint required.

Bad bytes return `{:error, {:decode_failed, message}}` rather than raising, so a placeholder can be rendered gracefully.

## Options

`ExRatatui.Image.new/2` accepts a keyword list:

| Option | Values | Default | What it does |
|---|---|---|---|
| `:protocol` | `:auto`, `:halfblocks`, `:kitty`, `:sixel`, `:iterm2` | `:auto` | Which terminal protocol to render with. `:auto` resolves at render time against the transport (see [resolution table](#protocol-resolution)). Explicit protocols are honored except over a plain `CellSession`, where `:halfblocks` is forced; a `CellSession` created with a `:font_size` turns every non-`:halfblocks` request into a pixel region. |
| `:resize` | `:fit`, `:crop`, `:scale` | `:fit` | `:fit` preserves aspect inside the rect (anchored top-left if smaller). `:crop` preserves aspect, fills the rect, crops the overflow. `:scale` preserves aspect like `:fit`, but also grows a smaller image to fill the rect. |
| `:filter` | `:catmull_rom`, `:lanczos3`, `:triangle`, `:nearest` | `:catmull_rom` | Resampling filter when `:fit` or `:scale` resizes the image (`:crop` never resamples). See [Resampling filters](#resampling-filters). |
| `:background` | `nil`, any `t:ExRatatui.Style.color/0`, or raw `{r, g, b}` | `nil` | Color used to fill transparent pixels / unused area for halfblocks. |

To change options later, build a new handle — the widget struct just wraps the resource ref.

## Protocol resolution

Each transport stamps its own capability hint. Widget-level `:auto` resolves against it:

| Transport | `:auto` resolves to | Explicit `:kitty` etc. |
|---|---|---|
| Local terminal (no probe) | `:halfblocks` | Honored — emits raw escapes |
| Local terminal (after probe) | The detected protocol | Honored |
| SSH (`image_protocol: :kitty`) | `:kitty` | Honored, default `(8, 16)` font size |
| SSH (`image_protocol: :kitty` + `image_font_size: {10, 20}`) | `:kitty` | Honored with accurate scaling |
| SSH (no opts) | `:halfblocks` | Honored |
| Distributed (`image_protocol: :kitty`) | `:kitty` | Honored, default `(8, 16)` font size |
| Distributed (`image_protocol: :kitty` + `image_font_size: {10, 20}`) | `:kitty` | Honored with accurate scaling |
| Distributed (no opts) | `:halfblocks` | Honored |
| `CellSession` (Livebook / Kino) | `:halfblocks` | **Forced to `:halfblocks`** (escape sequences can't survive cell diffing) |
| `CellSession` with `font_size: {w, h}` (e-ink, canvas) | A pixel region | A pixel region — the decoded bitmap ships in `Diff.regions`, see the [CellSession guide](cell_session.html#pixel-regions-for-surfaces-with-real-pixels) |

This means **the same model code is portable**: a slide deck that renders pixel-perfect Kitty graphics in a local Kitty terminal will silently fall back to halfblocks when the same `ExRatatui.App` is driven from a Livebook cell — no branching. A consumer that owns pixels and states its cell size gets the real bitmap from the same code, with `:resize` and `:background` behaving as they do on a terminal (`:fit` never upscales, `:scale` fills, `:crop` clips; `:background` fills the rest of the rect).

Image widgets work over **every** transport, including `ExRatatui.Distributed` — image bytes are snapshotted into the render tree, costing roughly the source file size per frame on the wire. Fine for stills; watch the bandwidth when animating large images.

### Probing the local terminal

`ExRatatui.Image.auto_local_protocol/1` writes a query escape sequence and waits for the terminal's reply. On success the result is cached on the terminal reference; on no-TTY / no-reply the cache stays empty and `:auto` falls back to halfblocks. Either way it's safe to call from any environment.

There are two ways to wire it in.

**Inside `ExRatatui.App`:** return `probe_image_protocol: true` from `mount/1`. The runtime calls `auto_local_protocol/1` right after mount, on the `:local` transport only (CellSession forces halfblocks; SSH / Distributed use the session-level `:image_protocol` opt instead):

```elixir
@impl true
def mount(_opts) do
  {:ok, initial_state, probe_image_protocol: true}
end
```

The probe is automatically skipped under `test_mode: {w, h}` so headless tests don't accidentally write probe escapes.

**Outside `ExRatatui.App`** (e.g. `ExRatatui.run/1`):

```elixir
ExRatatui.run(fn terminal ->
  ExRatatui.Image.auto_local_protocol(terminal)
  # ... rest of the app
end)
```

To make an independent decision based on the probe, `ExRatatui.Image.probe_terminal/0` returns `{:ok, %{protocol: atom, font_size: {w, h}}}` or `{:error, reason}` without touching any cache.

### Telling SSH / Distributed what protocol the client supports

An SSH or Distributed client terminal can't be probed, so the audience declares it at start time. Pair `:image_protocol` with `:image_font_size` to get accurate Kitty/Sixel/iTerm2 scaling — without the font size, the encoder falls back to `(8, 16)` cell pixels which mis-scales on most modern terminals (Kitty/Ghostty are closer to `(10, 20)`):

```elixir
# SSH daemon
ExRatatui.SSH.Daemon.start_link(
  mod: MyApp.TUI,
  port: 2222,
  image_protocol: :kitty,
  image_font_size: {10, 20}
)

# Distributed attach
ExRatatui.Distributed.attach(:"app@host", MyApp.TUI,
  image_protocol: :kitty,
  image_font_size: {10, 20}
)
```

Per-image explicit choices (`ExRatatui.Image.new(bytes, protocol: :sixel)`) are always honored, regardless of the session-level hint.

## Font-size caveat

Cells aren't pixels. The render pipeline needs the terminal's cell-pixel dimensions to scale Kitty / Sixel / iTerm2 payloads correctly. The default is `(8, 16)`; `auto_local_protocol/1` replaces it with the real value reported by the terminal. If Kitty graphics look mis-scaled, run the probe. On a `CellSession` the consumer supplies the value itself through `font_size:`, and passing the same tuple to `ExRatatui.Image.render_size/4` makes the prediction match the region the session ships.

## Examples

* [`examples/images/image_demo.exs`](https://github.com/mcass19/ex_ratatui/blob/main/examples/images/image_demo.exs) — interactive viewer with `p` to cycle protocol, `r` to cycle resize mode, and a live status panel showing the render output dimensions. Runs on every transport via the same script:

  ```sh
  mix run examples/images/image_demo.exs                # local terminal
  mix run --no-halt examples/images/image_demo.exs --ssh
  elixir --sname app --cookie demo -S mix run --no-halt \
    examples/images/image_demo.exs --distributed
  ```

* [`examples/cell_session/headless_image.exs`](https://github.com/mcass19/ex_ratatui/blob/main/examples/cell_session/headless_image.exs) — fetch a photo, render through `CellSession`, dump the cell grid to stdout with ANSI fg/bg colors. The Livebook / Kino path; safe to run anywhere (no TTY required).

Both accept an `IMAGE_PATH` env var, default to fetching from `picsum.photos` once at startup, and fall back to an embedded 1×1 PNG if the network is unreachable. The SSH demo also honors `IMAGE_PROTOCOL` / `IMAGE_FONT_W` / `IMAGE_FONT_H` env vars.

## Telemetry

Each `ExRatatui.Image.new/2` call emits a `[:ex_ratatui, :image, :decode]` span:

* `:start` — `%{format: atom, bytes: non_neg_integer}` (format sniffed from magic bytes: `:png`, `:jpeg`, `:gif`, `:webp`, `:bmp`, or `:unknown`).
* `:stop` — adds `:width` and `:height` on success, or `:error` (reason) on failure.

Per-render encode timing (Kitty / Sixel / iTerm2 payload generation) is rolled into the existing `[:ex_ratatui, :render, :frame]` span — they happen inside the same NIF render pass.

## `:fit` and `:crop` do not upscale — this is intentional

This is the single most common point of confusion, so it gets its own section.

Both `:fit` and `:crop` clamp output to the **source image's natural pixel size**. They never upscale. Given a 400×300 picsum photo in a render area sized 800×500 target pixels:

| Mode | Render output | Visible result |
|---|---|---|
| `:fit` | 400×300 | Image at natural size, anchored top-left, ~50% empty area |
| `:crop` | 400×300 | Identical to `:fit` here — both clamp to source dimensions |
| `:scale` | ~640×480 (aspect-preserving fill) | Image fills the area |

The difference between `:fit` and `:crop` only manifests when the **source is *larger* than the target** on at least one axis: `:fit` shrinks to fit (whole image visible, letterboxed); `:crop` keeps natural size and shows a window into the source corner.

This is upstream ratatui-image behavior, not something we layer on. We expose `ExRatatui.Image.render_size/4` to predict what each mode will do for a given combination of source dims, cell area, and font size — useful for status panels, layout decisions, or just understanding what's on screen. The `examples/images/image_demo.exs` example uses it to surface the render output dimensions live while cycling modes.

For "fill the area regardless of source size," use `:scale`. That's the only mode that upscales.

## Resampling filters

An image almost never lands on its slot at a whole-number scale, so `:fit` and `:scale` resample it. The default `:catmull_rom` (bicubic) keeps edges sharp and small text legible in both directions. `:lanczos3` looks about the same and costs a little more; `:triangle` (bilinear) is softer.

`:nearest` copies whole source pixels. At a non-integer scale it drops some rows and columns and doubles others, so 1-px strokes vanish or thicken: a screenshot of a dashboard comes out with misread digits and broken letters. Its use is pixel art and other images whose hard edges matter more than smoothness, ideally at whole-number scales.

The resample only runs when the rect or the cell size changes, not on every frame, so the choice of filter doesn't affect frame time.

## Known limitations (v1)

* **No animated GIFs.** First frame only. Frame-by-frame animation needs render-loop integration that isn't here yet.
* **No SVG.** The underlying `image` crate doesn't include an SVG decoder.
* **No streaming / progressive decode.** Bytes are decoded all at once at `new/2`.